Data validation teirs

Data is either valid or invalid right? Well, sort of. I'm working with medical data like the Blood CHEM-7 test. This data has the possibility of being valid and being abnormal. So there is this concept of valid capture data and data that is out of range.

For example here's the CHEM-7 data definition in English:

• BUN: 7 to 20 mg/dL


• CO2 (carbon dioxide): 20 to 29 mmol/L


• Creatinine: 0.8 to 1.4 mg/dL


• Glucose: 64 to 128 mg/dL


• Serum chloride: 101 to 111 mmol/L


• Serum potassium: 3.7 to 5.2 mEq/L


• Serum sodium: 136 to 144 mEq/L

I might code this as an Object and write validators to make sure that the saved data is in range. My Groovy object for GORM might look like this:

class Chem7 {
   static belongsTo = [patient:Patient]
   Integer bun
   Integer co2
   Float creatinine
   Integer glucose
   Integer serumChloride
   Float serumPotassium
   Integer serumSodium

   static constraints = {
      bun(nullable:false,range:7..20)
      co2(nullable:false,range:20..29)
      creatinine(nullable:false,range:0.8..1.4)
      glucose(nullable:false,range:64..128)
      serumChloride(nullable:false,range:101..111)
      serumPotassium(nullable:false,range:3.7..5.2)
      serumSodium(nullable:false,range:136..144)
   }
}

... but I've got a problem. The very reason that we conduct the CHEM-7 blood test is to find irregularities. The constraints that I've expressed only detail the normal range of data for a CHEM-7... so what should I do?

If I make the table definition in the database very generic. Allowing the table to accept NULL values or numbers of any kind for each of these values then I can do this with a chem7 object:

   chem7.save(validate:false)

... as long as the table underneath the Grails application can accept the data then I'll save my irregular chem7 test results. This would mean that anyone who was editing the chem7 would get a warning about how it was not a normal test result. Yet no matter how irregular the test was we could still save the results and log who entered them.

So imagine a "Forced Save" button next to the normal save button on your Grails application. Simple enough to write. But it allows us to use validation in a tiered fashion. Validation at the controller and validation at the database. The important thing to keep in mind is that the validation must be weakest at the database level.

Don't Bother Me with the Details

When the framework essentially gets out of your way and lets you work with the level that you are interested in at that moment... that's a good framework.

If I'm working Ajax and UI I don't want to be bothered with security, servlets, persistence, or anything else. When I'm working persistence I don't want to be bugged by UI elements. And so on. The idea of "don't bother me with the details" applies to all levels and means different things at different levels. I think the Grails team really understood that (if not explicitly then implicitly).

Grails gets the wonderful benefit of living on a stack where people have sweat the details for years. Yet it still manages to hide those details when they don't matter to the problem you are solving. So Grails has the potential of living in the best of all worlds. And it gets this virtue by way of the Groovy Programming Language.

As Marc Palmer points out there are still holes in the pure play Groovy/Grails space. The Grails community still has a learning curve to get over: what should be pure play Groovy/Grails and what should be hybrid Java/Grails? How ever we answer that it should mean that the framework gets out of the way and lets the developer work directly on the problem at hand.

New Year's Thoughts

If it takes ten years to get good at software development then I'm over due to get good by now I suppose. I am also going to have to take some calculated risks in the next year. So let's see what I was talking about last year and how things are shaping up so far.

Last year (a little in advance of the new year) I posted some thoughts I had been working on. In last year's "new year's thoughts" I said about software design:

Get to the core thought, the core idea, ignore as much minutiae as possible, get the problem solved. Keep the idea clean. Keep the core thought from being tangled in a mess of details that have nothing to do with the answer you are after.

... and I predicted that FP, Rules Engines, and DSL would become more important going forward ...

Functional programming, rules engines, and dynamic languages promise to free us from those mind-killing details that we don't really care about anymore as application developers. But they don't really address the need to create adaptive systems or systems that are able to span across computational mediums.

I had both identified the current trends and what I saw as the core problem. The fact of the matter is that the ideas behind the new fads in programming languages themselves won't really tackle the real problem in software engineering.

... so what is new in the last 365 days?

I recently found Kevin Kelly's the next 5,000 days TED talk. He has very keenly pointed out something that seems at once profound yet very simple. I can't imagine not having this idea to work with anymore.

If you want to really be relevant to the advance of technology today then you will position yourself to take advantage of Kevin Kelly's central point: There is only one machine and every screen in the world is a portal into that one machine. The reason the iPhone is a rampant success is that it really is a window into the one machine and acts like it.

The devices of the internet are approaching a complexity rivaling biology. To think we can understand them with our existing tools is a little foolish. We are yet to really develop the mental tools to deal with this new complex system. We will need these cognitive tools and we need them soon. Once we have them we will think it was odd they were so hard to conceive of. Much like the idea behind the number zero. It is hard to imagine how the first person conceived of the idea behind the number "zero" yet it is hard to imagine not having it.

This idea of "the one" is the guiding idea behind much of my own development efforts today. I didn't have a term for this before and I'm thankful to Kevin Kelly for coining a term for this idea. In many systems I confront there isn't an understanding of "the one" and data lives on barren islands of isolation.

In my new job I am very pleased to be allowed to participate in the development of interesting new technologies and I hope that I can find seemingly simple yet powerful changes to introduce that are like the idea of "zero" ... powerful, simple, and seemingly obvious once they are fully understood. That is the beauty of the truth when you have found it. It seems too simple and too obvious to have been missed.

There is a way to express UI, data, and logic that still is to be discovered. It will be obvious after we've found it. It will be profoundly different than what we have now. We won't recognize crossing into it until after we have already done so.

I expect that in 12 months we'll have better knowledge of which ideas are wrong and which ones more accurately reflect the emergent reality of the one machine. We will learn better the shape of the world it is carving out. That shape is not so much our invention anymore as it is a new undiscovered country which we are just now charting.