2009-12-10

Domain Specific Languages: fighting accidental complexity in a sea of technology

I had a great time presenting at UNC's Programming Practices group today. This was the first time I've presented in about a decade so I'm glad that folks stuck around and gave me feed back.

Slides from the presentation:



I wrote speaker's notes for nearly each slide. If you click the menu button and click the link to the full document you can find a button that will pop up the speaker's notes for you.

For code samples, take a look at this example by Tim Yates. My live coding example was a reimplementation of this using a Test Driven Design (TDD) technique. TDD doesn't translate well into a recorded medium. I tried to show how we could stumble toward the same (or similar) design by using TDD and what we knew to extend our knowledge and grow beyond it.

I made a simple extension on Tim's example. I added a simple way to make his internal DSL into an external DSL. I use the GroovyShell object:


// def list = DateDSL.interpret(stringOfCode) // could be a file?
static interpret(String code) {
def shell = new GroovyShell();
Closure closure = shell.evaluate("{->" + code + "}")
closure.delegate = new DateDSL()
closure.call()
}


Finally, I tried to tie the idea that Domain Specific Languages hide accidental complexity of a system behind "simpler" API and externalize-able syntax. These goals provide not just more focused ways for application developers to write software. They outline a design aesthetic. An aesthetic that fits well with the idea of empowering amateur programmers.

I ended with the assertion that this sea of technology will end in a world where everyone programs. But not everyone will be a programmer. We will think of programming the way we think of writing. Just because you can write it doesn't make you a writer. The ubiquity of computing means humanity will eventually view computer programming the same way... as ubiquitous as reading and writing.

One way to make sure this happens is to develop Domain Specific Languages that empower domain experts to write software.

2009-12-04

Surprized by how little I use multithreading...

Back in 2000 I was working on multithreaded applications. Since I was working in Posix environments in C I used the Pthreads library. I learned to use Pthreads from both C/C++ and Perl contexts but I had trouble with managing Pthreads and in 2000 I perceived that Java's threading would be easier to use.

For an honest and unbiased comparison of Pthreads and Java threads based on the state of things (at that time) you can look at this paper by Wim H. Hesselink which manages to cover in 8 pages most of the differences. It was these differences between C/C++ and Java that made me interested in shifting from Posix to Java development.

As it turns out, now that I work primarily in Java (something that took me the better part of a decade to orchestrate), I rarely make practical use of multithreading in Java. For the most part you don't need it when you work in an application server. In the cases I typically encounter if you structure your Java application properly it doesn't need explicit multithreading.

Notice, I said explicit. In Java Application Server environments we have tools like messaging queues and timers. Using these Application Server tools properly implicitly produces the same effects as you would get by setting up your own threads and these techniques tend to create fewer bugs since you aren't forced to write thread management code that is easy to make mistakes with.

So, what I'm looking for now are examples of problems that using JMS, asynchronous request processing, or Quartz timers won't provide a clean solution. Why do I need to use multithreading explicitly in today's managed Application Server world? What am I missing?

And once I'm developing multithreaded code... how do I test it?

I've gotten responses from some prominent developers out there already and I hope that we can collaborate and share from the experience. I am excited at the chance to learn from some of the people that I consider to be great thinkers in our community. Either way I know I'll learn something. In the end that's what I'm really after.

2009-11-10

Simple Grails Plugin Repository Mirror Script

We needed a local offline repository for Grails plugins. I thought this would be a good chance to contribute a grails application to the community. It turns out there already is one called plugrepo that you can just download and use.

The plugrepo grails application is a very nice looking and if I'd known about it I wouldn't have spent the effort building my own.

What started this is a command line Groovy script I had hooked to a cron job on our Ubuntu Linux Apache server to create a mirror. I'll go ahead and post this here in case someone would rather have a flat Apache served archive... otherwise take a look at plugrepo.

To use the archive setup your grails app with this additional configuration file:

// grails-app/conf/BuildConfig.groovy
grails.plugin.repos.discovery.myRepository="http://myhost.test.com/grails/"


The following groovy script is meant to be run on the Ubuntu command line and
populate the /var/www/grails directory as if it were a grails plugin repository.

#!/usr/bin/env groovy
import groovy.xml.MarkupBuilder
/**
* Shawn Hartsock
*
* This is a quick and dirty Groovy CLI script for creating Grails plugin
* repository mirrors.
*/

def uri = "http://myhostname.test.com/grails"
def destinationRoot = "/var/www/grails"
def version = "grails-1.1.1"
def separator = System.getProperty("file.separator")
def latestOnly = true

if(!args) {
println """

this script is designed to take plugins-list.xml files (as arguments) from
a ~${separator}grails${separator}1.1.1 (or other version) folder and download each
plugin to the $root working directory.

It will then output a modified a $destinationRoot${separator}.plugin-meta${separator}plugins-list.xml
file modified to reflect the new locations of the files.

"""
System.exit(1);
}

try {
def destination = destinationRoot + separator + version
new File(destination).mkdir()
def writer = new StringWriter()
def xml = new MarkupBuilder(writer)
List files = []
args.each { arg ->
if(arg == "-a") {
latestOnly = false
}
if(new File(arg).exists()) {
files.add(arg)
}
}
xml.plugins(revision:2) {
files.each { file ->
process(file, destination, separator, uri + "/" + version, xml, latestOnly)
}
}
new File(destinationRoot + separator + ".plugin-meta").mkdir()
def outFile = destinationRoot + separator + ".plugin-meta" + separator + "plugins-list.xml"
def out = new File( outFile )
if(out.exists()) {
if(out.delete()) {
out = new File( outFile )
}
}

out << writer.toString()
} catch(Exception ex) {
System.err.println(ex.message);
System.exit(0);
}

void process(String fileName, String destination, String separator, String uri, xml, latestOnly) throws Exception {
def text = new File(fileName).text // may throw file access exceptions
def plugins
try {
plugins = new XmlParser().parseText(text)
assert plugins?.plugin.size() > 0
assert plugins.plugin[0].release != null
assert plugins.plugin[0].release.file != null
} catch(java.lang.AssertionError ae) {
throw new Exception("File $fileName does not contain plugin definitions: " + ae.message)
} catch(Exception ex) {
throw new Exception("File $fileName could not be parsed as plugin XML: " + ex.message)
}
println "${plugins.plugin.size()} plugins found in $fileName "
println "\t"
plugins.plugin.each { p ->
xml.plugin( name:p.'@name', 'latest-release':p.'@latest-release') {
p.release.each { r ->
def get = true
if(latestOnly) get = (r.'@version' == p.'@latest-release')?true:false
if(get) {
def f = download(r.file.text(), destination, separator)
if( f ) {
def url = uri + "/" + f
println "\tregistering:\n\t " + url
release(type:r.'@type',tag:r.'@tag',version:r.'@version') {
title(r.title.text())
author(r.author.text())
authorEmail(r.authorEmail.text())
description(r.description.text())
documentation(r.documentation.text())
file(url)
}
}
}
}
}
}
}

def download(String urlString, String dest, String separator) {
// note: this is a web URL so you tokenize on "/" not file.separator
def fileName = urlString.tokenize("/")[-1] // last name on URL
def destFileName = dest + separator + fileName
if( new File(destFileName).exists() ) {
println "\t $destFileName already present in the repository"
return fileName
}
print "\tdownloading $fileName to $dest directory\t"
def file = new FileOutputStream( destFileName )
def out = new BufferedOutputStream(file)
out << new URL(urlString).openStream()
out.close()
println "."
return fileName
}

2009-11-02

Apache Ivy: Componentization? What's hot and what's not?

I'm working with Apache Ivy over the next few weeks. The problems I'm trying to solve are around the testability of a J2EE application and its functional decomposition. I have chosen Ivy after an evaluation period due to its simplicity and how easy it is to port Ivy into existing Ant build systems.

In the case of the Grails applications I'm working with this problem domain is all rather straight forward. The Grails applications can be functionally decomposed easily along plugins and modules. They were all developed with a Test Driven Design (TDD) mentality and so have a large suite of test automations around the application. NOTE: I have not yet picked a code-coverage tool for this environment, however, so don't ask about code coverage for now okay... suggestions are welcome.

The problem still lies with the J2EE application. I have several components I can identify that have not changed in years since the original system designers bequeathed the code to their heirs. The original system had no automated testing but had ample "test scripts" which were human driven. In some respects I'm shocked at this approach to testing but I can't say it surprises me.

I've been in work environments that literally employed armies of testers. (No really, it was literally the Army... literally armies of testers.) And while there is a place for this if you can afford it ... doesn't it make sense to focus those human-level testers on testing human-level problems? I'm not talking about getting rid of those armies of testers just focusing them on the most interesting problems, saving their collective power for the big issues.

So, here's my hypothesis about how you should decompose an application already long in development, with no automated test harness so it can be better managed. It's a work in progress so please help me knock off the rough edges or if you think I'm daft... let me know.

I'm operating under the theory that the human testing in fact exercises all relevant existing code as it is compiled and bundled up as a part of the whole system. Any components we identify were in that tested whole. Any components we create will be tested under the unified whole. Therefore any movement of components creates no net change in the whole. This initially appears pointless but positions us to begin creating automated unit and integration tests around the identified components. The outcome creates no user-visible results initially but is incredibly important since it makes adding features much more certain.

The end goal of the introduction of Ivy is to identify stable framework components and put those components under tests that mimic the current human-based test scripts. The end result will be the ability to identify the volatile system components and isolate them for focused testing and design work. This is desirable because you isolate the system's accidental complexity and help keep it away from its intrinsic complexity. You should in the process be able to identify layers of abstraction in the system.

NOTE: An interesting side-effect is that classes designed for use with RMI may not change frequently right now but we have observed that they will "break" backwards compatibility at seemingly random intervals. Since our system is distributed this poses a problem. Decomposing these RMI interfaces and classes into their own Jar (compiling them separately and only on the event that they are changed) means actions that change them thus breaking compatibility between nodes will become very apparent since it will be harder to make the change accidentally.

Working theory: Componentization of large existing systems

I'm thinking that (in the large existing system) you want to only decouple packages of classes from the grand unified build that have little change between revisions. You should be able to identify these "stable components" by creating a "heat map" of the repository and watching the rate-of-change in the change control system. The more frequent the changes the "hotter" the class. The "hotter" the class the closer to the other frequently changing components it should be... ostensibly going under a new round of full tests with them. I would only select the coldest classes to be moved into components for control by Ivy.

I will take these carefully selected classes and move them to a separate module to be built and packaged as a single Jar file. These will be placed in the Enterprise Ivy repository for management by Ivy. At build time the project will download these Jar files, just like other Ivy dependencies, from the Enterprise's Ivy repository.

When the unified whole goes under test part of that whole will be the independent Ivy managed Jars. These Jars can be instrumented during our human-driven tests to see how they are exercised by those armies of humans. With that documentation I can then devise unit and integration tests to reenact those human-level tests on the isolated Jars. That means, the next round of the application's life cycle will have a set of automated tests that document how the system operates.

Once we know what the colder classes do we can begin to formulate a framework based on them. And that begins the first steps to identifying and targeting changes to the system to add to what it can do... or designing a replacement... or designing a new feature. Each time behavior changes in the future it will be more explainable and thus more controllable.

And you start getting there by identifying what's hot and what's not.

Have I gone wrong? Commentary?

2009-10-01

Should I put that logic in my Grails domain class, service class, or a controller?

I don't honestly know. I do have a feel for when this is appropriate and when it isn't. For example I think Anemic Domain Model is a valid anti-point. It completely jibes with how I feel the benefit of an Object Oriented Programming (OOP) system should work... in that I don't think domain classes should be just a representation of data.

The whole point of OOP is to encapsulate data and methods in a modular way keeping logic bundled together with the data it operates on. Let's work on a concrete example... assume this is a domain class in a Grails project.


class Person {
String firstName
String lastName
Date birthDate
}


So far so good. So if I want to figure out a person's age... where do I put that? The class seems like a good place...


class Person {
String firstName
String lastName
Date birthDate

def getAge() {
def now = new Date()
now[Calendar.YEAR] - birthDate[Calendar.YEAR]
}
}


... in fact anything that operates on the domain data in the object seems like it fits well here. If we run generate-all on this class we get a UI that looks like this...


Marking non-persistent services as transient keeps them from getting sent to the database should we need them... for example if I had a "fooService" then I would get this injected in my domain class like so...

class Person {
transient fooService

// ... snip ...
... but I'll not bother with an example just now.

So when does the logic belong in a service or a controller? Well, when it stops being directly driven by the domain model. For example things could get messy if we start injecting security checks into the domain model. I would find that a bit messy to maintain.

Similarly if there was a Person to Person relationship that required some logic that checked states of both objects...


class Person {
Person spouse
//... snip ...
void marry(Person other) {
if(!other.spouse || other.spouse.id == this.id ) {
other.spouse = person
}
else {
throw new Exception("These two cannot wed")
}
}


... while this works it feels a tad clunky because I have to do this bit ...


if(!other.spouse || other.spouse.id == this.id )

... and since we have to somehow modify the other class too and it feels odd to have the this person object modify the other person object.

... but if there was a minister service I could write this...


class MinisterService {
//... snip ...
void marry(Person left, Person right) {
if(!left.spouse && !right.spouse) {
left.spouse = right
right.spouse = left
}
else {
throw new Exception("These two cannot wed")
}
}
// ... snip ...
}


... and this feels a tad better since the Person class no longer needs to carry around the knowledge of all the possible marriage rules that a lawyer might enact. This also models real life better since people visit ministers, justices, or captains to get married in real life... and we cleanly avoid the issue of people modifying each other.

There's other rules that I could probably explain if I was more articulate. Has anyone else thought about this? What tweaks would you put on these ideas? How would you express this rule?

Note: this is a quick reply to this tweet.

2009-09-08

A Programmer's Isogloss

One of my coworkers has a degree in linguistics and this idea comes from talking with her about language and isolation. Call it my you should really go to that conference rant if you like. The topic of conversation give me a stark realization that I think anyone that is responsible for the care an feeding of programmers should be cautious of.

Naturally, computer programmers use programming languages and the more complex the language and free form it is the more likely it is to have dialect, idiom, and isogloss just like natural languages.

In particular if a speaker or group of speakers live in isolation long enough they will develop their own little quirky habits. These quirks start off as innocent inside jokes which include conventions that we recognize in API. Given time the isolated group can start to invent new idioms which approach an unintelligible complexity to an outsider. The divisions between various frameworks that share Java as a base language and their definitions of terms is a good example.

So how do you break this cycle of main languages fragmenting and breaking up into new languages? You go out and regularly talk to others about your code, your architecture, and you get a feel for how others say words or convey meanings. You make your community of language larger. Every company needs to get its programmers out there and talking to other developers. One of the best ways is through conferences.

The benefits will be that your corporate architecture will stay closer to the wider world's interpretation of the language and idioms you use (think: style, practice, and API). Your developer's will be more creative and valuable as they benefit from the exchange of ideas. And you will know that you have a pool of similarly talented consultants and developers to seek out when you need them.

Other good ways to let your programmers get out there and learn more is to encourage the use of social networking in responsible and applicable ways. I used this recently to help think about const versus final in our code base.

Another idea I've been toying with is to have at least one small company backed Open Source project that your people are encouraged to contribute to that you can also use for getting outside feedback and perhaps to recruit new people into your company from. This is a novel idea but I think with some sensible boundaries it can pay off even for closed source companies.

Those are just some of my thoughts on the topic. But essentially you want to avoid putting your programmers in the position that all they ever do is "talk" (via programming) to the computer and themselves. You need to talk with other people whether in code or English to keep your skills sharp and to avoid drifting off into your own strange dialect. The dangers to both the company and the employee of drifting off into your own little world are profound. It can mean obsolescence for both parties and that doesn't help anyone ... except maybe your competitors.

2009-09-05

Configuration Best Practice

So rattling around in my head is this idea of configuration. When should you configure? What should you configure? How much should you configure?

As a rule of thumb I need to perform a software task in three relevant contexts to spot the important patterns for the system. That is to say I need three unique implementations of a system to spot how to generically implement a framework around the problem domain. The same is true for configuration.

Until I see three distinct scenarios where a system will be used I usually can't rightly discern what is going to need to be configurable and what isn't. I can usually make a good guess as to what will be needed by the time I'm working on two implementations ... but it really takes the third one for things to gel.

So if I were to number from best to worst case for configurations... I would put them in this order:

  1. configuration by convention (as we see in Grails)

  2. dynamic configuration by environment

  3. explicit configuration by administrator



Once you figure out that part of a system is going to actually need to flex then you should refactor for just enough modularity to reach the changes you need see. When doing this work the best thing to do is to make your system self configure by convention. This way we provide the necessary configuration without increasing the load on the end developer or administrator.

Next best we can do configuration by picking up things from our environment and working out the values we might otherwise need configured. This allows the administrator of our application to merely move the application code base to the correct place in the "cloud" or the correct server.

The last thing we want is for the administrator to have to open a dialog and enter data about the environment for your system to use at run-time. This can be an error prone process and it is possible that the environmental factors will change forcing the administrator to always be conscious of additional variables in the system.

In the perfect world our software systems would have enough intelligence to ask the environment for what they need and the environment would be intelligent enough to compute those dependencies as they occurred.

Of course the real world is a long way from perfect.

2009-09-02

Working on a new Audit plugin release

I'm working on a new release for the Grails Audit Logging Plugin that deals with issues some people were having with transactional databases.

I'd like to test this on multiple databases and with multiple configurations. So I'm putting the word out that I'd like some help with testing this. I don't have more than four databases to test on so I'd like some help.

If you can help out let me know. Contact me through this blog, twitter, or carrier pigeon. Thanks.

2009-08-21

Groovy Automatic type casting tricks...

I've been working on a Domain Specific Language. We wanted to hide the fact that you were passing string parameters to methods... so you could say things like:


something 'foo'
something foo


...and they would be equivalent. As the project progressed I needed to make calls like these...

something 'foo.bar'
something foo.bar

... again we want the two calls to be equivalent. But then I needed to reference the implicit tree of properties created by calling 'foo.bar' and 'foo.bar.baz' when I was interpreting these statements. So I needed to start tracking the symbols.

So my text-book meta-programming trick ...

def propertyMissing(String name) {
return name
}

... stopped working since foo.bar would result in a property missing on the String object. I could do some meta-programming on the String object but I also wanted to preserve the tree relationship for other work later. So with that in mind I wrote a class to wrap the symbols.

But there's a problem with that. If I introduce this new class... there are many places I just want the symbol like 'foo.bar.baz' and not the complex relationship. And by this point in the project I've written a lot of code to deal with things as strings.

Wouldn't it be great if I could ignore the complex nature of the object most of the time... but pick out the complex bits when I wanted them? To that end I created this class ComplexSymbol to do that work. I empowered it to know how to turn itself into a string silently in all sorts of situations.

So I wrote some tests to express what I wanted to see...

assert symbol == "foo"
assert symbol.bar == "foo.bar"
assert symbol.baz.bing == "foo.baz.bing"
assert symbol.baz.blat == "foo.baz.blat"
assert symbol.bar.blat == "foo.bar.blat"

... and I wanted to see the class of Symbol silently swap in for String whenever the method that was being called was expecting a string and not a complex symbol.

Take a look at the class I came up with to do this job...

public class ComplexSymbol {
ComplexSymbol parent
String symbol
Map symbols = [:]
ComplexSymbol(String str) { symbol = new String(str) }
ComplexSymbol(String str, ComplexSymbol other) {
symbol = new String(str);
parent = other
}
String toString() {
if(parent) {
return parent.toString() + "." + this.symbol
}
return symbol
}
def propertyMissing(String sym) {
if(symbols.containsKey(sym)) {
return symbols[sym]
}
def obj = new ComplexSymbol(sym,this)
symbols[sym] = obj
return obj
}
def asType(Class clazz) {
Object obj = null
switch(clazz) {
case java.lang.String:
obj = this.toString()
break
}
return obj
}

boolean equals(Object other) {
this.toString().equals(other.toString())
}

}


I've got lots of explicit typing in places to document the use of the class since I expect this class to show up in Java and Groovy code. The typing is really for documentation as it should be.

In the class where I want ComplexSymbol to stand in for naked strings I now do this:

def propertyMissing(String name) {
return new ComplexSymbol(name)
}

... which sets things up nicely so that the following works ...

someMethod foo.bar.baz

... and someMethod should get called with a string "foo.bar.baz" but before we get to that point I need to sanity check things.

Now to test the class with some methods... I just dumped the text for the above class into a file called symbols.groovy and stuck these methods at the end of the script so I could call them.


boolean stringify(String str) {
return str != null
}


This one should just see if it gets a string and I'll pass my ComplexSymbol to it and see if it gets the string.


boolean stringify(ComplexSymbol sym) {
return stringify(sym as String)
}


This method explicitly takes the complex symbol and uses the asType explict cast on it. This should show that the cast works.


boolean implicitlyCast(String sym) {
return sym != null
}


The implicit cast is my holy grail. If I can get the type system to see String whenever it asks for a String or the full ComplexSymbol type when it can handle that then I've got the whole enchilada.



/** Here's the tests again **/

def symbol = new ComplexSymbol("foo")
assert symbol == "foo"
assert symbol.bar == "foo.bar"
assert symbol.baz.bing == "foo.baz.bing"
assert symbol.baz.blat == "foo.baz.blat"
assert symbol.bar.blat == "foo.bar.blat"
assert stringify(symbol)

println "-" * 40
println symbol
println "-" * 40
println symbol as String
println "-" * 40
println symbol.toString()
println "-" * 40

assert implicitlyCast(symbol)


... command line output of this ...


$ groovy symbol.groovy
----------------------------------------
foo
----------------------------------------
foo
----------------------------------------
foo
----------------------------------------
Caught: groovy.lang.MissingMethodException: No signature of method: symbol.implicitlyCast() is applicable for argument types: (ComplexSymbol) values: [foo]
at symbol.run(symbol.groovy:71)


Notice that the last assert (my holy grail) fails. What I have is a very nice taco if not the whole enchilada.

Methods like print and println are going to implicitly handle the ComplexSymbol properly since they'll call the toString() on the object. And for methods that don't call ".toString()" we can either call it for them or explicitly cast to a String just before calling them.

All and all, it's a nice compromise with static and dynamic typing that allows me to use this class in plain Java code or in Groovy code very easily.

If someone has some thoughts on how I can get assert implicitlyCast(symbol) to work I'd love to hear them. With out it though, it's simple enough to spot the problem in a test case and put "as String" in the call like so...


assert implicitlyCast(symbol as String)


... and as I have incorporated this class into my work I've found points where I don't want the implicit cast to work all the time! I actually want to go into ComplexSymbol and ask for the symbol property directly. It turns out I actually need to hold the leaf nodes by their leaf node value sometimes and having them automatically resolve to their whole name would cause problems.

So in the end I'm glad the implicitlyCast test doesn't work the way I imagined.

2009-08-05

Grails in a J2EE world - RMI @ GroovyMag

In this month's GroovyMag I talk about using Grails code from a J2EE application using RMI. It's some practical work with the remoting plugin. Thanks again to Craig Wickesser for editing.

This is my last installment in the Grails in a J2EE world series. I had specifically been inspired to write these articles by several developers I had met over the last year that are interested in introducing Grails but who are also maintaining infrastructures that are built on EJB2. Yes, that's right... EJB2. I did not have time or space to cover every aspect that I wanted to, but, I hope that those of you that are in that particular boat and are interested in Grails can use these articles as a starting point for bringing Grails into your organizations.

Please head over to GroovyMag and check out the article. Feel free to give me some feedback below about the article. Together we can help cover this topic area better.

Thanks.

2009-06-04

Grails in a J2EE world - The Database @ GroovyMag

In this month's GroovyMag I talk about sharing a database between a J2EE application and a Grails application that is using GORM. If you haven't spent anytime working with concurrency in database applications you will really want to check this out. If you have a Container Managed Persistence (CMP) application that you are working with and you have a green-light on using Grails you might consider moving directly from EJB2 to GORM in a phased manner sharing parts of the database on the way.

Thanks again to Craig Wickesser for editing.

Please head over to GroovyMag and check out the article and give me some feedback.

EDIT
I'll also be giving out 5 free PDF copies of GroovyMag to the first five people who can Direct Message me on Twitter the answer to this riddle: What do you call an organized body of related information?

EDIT
That went fast! Congrats to @vintagedance, @evanbooth, @paulk_asert, @iansari, and @unixfudotnet

@unixfudotnet has offered his coupon up to the first person who wants it. Congrats to @dferrand.

Our answer? A database

2009-05-01

Grails in a J2EE world - The Web @GroovyMag

The new GroovyMag is available. This month I've made a contribution titled "Grails in a J2EE world - The Web" as I've been exploring ways that Grails and J2EE applications can interact I've found that the topic area is so rich that I can't possibly cover the whole thing in just one article. Instead I'll be making contributions in installments on the topic.

This particurlar article is inspired by the Kevin Kelly TED talk: Predicting the next 5,000 days of the web and how that has caused me to reevaluate my attitude toward software design and construction.

The article also covers some of the techniques I've found involving Java platform web technologies. I had a much longer narrative in mind but, honestly, I've come to enjoy some of writing of the authors who are much more succinct. I'm trying to emulate that style in my work. So for me writing involves one long session of pouring out followed by a session of cutting down. So in the end half of what I write gets deleted.

In addition, I had worked out a few more techniques that I could not fit in for time concerns. Every example code listing and technique comes from real working code. Often I have to address the issues of clarity and code ownership so I don't always get to share everything that I know.

I would like to thank my editors who helped with this month's article. Thanks to both Darryl Bloch and Craig Wickesser your feedback was very helpful and I really appreciate it. You both helped me tighten up my writing style quite a bit.

Please head over to GroovyMag and pick up a copy! As always your feed back is appreciated and will only go toward making me a better author and developer. Thanks!

2009-04-27

Grails 1.1, JBoss 4.2.x, and Oracle xmlparserv2.jar

If you want to use Grails 1.1 and the XMLType from oracle (the class is found in the Oracle jar xdb.jar) then you're going to need the Oracle xmlparserv2.jar in your Grails project. The problem is that Grails 1.1 will die if you just drop xdb.jar and xmlparserv2.jar into the lib directory of Grails. Why?

Well it turns out that this post tells you why. It was not cheap to find this answer at all... it cost me days of web surfing!

When you use JBoss in the $JBOSS_HOME/lib directory you'll find and endorsed sub-directory in there are two jars of particular interest to us. The xalan.jar and the xercesImpl.jar these provide the following services defined in their META-INF/services directories... xercesImpl.jar provides javax.xml.parsers.DocumentBuilderFactory and javax.xml.parsers.SAXParserFactory and the xalan.jar provides javax.xml.transform.TransformerFactory.

These three services are also supplied by the Oracle provided jar file xmlparserv2.jar and will effectively evict the JBoss provided implementations of these Java services. So what you have to do is replace the class references inside the xmlparserv2.jar with the same ones inside JBoss's endorsed libraries.

In short, explode xmlparserv2.jar then copy over the javax.xml.parsers.DocumentBuilderFactory and javax.xml.parsers.SAXParserFactory from xercesImpl.jar and the javax.xml.transform.TransformerFactory from the xalan.jar... re-jar the archive and redeploy.

2009-03-03

Grails in a J2EE world Part "zero" in GroovyMag

I've been very busy this year. I've been working with J2EE a lot lately. Normally you wouldn't necessarily be proud of working in J2EE but I'm also learning about a unique and thriving industry in my region. Namely the biotech and pharmaceutical industry which was very successfully colonized by J2EE earlier in this decade.

The result is that I've been working with ways to integrate Grails applications with J2EE applications. The techniques I've come up with aren't limited to just the integration of Grails and J2EE but I felt the title Grails in a J2EE world would probably help address the concerns that I've heard from developers working at large institutions such as banks, hospitals, and pharmaceutical companies like myself.

These companies who live in worlds dominated by J2EE can't just walk away from their investment. For people who live in worlds like that but want to add powerful new features found in Grails I've decided to write up a series of articles for publication in GroovyMag.

The techniques I'll discuss are applicable to far more than just J2EE and Grails integration, check it out.

2009-01-08

Data validation teirs

Data is either valid or invalid right? Well, sort of. I'm working with medical data like the Blood CHEM-7 test. This data has the possibility of being valid and being abnormal. So there is this concept of valid capture data and data that is out of range.

For example here's the CHEM-7 data definition in English:

  • BUN: 7 to 20 mg/dL

  • CO2 (carbon dioxide): 20 to 29 mmol/L

  • Creatinine: 0.8 to 1.4 mg/dL

  • Glucose: 64 to 128 mg/dL

  • Serum chloride: 101 to 111 mmol/L

  • Serum potassium: 3.7 to 5.2 mEq/L

  • Serum sodium: 136 to 144 mEq/L



I might code this as an Object and write validators to make sure that the saved data is in range. My Groovy object for GORM might look like this:

class Chem7 {
static belongsTo = [patient:Patient]
Integer bun
Integer co2
Float creatinine
Integer glucose
Integer serumChloride
Float serumPotassium
Integer serumSodium

static constraints = {
bun(nullable:false,range:7..20)
co2(nullable:false,range:20..29)
creatinine(nullable:false,range:0.8..1.4)
glucose(nullable:false,range:64..128)
serumChloride(nullable:false,range:101..111)
serumPotassium(nullable:false,range:3.7..5.2)
serumSodium(nullable:false,range:136..144)
}
}


... but I've got a problem. The very reason that we conduct the CHEM-7 blood test is to find irregularities. The constraints that I've expressed only detail the normal range of data for a CHEM-7... so what should I do?

If I make the table definition in the database very generic. Allowing the table to accept NULL values or numbers of any kind for each of these values then I can do this with a chem7 object:


chem7.save(validate:false)


... as long as the table underneath the Grails application can accept the data then I'll save my irregular chem7 test results. This would mean that anyone who was editing the chem7 would get a warning about how it was not a normal test result. Yet no matter how irregular the test was we could still save the results and log who entered them.

So imagine a "Forced Save" button next to the normal save button on your Grails application. Simple enough to write. But it allows us to use validation in a tiered fashion. Validation at the controller and validation at the database. The important thing to keep in mind is that the validation must be weakest at the database level.

2009-01-06

Don't Bother Me with the Details

When the framework essentially gets out of your way and lets you work with the level that you are interested in at that moment... that's a good framework.

If I'm working Ajax and UI I don't want to be bothered with security, servlets, persistence, or anything else. When I'm working persistence I don't want to be bugged by UI elements. And so on. The idea of "don't bother me with the details" applies to all levels and means different things at different levels. I think the Grails team really understood that (if not explicitly then implicitly).

Grails gets the wonderful benefit of living on a stack where people have sweat the details for years. Yet it still manages to hide those details when they don't matter to the problem you are solving. So Grails has the potential of living in the best of all worlds. And it gets this virtue by way of the Groovy Programming Language.

As Marc Palmer points out there are still holes in the pure play Groovy/Grails space. The Grails community still has a learning curve to get over: what should be pure play Groovy/Grails and what should be hybrid Java/Grails? How ever we answer that it should mean that the framework gets out of the way and lets the developer work directly on the problem at hand.

2009-01-01

New Year's Thoughts

If it takes ten years to get good at software development then I'm over due to get good by now I suppose. I am also going to have to take some calculated risks in the next year. So let's see what I was talking about last year and how things are shaping up so far.

Last year (a little in advance of the new year) I posted some thoughts I had been working on. In last year's "new year's thoughts" I said about software design:

Get to the core thought, the core idea, ignore as much minutiae as possible, get the problem solved. Keep the idea clean. Keep the core thought from being tangled in a mess of details that have nothing to do with the answer you are after.


... and I predicted that FP, Rules Engines, and DSL would become more important going forward ...

Functional programming, rules engines, and dynamic languages promise to free us from those mind-killing details that we don't really care about anymore as application developers. But they don't really address the need to create adaptive systems or systems that are able to span across computational mediums.


I had both identified the current trends and what I saw as the core problem. The fact of the matter is that the ideas behind the new fads in programming languages themselves won't really tackle the real problem in software engineering.

... so what is new in the last 365 days?

I recently found Kevin Kelly's the next 5,000 days TED talk. He has very keenly pointed out something that seems at once profound yet very simple. I can't imagine not having this idea to work with anymore.

If you want to really be relevant to the advance of technology today then you will position yourself to take advantage of Kevin Kelly's central point: There is only one machine and every screen in the world is a portal into that one machine. The reason the iPhone is a rampant success is that it really is a window into the one machine and acts like it.

The devices of the internet are approaching a complexity rivaling biology. To think we can understand them with our existing tools is a little foolish. We are yet to really develop the mental tools to deal with this new complex system. We will need these cognitive tools and we need them soon. Once we have them we will think it was odd they were so hard to conceive of. Much like the idea behind the number zero. It is hard to imagine how the first person conceived of the idea behind the number "zero" yet it is hard to imagine not having it.

This idea of "the one" is the guiding idea behind much of my own development efforts today. I didn't have a term for this before and I'm thankful to Kevin Kelly for coining a term for this idea. In many systems I confront there isn't an understanding of "the one" and data lives on barren islands of isolation.

In my new job I am very pleased to be allowed to participate in the development of interesting new technologies and I hope that I can find seemingly simple yet powerful changes to introduce that are like the idea of "zero" ... powerful, simple, and seemingly obvious once they are fully understood. That is the beauty of the truth when you have found it. It seems too simple and too obvious to have been missed.

There is a way to express UI, data, and logic that still is to be discovered. It will be obvious after we've found it. It will be profoundly different than what we have now. We won't recognize crossing into it until after we have already done so.

I expect that in 12 months we'll have better knowledge of which ideas are wrong and which ones more accurately reflect the emergent reality of the one machine. We will learn better the shape of the world it is carving out. That shape is not so much our invention anymore as it is a new undiscovered country which we are just now charting.