2008-03-28

Grails, two databases, and you

I'm working a project right now that requires me to access a legacy LAMP system and then import these into GORM/JPA persisted objects. Douglas Fils has some good notes on this topic in his post "grails datasource in resource.xml." So I'm not going to cover how to get access to a second database in detail since it doesn't really need covering.

I will point out that if you are thinking you'll run two GORM instances... persisting one group of domain objects to one database and persisting another set of object to a different database... well, I haven't figured that out yet. What I will talk about is one database connection that uses groovy.sql.Sql to create simple fast DAO.

What I've been doing lately is writing queries against my old database that "normalizes" the result sets into exactly the properties I have in my Grails domain classes. For example let's say we have a Person table on our legacy (non GORM) database that looks like this:

+------------------------+--------------+------+-----+-------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+--------------+------+-----+-------------------+-------+
| PersonID | int(11) | NO | PRI | 0 | |
| FirstName | varchar(255) | NO | | | |
| MIorMiddleName | varchar(255) | NO | | | |
| LastName | varchar(255) | NO | | | |
| EmailAddress | varchar(255) | NO | | | |
| Gender | varchar(255) | NO | | | |
| DataIDontCareAbout1 | varchar(255) | NO | | | |
| DataIDontCareAbout2 | varchar(255) | NO | | | |
| DataIDontCareAbout3 | varchar(255) | NO | | | |
| DataIDontCareAbout4 | varchar(255) | NO | | | |
+------------------------+--------------+------+-----+-------------------+-------+


Some of these columns we'll ignore... some we want to pull out and use in a new layout that better normalizes the data for our use. In my example to keep things simple let's say we decided to pull the name attributes out as a separate object and table. There are other cases that can result in rather complex pivot queries but I'll leave those for another time... suffice to say you can put a lot of work into the queries.

So in my MySQL queries I have something like this:

-- person query
SELECT PersonID AS 'id'
, EmailAddress AS 'email'
, SUBSTR(Gender,1,1) AS 'genderCode'
FROM PeopleContact
WHERE
EmailAddress = ?

... and ...

-- name query
SELECT FirstName AS 'firstName
, MIorMiddleName AS 'middleName'
, LastName AS 'lastName'
FROM PeopleContact
WHERE
PersonID = ?

These queries now match their result columns with the properties of the classes I have in my Grails domain ...

class Person {
Long id
Long version

Name name
String email
String genderCode
static constraints = {
name(nullable:true)
email(nullable:false,email:true)
genderCode(nullable:true,blank:true,inList:['M','F','U'])
}
}

... and ...

class Name {
Long id
Long version

static belongsTo = ['person':Person]

String firstName
String middleName
String lastName
}


Side NOTE: This particular normalization trick is interesting... I'm not necessarily recommending it... two people with the same name could end up pointing to the same name object. We would have to write our services & controllers to be sure that changing the name of one person did not change it for both... this is a point of debate with our system designers right now. However, we are doing something similar with mailing addresses which are currently in-line in several tables. Everyone agrees that normalizing addresses will make managing mailing addresses easier in our case.

To help with putting query data into objects I've written a mapping method that uses features of the GroovyRowResult class to our advantage:

/**
* A generic mapping utility assumes that the row
* has keys that map one-for-one onto the object.
*
* This is useful only if the queries are specially
* constructed so they can be mapped.
*/
public static void map(obj,row) {
row.keySet().each({ key ->

if( obj.properties.containsKey(key) ) {
obj.properties[key] = row.get(key)
}

})
}

...this is where working with groovy really shines!

Now that I've laid this ground work, in a service that has a groovy.sql.Sql object I've referred to as db. I've loaded the queries into strings and now I'll take the db object and pass it the queries and a parameter to go where the question mark is... in our example first we will query for a person based on their email...

def person = new Person()
def personRow = db.firstRow(personQuery,[email])
map(person,personRow)

... and for the name...

person.name = new Name()
def nameRow = db.firstRow(nameQuery,[personRow.id])
map(person.name,nameRow)


Now I've got a person object and a name object. To save them to the database now I just call the Gorm save commands on them... I can work with validation and I can work with all the other nifty Grails tools at hand. That means you could use the DAO snippets from a controller with a special "import" action that then forwarded to a "create" based action.

So we've seen code fragments that can be used to create an import DAO that can map rows from straight SQL queries onto GORM persisted objects. You can imagine that reversing the process would be very easy using the same tricks in reverse with update statements. That means hooking up a Grails project to import and export to a LAMP database can be very easy.

I'd love to know if anyone else is trying this and what their observations are or if they have better ideas.

2008-03-27

Grails and Quartz

When an action takes longer than about 10 seconds to perform web developers have a problem. Basically you can't expect a user to sit around for 15 or 20 minutes while an action completes. In LAMP and RoR land we solve this problem by using a cronjob or messaging a daemon... in Grails we have something better Quartz.

The Grails Quartz plugin may have a low version number but it's no slouch. The plugin works well and is easy to install. Once installed you can add Quartz jobs to the grails-app/jobs directory. And that's one reason Quartz is better than cron.

When creating a cronjob the job is separate from the project code. When creating a Quartz Job the quartz job's code is embedded inside the project that it supports. That's an advantage because it keeps the project together.

The way you schedule Quartz jobs in Grails has other advantages over cronjobs too. For example you can set a Quartz job to run every 5 seconds if you want. That's something you can't really do well in cron.

When working with cronjobs sometimes you have to instantiate separate connections to the database, file system, and other networking resources. That can occasionally mean you have to build up a separate instance of part of your framework just for your cronjob.

When working with Quartz jobs I will typically create a service to do the work. Quartz jobs in Grails can get at services by IoC conventions... which means you don't have to figure out how to wire up the Quartz job and the service. Instead just use the Grails naming conventions for example:


class MyJob {
MyService myService
def timeout = 5000l

def execute() {
myService.myMethod()
}
}


... would be automatically wired up to an instance of the MyService class and would fire the execute method every five seconds. Working this way we don't have to worry about other problems we can have with cronjobs. For example, what if a job takes too long and a second instance of the cronjob fires while the first is still running? Many developers simply ignore this possibility and create really long delays between their jobs. We don't even have to worry about it working with Quartz and a service. If we mark MyService as transactional it will thread-lock keeping two copies from running at once.

For example:

class MyService {
boolean transactional = true
.... code ....


That transactional mark at the top of the class tells Grails to allow only one copy of this service to run at a time. It means a potentially hairy problem which could involve creating semaphores and locks can be completely avoided.

Coupled with the advantages of scheduling inside the job itself, keeping related code packaged together, and running jobs in the same framework as the services they use Grails + Quartz solves the problems of working with long running tasks in a browser window much better than just a cronjob.

2008-03-24

Grails, GORM, Domain Classes and Enum

I'm currently building a domain model in Grails that uses Enum heavily. This is a tad bit of a problem since Grails doesn't support Enum directly. See GRAILS-2061 for more information. Now then, I have found this Enum WorkAround and I wasn't quite happy with that. I've come up with my own work-around that fits what I want to see on my database better.

I came up with this work-around based on my experiences of using an existing database with Grails. When using an existing database the GORM can treat any Enum on your database as a string. Next if you are manually building up the domain from this database model you would want to constrain your faux string by the values of the Enum as expressed in the database. When working from an existing table I would simply cut and paste the values into the inList constraint for the domain class. But what if you have an existing Enum class?

For a really simple example let's consider:


public enum GenderCode {

M,
F;

public String value() {
return name();
}

public static GenderCode fromValue(String v) {
return valueOf(v);
}

}


This is a really simple enumerated type that represents the "M" and "F" codes that many systems use to indicate Male or Female. Now we would probably use this class like this:

class Person {
String name
GenderCode gender
}

... but Grails will not know what to do with the enumerated type. So we can "work around" this limitation by using our good friend constraints... like so:

class Person {
String name
String gender
static constraints = {
name(nullable:false)
gender(inList:GenderCode.getList())
}
}

... only problem... how do we get the GenderCode enumerated type (a straight java class by the way) to give us a List object to use in the inList constraint?

It's not too hard...

public enum GenderCode {

M,
F;

private static List list;

public String value() {
return name();
}

public static List getList() {
if(list != null) {
return list;
}
return buildList();
}

private static synchronized List buildList() {
// List was not initialized...
list = new ArrayList();
for (GenderCode c: GenderCode.values()) {
list.add(c.name());
}
return list;
}

public static GenderCode fromValue(String v) {
return valueOf(v);
}

}

You might notice I chose to use the synchronized key word on the buildList method. Also note that I store the Enumerated types in the grails project under src/java. The domain classes store only strings.

So now when I go to generate scaffolding code for the Person class I'll get a GSP that will use a drop-down list of values from the enumerated type. If I alter the enumerated type to include the value "U" in the enumerated type there is no need to re-generate any of the views... re-populate a database... or update a boot strap script.

If you want the Enum type on your database itself now your DBA can alter that column on that particular table to be of type Enum and coordinate with you what values are in the list. GORM won't complain. And, your DBA will be happy to see true enumerations in their database which are both human-readable and space efficient. Did I also mention that they cut down on the number of joins to lookup tables making queries in all application spaces faster?

What do you think?

2008-03-17

Working with Grails 1.0.1 inside Eclipse

I've started on a new project using Grails 1.0.1 and I've had to change a few things since working with Grails 1.0-RC4. Fortunately I did get an upgrade to Grails 1.0 into my application before deploy but 1.0.1 came too late to make it through our regression testing on that application. Today starts a new development project from scratch so I'm using 1.0.1 and when I went to use grails create-app it crashed on me... sorry I wish I had saved the trace... but I fixed this issue by deleting the ~/workspace/.metadata directory from my workspace.

Unfortunately, when you delete the .metadata directory Eclipse forgets many of your settings. In setting up my new eclipse project for Grails I set up the Eclipse Groovy plugin to output to bin and I set the build path of Java to output to project/bin which made both Groovy and Java compilers output to the same directory. This seems to work well with JPA classes too. After I get farther with this project I'll write up yet another Groovy, Grails, and the JPA tutorial using the new 1.0.1 conventions.

This next project will make heavy use of XML and XSD files as well so this should be a very informative project. Another problem to solve will be using SSL certificates and CAS. The CAS work has been very easy thanks to the CAS Client Plugin for Grails (it works great if you already have CAS set up). Now I need to add role management... but that's another story.

2008-03-15

... with the Eclipse RCP

I've been working with Eclipse RCP and the SWT on a project lately.

I started writing software in C/C++ about a decade ago. I remember working with C programs and UI back then as state machines. Essentially the UI would set states that a processing thread would later act upon. If you think about it this was an early form of Model View Controller separation.

I understood this type of separation well because the main process and the UI process essentially communicated through a set of state variables. If you were smart you nailed down these state variables and kept them in a central location and made them easy to understand.

I actually worked this way for most of my career... except for a few years back around 2000 when I joined the Internet hordes... and except for the last 4 years when I started working on Ajax applications (well, we didn't call them Ajax applications until recently).

I was pretty well familiar with Ajax by the time an opportunity to work with SWT came around. I had one project in Swing in the fall of 2001 but since then Java UIs weren't in the picture for me. I'll tell you EJB3 was a welcome change from the J2EE work I had done back in 2001 but that's another post entirely. Let me focus on the SWT in this post.

When I was given the chance to take apart and enhance an Eclipse RCP project I saw it as a golden opportunity to gain familiarity again with Java desktop applications. That's not an area that I personally get to work with much. Most of my time is spent working with server-side and database projects... ironically my university emphasis was on 3D graphics in OpenGL... so a chance to work with graphics again was really welcome.

The SWT and Eclipse are not kind to new comers though. In particular you need to be careful of "kitchen sinking" your Eclipse IDE when you go to develop on it. The rule for how to keep Eclipse from getting "kitchen sunk" seems to be to keep the plugin base as minimal as possible. A bit of a pain especially since the Eclipse plugin manager will let you break Eclipse out-right by letting you get into a configuration that "does not contain the platform" (whatever that means) so that you have a nicely dead eclipse.

I hadn't realized that I had been lucky with eclipse up to this point. Most of my projects being server side didn't need advanced Eclipse features such as the GEF. Apparently if you try to use VE, GEF, jface, and eclipse.ui in the same project you're going to be doing some juggling.

Soon after I launched into this project I decided I needed training material since I wasn't intuitively picking up on SWT. The Visual Editor (VE) works with both SWT and Swing components but that's no consolation if you don't know either very well. It still isn't easy to make progress.

And this is where I think I'm starting to change my mind about a few things. I'm sure you've seen the GWT (that's the Google Web Toolkit if you didn't know) and it's basically a Swing/SWT type of web programming UI library. This is great if you like that kind of UI work... but really... I don't think I do.

I think I actually like Ajax and that mess of HTML and I think I like it because it's so easy to see what a UI element is going to look like. I think the HTML forces such a clean break between the controller and the view that it's actually beneficial. It's a disconcerting view for me to entertain because I actually come from a C/C++ background and I would probably prefer a more rigorous UI development environment.

In my utopia VRML would actually have lead somewhere instead of dying out. The concept behind a VRML interface and a web page is that presentation should be scripted. The idea of a mark up language hinting at a layout for a render engine is some what radical by itself but once you accept the idea ... you have to wonder why not use it everywhere?

I've actually had to write postscript drivers and code generators for postscript code. Postscript is actually a remarkable display oriented language. It is also far from a mark up language looking more like a custom graphics DSL than anything. So it's no shock to me that it may have taken two decades for the industry to accept the idea of markup for describing how to display things.

Consider that HTML and VRML are concerned with general descriptions of elements and leave the particulars of display up to the render engine. And this is the liberating feature of the markup language you get to care about only the particulars you want to. That frees the designer to paint with a broad brush and can separate the design and implementation functions very nicely.

Certainly the same discipline can be applied to work done with SWT, GWT, and Swing but that requires the effort and attention of the programmer. The web interface and its restrictions actually force the issue of the separation of design and function.

*sigh* but, design and function aren't actually separate are they... design dictates what a function must do and a function is only comprehended through the design of the interface to the function. A chair must have the shape of a chair... a cup must be a cup. Handles and legs and decoration can be added without too much troubling the core purpose of the artifact but ... really the design and function are one and the same.

So what am I saying about the SWT, GWT, Swing, or any other UI design API for that matter? I guess I'm trying to get at this idea that you need to be able to describe the UI in graphical terms. Component UI toolkits procedurally describe how a UI is built up in steps... and it's still procedural even if the procedures are encapsulated in objects.

Consider in all the UI toolkits I know (other than the Ajax ones) you build a menu object, build a menu item object, give the menu item object a code, enum, or pointer to a method to call, and then inject it into the menu object at menu initialization. This is very procedural and has virtually nothing to do with what a menu is graphically.

Now consider an Ajax design: You mark up a set of blocks to hold the menu items and apply style guides to them and then hook the onclick event to the appropriate action or link. The trick here is we are describing visual elements and the only thing that makes anything a menu is how it is used. That's a subtle but important paradigm shift.

It's subtle because a menu element and a menu object aren't conceptually that different. It's important because a menu element is really just a display block primitive with some style guides and a menu object is a much heavier complex and rich thing. But in the end they both do the same thing. It's just that one way is describing a thing that happens to get shown in a particular way graphically and the other is describing a graphical thing that happens to get used in a particular way programmatically.

The markups are interested in describing visual elements given a handful of visual primitives but the toolkits are focused on the aggregation of visual elements in a prescribed taxonomic hierarchy. That means that the toolkits are actually in danger of creating an impedance mismatch akin to the famous object-relational impedance mismatch that we've fought in OOP and RDBMS circles for the last decade or so.

I wonder if we won't see talk of a UI-Object impedence mismatch or other metaphorical terminology for the mismatch between UI and UX expressions and the creation of component based taxonomic object hierarchies that pervade the practice of OOP. This is an especially ironic argument to make since OOP was created to better match a graphical environment. It's actually almost silly to make the argument against using OOP for GUI work... but here I am entertaining the notion.

I am looking to technologies like ZUML in the Zk Framework to start showing us if markup-componentized hybrid frameworks are actually feasible in a non-proprietary system. For now it's simply not practical for a "main stream" application to entertain the use of a Zk based design. Unless you and your users are ready to surf the cutting edge you will have to ride along with the rest of us and our component based frameworks.

Or you could go with Ajax.

For now I've decided to continue slogging through my work with SWT and Eclipse but it's really because I've got too much into this particular direction to back out now. I will say I have high hopes for the GroovySWT project and I've already sampled the Groovy Swing project. These are very nifty projects that promise the creation of a builder or markup-like way to express GUIs in both Swing and SWT. Hopefully both projects can find ways to "macro" or simplify UI building in the process of providing a new way to rapidly build up UI. The SWT builder already provides a very intuitive container syntax that eliminates some of that counter-intuitive procedural buildup code you find in Eclipse RCP Perspective and Activator objects.

I for one welcome our new groovy-builder overlords.