Showing posts with label mysql. Show all posts
Showing posts with label mysql. Show all posts

2008-03-28

Grails, two databases, and you

I'm working a project right now that requires me to access a legacy LAMP system and then import these into GORM/JPA persisted objects. Douglas Fils has some good notes on this topic in his post "grails datasource in resource.xml." So I'm not going to cover how to get access to a second database in detail since it doesn't really need covering.

I will point out that if you are thinking you'll run two GORM instances... persisting one group of domain objects to one database and persisting another set of object to a different database... well, I haven't figured that out yet. What I will talk about is one database connection that uses groovy.sql.Sql to create simple fast DAO.

What I've been doing lately is writing queries against my old database that "normalizes" the result sets into exactly the properties I have in my Grails domain classes. For example let's say we have a Person table on our legacy (non GORM) database that looks like this:

+------------------------+--------------+------+-----+-------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+--------------+------+-----+-------------------+-------+
| PersonID | int(11) | NO | PRI | 0 | |
| FirstName | varchar(255) | NO | | | |
| MIorMiddleName | varchar(255) | NO | | | |
| LastName | varchar(255) | NO | | | |
| EmailAddress | varchar(255) | NO | | | |
| Gender | varchar(255) | NO | | | |
| DataIDontCareAbout1 | varchar(255) | NO | | | |
| DataIDontCareAbout2 | varchar(255) | NO | | | |
| DataIDontCareAbout3 | varchar(255) | NO | | | |
| DataIDontCareAbout4 | varchar(255) | NO | | | |
+------------------------+--------------+------+-----+-------------------+-------+


Some of these columns we'll ignore... some we want to pull out and use in a new layout that better normalizes the data for our use. In my example to keep things simple let's say we decided to pull the name attributes out as a separate object and table. There are other cases that can result in rather complex pivot queries but I'll leave those for another time... suffice to say you can put a lot of work into the queries.

So in my MySQL queries I have something like this:

-- person query
SELECT PersonID AS 'id'
, EmailAddress AS 'email'
, SUBSTR(Gender,1,1) AS 'genderCode'
FROM PeopleContact
WHERE
EmailAddress = ?

... and ...

-- name query
SELECT FirstName AS 'firstName
, MIorMiddleName AS 'middleName'
, LastName AS 'lastName'
FROM PeopleContact
WHERE
PersonID = ?

These queries now match their result columns with the properties of the classes I have in my Grails domain ...

class Person {
Long id
Long version

Name name
String email
String genderCode
static constraints = {
name(nullable:true)
email(nullable:false,email:true)
genderCode(nullable:true,blank:true,inList:['M','F','U'])
}
}

... and ...

class Name {
Long id
Long version

static belongsTo = ['person':Person]

String firstName
String middleName
String lastName
}


Side NOTE: This particular normalization trick is interesting... I'm not necessarily recommending it... two people with the same name could end up pointing to the same name object. We would have to write our services & controllers to be sure that changing the name of one person did not change it for both... this is a point of debate with our system designers right now. However, we are doing something similar with mailing addresses which are currently in-line in several tables. Everyone agrees that normalizing addresses will make managing mailing addresses easier in our case.

To help with putting query data into objects I've written a mapping method that uses features of the GroovyRowResult class to our advantage:

/**
* A generic mapping utility assumes that the row
* has keys that map one-for-one onto the object.
*
* This is useful only if the queries are specially
* constructed so they can be mapped.
*/
public static void map(obj,row) {
row.keySet().each({ key ->

if( obj.properties.containsKey(key) ) {
obj.properties[key] = row.get(key)
}

})
}

...this is where working with groovy really shines!

Now that I've laid this ground work, in a service that has a groovy.sql.Sql object I've referred to as db. I've loaded the queries into strings and now I'll take the db object and pass it the queries and a parameter to go where the question mark is... in our example first we will query for a person based on their email...

def person = new Person()
def personRow = db.firstRow(personQuery,[email])
map(person,personRow)

... and for the name...

person.name = new Name()
def nameRow = db.firstRow(nameQuery,[personRow.id])
map(person.name,nameRow)


Now I've got a person object and a name object. To save them to the database now I just call the Gorm save commands on them... I can work with validation and I can work with all the other nifty Grails tools at hand. That means you could use the DAO snippets from a controller with a special "import" action that then forwarded to a "create" based action.

So we've seen code fragments that can be used to create an import DAO that can map rows from straight SQL queries onto GORM persisted objects. You can imagine that reversing the process would be very easy using the same tricks in reverse with update statements. That means hooking up a Grails project to import and export to a LAMP database can be very easy.

I'd love to know if anyone else is trying this and what their observations are or if they have better ideas.

2007-11-16

Magical MySQL URL parameters Save my Groovy Code

I've been working with legacy databases in MySQL and basically forbidden from using hibernate by a bug I had with all zero dates of the format '0000-00-00' which is perfectly legal in MySQL but completely disallowed in JDBC.
Stack trace for this problem contains:

SQL Exception: Value '0000-00-00' can not be represented as java.sql.Date

The answer? Apparently there is a hidden setting to use...

dataSource {
driverClassName = "com.mysql.jdbc.Driver"
dialect= org.hibernate.dialect.MySQLMyISAMDialect
url = "jdbc:mysql://localhost:3306/legacy?zeroDateTimeBehavior=convertToNull"
username = "legacyUser"
password = "legacyPassword"
}


That little flag on the URL is absolutely magical and now I can map my Grails objects onto a legacy database. Isn't life grand?

2007-11-14

Groovy Grails and the JPA

If you are using Grails 0.6 or later chances are the tutorials you've found for making EJB3 persistence annotated POJOs work with Grails don't match up exactly. If you're like me, probably need to be made aware of a few differences in configuration. They aren't big... but the little differences could trip you up.

First note my version of myApp/grails-app/conf/DataSource.groovy

import org.codehaus.groovy.grails.orm.hibernate.cfg.GrailsAnnotationConfiguration
dataSource {
configClass = GrailsAnnotationConfiguration.class
pooled = true
dbCreate = "create-drop"
driverClassName = "com.mysql.jdbc.Driver"
dialect= org.hibernate.dialect.MySQLInnoDBDialect
url = "jdbc:mysql://localhost:3306/grails"
username = "grails"
password = "grails"
}

Is not the same. Just because configClass = GrailsAnnotationConfiguration.class is set and your IDE says the class is found in this context doesn't mean that it's working for you... that import above still has to happen...

Next, I have added a file under "grails-app/conf/hibernate/hibernate.cfg.xml" and this part goes pretty much like the tutorials from last year... you register each class here one at a time. I need to research this one but I think grails mixes in this hibernate.cfg.xml with another one at run time so just stick to registering your classes. I tried to get fancy here and it didn't work out too well. My working hibernate.cfg.xml looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration SYSTEM
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<mapping package="com.mycompany.shared.domain" />
<mapping class="com.mycompany.shared.domain.Subject" />
</session-factory>
</hibernate-configuration>

You'll need your Java files to be in "src/java" and annotated up. I will be trying to make a shared JAR file full of only the shared domain objects for the various projects that need them. I don't know how well this is going to work but my idea is to create one set of JPA annotated EJB3 Entity classes and put them in a JAR and share them between various Spring applications. I don't know if that's a good idea or not...

Finally, you'll need to get the compiled class files for these annotated POJOs into "web-app/WEB-INF/classes" that means you need to do a "grails run-app" or similar first so that the Java files get compiled into classes and dropped into the "web-app/WEB-INF/classes" directory.

Once that is done you can run the "grails generate-all" command on your Java classes.

If you're interested in working through the EJB3 examples I have posted ejb3_grails.zip which uses this tutorial by Jason Rudolph but also uses Grails version 1.0_RC1 and the Groovy Eclipse Plugin.

EDIT: in version 1.0 and higher recent experience shows that if you set the Eclipse Groovy plugin to output to bin and set the build path of the Java compiler to output to project/bin class resolution will work fine for the IDE. When Grails itself goes to run the application or build the WAR file, the right class files will make it into "WEB-INF/classes" since Grails uses a separate class cache anyhow. It seems the reason I advised to output Groovy and Java classes to "web-app/WEB-INF/classes" is no longer true.

2007-07-05

The Woe of MySQL and java.sql.Date

The class java.sql.Date cannot hold a date of "0000-00-00" (ISO date format YYYY-MM-DD) because that is simply NOT a valid date. Our legacy Perl application uses the string "0000-00-00" as a date for any uninitialized date in the system. That is because the original system designer forbade the use of NULL for dates. This becomes a problem when working with Perl legacy dates in Java. Specifically the JDBC drivers will throw exceptions when attempting to work with a date that has the value "0000-00-00" and exception handlers for these exceptions are not prompted for by the Eclipse IDE.

I have developed a few strategies for dealing with this date woe problem.
On the issue of selecting "0000-00-00":

Using the NULLIF mysql function inside select statements means that you must map out every single select column returned and catch every single date so that it gets set to NULL for the JDBC drivers...

NULLIF(Candidate.CertificationDate, '0000-00-00') AS certificationDate

On the issue of saving "0000-00-00":

Since you can't create a date of "0000-00-00" you'll need to leave the date null but you'll have a problem setting the column NULL in MySQL since the column will be restricted to prevent your insert of a NULL value. Here's what I have done:

try {
ps.setDate(107, new java.sql.Date(a.getCertificationDate1().getTime()));
} catch (NullPointerException npe) {
ps.setString(107, "0000-00-00");
}


Note: The IDE doesn't "know" about the NullPointerException? since it can be thrown from every single line of code in every single java program ever written and handling it on every single line of code would get a little silly.

The Date stored in object a may be null. If it is we use the feature of MySQL that will take the string "0000-00-00" and map it into a date.
FAQ

1. Q: Why not change the database? A: Perl programs look for the string "0000-00-00" to be returned indicating a NULL date (an uninitialized date). You will have to rewrite all these Perl programs.
2. Q: Why not work with dates in Java as strings? A: Java's date parsers are deprecated and you would have to perform custom parse and casts on these date strings since they do not match the Java date formats... if you wanted to work with the dates as dates. In other words: you would have to write your own date handling.
3. Q: Why not toString all dates for the database? A: Java and MySQL use different native date string formats... and we still would have to make special cases for "0000-00-00" any way.
4. Q: Why not write a custom Perl2Java Date class? A: Your company would have to dedicate a small portion of staff time to monitoring for changes in ISO date formats, time zones, DST, and watch for potential date bugs.

Other possible solutions

1. Write a helper utility class that would manage the casts taking all dates to MySQL strings. (only helps saves, can't help loads)
2. "fix" all the Perl programs and write helpers in Perl to handle the cases where the Perl programs need "0000-00-00" then set all null dates to null.
3. don't fix this and wait for the legacy Perl to age out.

NOTE: I found the answer to this problem here.