On the topic of Grails Audit Logging Plugin version 0.5.4

Last night, I released a new version of my Audit Logging Plugin. Release notes are here. There had been an outstanding compatibility issue with Grails 1.3.x based on JUnit 4's Integration tests. The bug documented here has been open for several months as I haven't had time to come back to the project to work on it.

Normally I don't blog about tickets but I figure this one requires some attention because I've run into a few plugins that I need in my day job that are "broken" in 1.3.x and I'm not sure who else is having these troubles. In the interest of helping the community as a whole I want to cover my experience.

It turns out that GRAILSPLUGINS-2243 (titled "Integration tests with auditable domain class do not observe test transaction boundary") is actually a more accurate description of the actual bug. The symptom is during integration testing you can't delete entities that are audited. If you take the same code and run it in development or production mode you can delete the objects.

The issue with my code is in this block... (extraneous logging and debugging code removed)

Session session = sessionFactory.openSession()
def trans = session.beginTransaction()
def saved = session.merge(auditLogObject)
session.flush()
trans.commit()
session.close()

Keep in mind this block functions exactly as expected in Grails 1.2 under all environments I know of. The one environment this code is horribly wrong is Grails 1.3.x in Integration testing. Only at that time does this code cause a problem.

So let me ask you... what's wrong with this code?

Before addressing the real issue, questions I anticipate:

1. Why are you opening your own session?

   Because on a transactional database in some cases the audit log must fire after the database insert so that I can log the automatically assigned object Id. This means you can't use the open session without invalidating it.



2. Why are you using merge and not save?

   Because there are edge cases where the AuditLog object will be saved twice and I don't want to dictate the logic for that in this block to keep it simple



3. Why are you flushing the session?

   Because we want to make sure the audit log is being written outside the regular transaction cycle and this particular session is going to perform no other job we want to ensure the log is written out to the database

So ... what is wrong with this code again?

def trans = session.beginTransaction()

Apparently, under integration testing in Grails 1.3.x if you start a transaction in the way I have you will cause a problem with the other independent transactions in the rest of the application. This is simply not the behavior I see in my production systems under Grails 1.2 or in my staging systems running Grails 1.3 for that matter.

My solution is a bit of a work around. In the application context initialization for the Audit Logging plugin (the bit that runs just as you start up your grails application I have a block that does this...

if( Environment.getCurrent() != Environment.PRODUCTION && ConfigurationHolder.config.auditLog?.transactional == null ) {
       transactional = false
}

... which simply says in environments that are not production (Notice I'm using the Grails 1.1 API to find that breaking compatibility with Grails 1.0) the boolean flag "transactional" is set to false. I then wrapped the begin and commit calls to the transaction around a check on this boolean. For good measure, I gave the end developer the ability to override this behavior using the setting "auditLog.transactional" in their Config.groovy file. (It would figure that I would release this as a feature and someone needs to use those transactions in their integration testing because they run their testing environment against Oracle or something so here's an out for you to get the old behaviors)

What took so much time to find this bug? It's not clear cut basically. I stumbled on this answer by blindly trying different things until I found a combination that worked. Then I tried to build a theory around this result that fit the facts that I had on hand.

My theory is the JUnit 4 integration environment does something to the session factory so it does not really produce new sessions such that the session that I get in my plugin's listener code is not really a separate session or transaction. I call commit on this transaction and when the session is returned to the normal GORM code it is already committed so the actual entity delete call can't happen and is swallowed.

What is disturbing about this is I didn't invent this technique for Audit Logging I'm implementing here. I created this code after being educated by the Hibernate wiki. This means if my theory holds it's not just a Grails application that will observe this strange-ness ... pretty much any Hibernate application with this kind of Audit Logging will be affected when integration tested under JUnit 4.

Beyond just this anyone developing applications that open their own sessions independent of the GORM session may potentially see bugs during integration testing that do not manifest in other environments. That can be a very difficult situation to deal with. So I've over documented things here for the benefit of the community at large.

In other news...

In other places I've noticed a change in Groovy 1.6 and 1.7's builder API. In particular I use RichUI in one project which makes calls like...

builder.yield(" var ${attrs.yuiVariableName}DataSource = new YAHOO.util.XHRDataSource(\"${attrs?.action}\");\n", false)

... under Groovy 1.6 yeild was a builder method.

But, under Groovy 1.7 this method is not present. That means the HTML rendered by code that used the yield and yieldUnescaped calls now create tags <yield> and <yieldUnescaped> which breaks some pages (but not all).

So far these are the only surprising issues I've run into that has kept Grails 1.3.x off of my production servers. I haven't yet seen and great benefit to 1.3 so I can't say I'm deeply compelled to move to the new version.

I am looking to rewrite the plugin in light of GRAILS-5725 but this will be a breaking change. Frankly, it's such a large change I'm surprised it was not pushed to Grails 1.4 ... however, it does not appear to interfere with the operation of plugins using the listener injection technique I am also using so no harm no foul.

I see I've lost a whole star in my plugin ratings so, my apologies to my end users who have not had a plugin that works with your Grails 1.3 integration tests. I sincerely hope I can earn back that extra star from you. Thanks for your support and help. I am reviewing code submissions for another version of the plugin coming up and I greatly appreciate your support.

TriDroid: QRCode, Linkulator & other "move the data" ideas

At tonight's TriDroid we discussed QRCodes and other techniques for moving data between the environment and our Android phones. Here are hastily-hand-drawn my slides for anyone interested in them.



Other things we talked about:

• Chess Data format based on Algebraic Chess Notation


• 3D Barcodes for manufacturing


• 4D barcodes (time phased codes for higher data density)


• color barcodes where color is used as a new "dimension" to store data in.

EDIT:

As George points out (and many have pointed this out repeatedly) the Google Chart API has had QRCodes incorporated in it for some time. Once again, being lazy would have paid off much more than working hard. A lesson the universe seems to be trying to teach me repeatedly.

App Inventor

If you haven't heard about it App Inventor is available to educators wanting to teach students about programming. It allows novice programmers to create Android applications using a graphical programming environment similar to Scratch.

The educational perspective that motivates App Inventor holds that programming can be a vehicle for engaging powerful ideas through active learning. As such, it is part of an ongoing movement in computers and education that began with the work of Seymour Papert and the MIT Logo Group in the 1960s.

This is probably one of the more interesting and exciting ways to engage lay-programmers in software creation. It is possible to view the graphical programming language blocks as a Domain Specific Language but I'm not clear on whether this is actually the case or not. In particular:

1. the types of statements are limited to available blocks


2. these statement blocks are limited to guided classifications specific to phone app development


3. user and device interactions are limited to the palette

In these ways the "developer" is "on-rails" (in the video game sense) limited to a certain type of application. This kind of limitation is beneficial in some cases guiding the developer toward a "good" application. In this case the limitation keeps the developers from heading down paths that require knowledge about the halting problem or multi-threading. For this class of application... that's a good thing.

The OpenBlock abstract certainly sounds like a description of a Domain Specific Language (emphasis added):

Graphical programming systems have been built to lower the threshold to programming for beginners. However, because these systems were designed to make programming more accessible to novices, they were developed with narrower intentions for their users and applications. For example, in StarLogo TNG, a graphical block programming environment, users may create games and simulations, but they cannot use this same system to create programs that can automate their computer processes, like the text-based scripting system AppleScript. Application developers can create their own programming systems, but doing so can take a significant amount of time to design and implement. This thesis describes an extendable framework called OpenBlocks that enables application developers to build and iterate their own graphical block programming systems by specifying a single XML file. Application developers can focus more on the design of their systems instead of oil the details of implementation. The design and implementation of OpenBlocks are described, along with a user study conducted to test its usability and extendability.

On the other hand it could be argued that the graphical programming environment is just an immature environment and is only limited because of how new the tool set is. There is no intrinsic reason a graphic block based programming language couldn't eventually become as rich as a traditional text based language. The real differentiator between a (DSL) Domain Specific Language and a GPL (General Purpose Language) is that the scope of the programming done.

You evolve a DSL either by creating a very rich API that "begins to talk" or you create a deliberately limited subset of your GPL that can be exposed and interpreted using limited API.

So is App Inventor using a new(-ish) Domain Specific Language or a new way to wrap a General Purpose Language?

Three Small Ideas

I was glad to be interviewed by DZone a while ago. The second interview was on my current Open Source work and I gave what is probably a non-traditional answer on why Open Source contribution is important for a developer. The google appengine project I'm talking about is the qrcode linkulator. And, for the record, I've got a better hair-cut now.



My statistics are from this article on Functional Fixedness at wikipedia. The over-lapping concept I'm trying to tease out is that there are problem solving exercises that are not served by tightly focused environments like business. As a developer you need to be able to work in spaces that allow for creative freedom and the lessons you learn there (for better or worse) translate back into your "high pressure" environment. The result is a better developer that is able to think outside the parameters of the problem.

A nice animation covering the idea popularized by Daniel Pink (who is an infinitely better speaker) is on youtube:


My contribution is: not only is open source a product of intrinsic motivation... it creates intrinsic motivation to improve yourself as a developer. These improvements translate back to your "day job" ... so for purely selfish reasons you *should* create open source code. You will be forced to learn. You'll get interaction with other developers. You'll learn much more than if you stay locked in your office and never get challenged by people who never would think like you do and have no reason to pretend they do.

Even if the project is a failure and your contributions are insignificant... you'll learn something.

Grails + Acegi + CAS + Proxy Ticket

Should you have Acegi + CAS setup in Grails and you want to proxy a URL secured behind the same CAS instance you should do something like this:

    URL url = new URL(urlString)
    SecurityContext ctx = SecurityContextHolder.getContext();
    CasAuthenticationToken auth = ctx?.getAuthentication();
    Assertion assertion = auth?.getAssertion();
    Principal principal = assertion?.getPrincipal();
    assert principal.proxyRetriever != null 
   // the above line asserts that proxy system is properly setup and working
    String proxyTicket = principal?.getProxyTicketFor(url?.toString());
    assert proxyTicket != null
    URL readUrl = new URL(url.toString() + "?ticket=${proxyTicket}")

This only works if your CAS supports proxying and you have a working proxy call back. For details on troubleshooting this with Grails-Acegi 0.5.3 see GRAILSPLUGINS-2231 which includes a working proxy callback controller. Hopefully this will be fixed in an upcoming version so you won't need this reference for long.

If your setup is not working properly the proxyRetriever embedded in the Principal object will be null. If it is null then getProxyTicketFor returns null no matter what. Once you get your CAS and SecurityConfig.groovy working properly the proxyRetriever starts getting magically injected into your application. This doesn't mean you actually have proxying working yet but you can almost rule out client-side configuration errors.

Your Grails application should also have something at /secure/receptor that can catch responses issued by the CAS server. You should see this URI respond to something just after you successfully log in. That URI also needs to be open so the CAS server can post there without needing to login to itself.

NOTE: your CAS server must be able to resolve the URL of the proxy call back. That means if you are running your proxy callback on localhost the responding CAS server must be able to post to that URL. Why? When you request a proxy granting ticket CAS will send the ticket back to your Grails application at the URI /secure/receptor by default. That means your application's URL must be resolvable by the CAS server for proxy tickets to function.

In practice, I run a small CAS server on localhost at an obscure port with the URL to proxy also on localhost when testing proxy ticket granting and other CAS secured proxy services. I work around the SSL issues by using a custom implementation of javax.net.ssl.HostnameVerifier that is installed during bootstrap in development and test-app modes only. The custom verifier (written in Java) looks like:

import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLSession;
import javax.net.ssl.HttpsURLConnection;

public class CustomHostnameVerifier implements HostnameVerifier {
    public String[] exemptNames = {"localhost","testing.com","example.com"}; 
   // whatever hosts need exempting from having a "real" SSL certificate
    public boolean verify(String hostname, SSLSession session) {
        boolean exempt = false;
        if(exemptNames != null) {
            for(String name:exemptNames) {
                if(hostname.toLowerCase().endsWith(name)) {
                    exempt = true;
                }
            }
        }
        return exempt;
    }

    // call from BootStrap.groovy or other opportune initializer
    public static void init() {
        HttpsURLConnection.setDefaultHostnameVerifier(new CustomHostnameVerifier());
    }

}

Other work arounds would involve a staging CAS server that was able to resolve the development or testing host during work on proxying. These are up to you.

When NOT to use a Domain Specific Language

There's really one rule I can use to sum up why you would not use a domain specific language.

Do not use a Domain Specific Language when implementation specifics out weigh domain specifics.

This one rule covers numerous cases:

1. when intimate knowledge of API are as important as function


2. when performance is critical and requires expert knowledge


3. when implementation details are nearly as important as semantic details

All these cases and numerous variations on this theme are summed up by acknowledging that the choice of using a Domain Specific Language (DSL) is a design trade off. You are deliberately adding a layer to your application stack. This layer intrinsically incurs a maintenance and/or performance cost. (Not all DSL cost additional clock cycles at run time it depends on your implementation.)

With these costs and benefits in mind you can make an intelligent choice about when and where to use this tool. It is far from a panacea but Domain Specific Languages do have the allure of potentially netting you accidental programmers and preserving folk knowledge about your system and its real world uses that few other techniques offer.

Never edit icky JBoss XML files again!

Do you have a JBoss application server? Do you have configuration files in that server that need to be edited or changed periodically? Do you have an allergy to XML?

Have I got a solution for you!

JBoss has the amazing ability to load *.properties files from any location (on the file system or on the internet) as part of its start up routine. So you can set properties in a simple properties file instead of an icky XML file.

NOTE: I'll refer to the directory you have installed JBoss into as $JBOSS_HOME you may have your JBoss server installed in /opt/jboss or /usr/local/jboss for example. It doesn't matter.

NOTE: I'll assume your JBoss is set up to load the "default" server directory. If it isn't then I think you are smart enough to figure out what to do on your own.

1. Stop the JBoss server


2. open the file $JBOSS_HOME/server/default/deploy/properties-service.xml in your favorite text editor.


3. Find the tag

   <mbean code="org.jboss.varia.property.SystemPropertiesService" 
     name="jboss:type=Service,name=SystemProperties">



4. Add this tag:
   

   <attribute name="URLList">
         /etc/jboss/system.properties
   </attribute>

   
   


5. Comment out or delete any other instances of the <attribute name="URLList"> ... </attribute> tag

Great! Now when we start up the JBoss server again it will read the file /etc/jboss/system.properties and set its properties from there. But, wait... the file doesn't exist yet. If I start my JBoss server and that file isn't easily readable it will crash.

So let's create the directory...

$ sudo mkdir -p /etc/jboss

... and the file ...

$ sudo touch /etc/jboss/system.properties

... now we can restart JBoss safely.

We don't get much benefit yet though. Nothing in JBoss is using that file. So let's take a database-ds.xml file and configure it to use system.properties for its connector-url, user-name, and password.

Assuming you have a file like $JBOSS_HOME/server/default/deploy/database-ds.xml you can edit it to look like:

<?xml version="1.0" encoding="UTF-8"?>
<datasources>
   <local-tx-datasource>
      <jndi-name>MyDatabaseDS</jndi-name>
      
      <connection-url>${MyDatabaseDS.connection-url}</connection-url>
      <driver-class>${MyDatabaseDS.driver-class}</driver-class>
      <user-name>${MyDatabaseDS.user-name}</user-name>
      <password>${MyDatabaseDS.password}</password>
      
      <min-pool-size>10</min-pool-size>
      <max-pool-size>20</max-pool-size>
      <prepared-statement-cache-size>10</prepared-statement-cache-size>      
   </local-tx-datasource>
</datasources>

Now edit /etc/jboss/system.properties to hold those values... it should look like this

# config for database-ds.xml
MyDatabaseDS.connection-url=jdbc:postgresql://localhost:5432/database
MyDatabaseDS.driver-class=org.postgresql.Driver
MyDatabaseDS.user-name=username
MyDatabaseDS.password=password
# this example is for postgresql but you could substitute any database.

... when you start up JBoss it will read the /etc/jboss/system.properties file and when it looks at database-ds.xml it will see the ${variable} values and substitute the values you set in system.properties for the ${variable}. If a ${variable} is not present, there will be no substitution... so be sure that every time you use a variable you set its value somewhere.

You can also set system wide system properties this way too. Commonly on servers that I work with the unix host name has nothing to do with the hostname being served. So I commonly set:

server.host=server.example.com

For use in my Grails applications which I then read in Config.groovy like this:

grails.serverURL = "https://${System.getProperty('server.host')}/${appName}"

... but that is another story ...

QRCode Linkulator the interesting bit.

For me the interesting bit about writing QRCode Linkulator was creating a PNG on AppEngine. The JRE Class White List doesn't include any Graphical libraries. Notice that most of java.awt.* is missing?

Why would Google do that? Well, try downloading a copy of the OpenJDK. Go ahead. I'll wait. Got it now? Great. Now look in openjdk-6-src/jdk/src/share/native/sun/awt ... what do you see? A mess of native C code.

So key components of OpenJDK are written in C. Key components of AWT. That means if you want to do graphics (server side) in Java you need to hit some chunks of C code. I bet that's dangerous in AppEngine.

Combine this with the fact that pngj is the only project I could find that had a pure Java implementation of the PNG specification and the challenge of doing the "impossible" and the temptation to pull this off was irresistible. Consider how many programs rely on those classes and there is NO pure Java solution in existence.

At my current workplace we do a lot with cloud computing. Because of this I've got a special sympathy for the guys working on AppEngine. They probably have some technical infrastructural reason they can't let you get at those AWT classes either directly or indirectly. My guess is it has to do with how they implemented the custom JRE that your app runs in.

That combined with an interesting socio-economic constraint they placed on AppEngine applications makes me think that AppEngine isn't just "free" as in beer hosting but really an interesting way for Google to codify their chief aesthetic concerns about how applications function.

This aesthetic is born of cloud computing on a scale no other company gets close to. At this scale, apparently, an aesthetic similar to that found in embedded systems starts to come through. Which in my mind is extremely ironic. See... I've been doing some Android development lately.

Consider that your app-engine app may be sitting cold with no one using it. Then the user hits the application for the first time... perhaps deep linking into your application. The deep link re-activates the app. It's cold, dark, and if you are a scripting environment... not compiled requiring an interpreter to fire up, load your script and then interpret it.

Huge penalties in that environment. So I hear some clever developers have kept their applications hot by writing Google AppEngine Cron Jobs which hit the web application in some low cost way to keep it warm. That's pretty clever. But only works until your application scales.

Yes. That's right. It works until your application becomes really really popular. Why?

When your application scales your application is going to get provisioned with another "machine" on the Google Cloud. That application will be cold. You'll pay the warm up penalty for that application and that first user gets the raw deal having to wait while your app spins up. Bad news. This happens every time your app scales up one "machine" and the net effect can be Fail Whale like badness.

Besides, inflating your own app's "popularity" by "hitting yourself" seems a tad... well... evil. So please. Stop hitting yourself.

The "right" answer is to somehow mitigate that first page hit cost. You get that first page view cost down as close to zero as you can... then take it a little farther.

How? Anyone have ideas?

QRCode Linkulator speaking Gaelyk

I've just written my first application using Gaelyk a Groovy based framework for Google App Engine. I ported the qrcode plugin I had written for Grails over to Gaelyk in a matter of hours.

This isn't really an application for iPhone or Android but it relies on the Barcode Scanner on Android or an app like
Barcodes on the iTunes AppStore for iPhone. Combined with a tiny "bookmarklet" I found the result is quite useful for moving a link between my desktop and my phone. I made a demonstration video since this can be a little hard to explain.

You can get a gander at the app here: QRCode Linkulator. I make use of the HTTP referer header property to determine where you came from so if you were to click http://qrcodelinkulator.appspot.com/300 for example then you would get a QRCode for the URL you were at *before* you clicked the link. The error mode (should you have no "referer" associated with your browser (for example if you were to right-click the link and open it in a new window) is just the logo for the app.

You might notice I tested out this feature in my blog. When you hit a given page in my blog there will be a QRCode for the URL you are currently looking at embedded in the page. This is done with simple image tags and no javascript.

Right now, I'm just hosting the app on the free AppEngine account and hoping that is enough CPU + Bandwidth for the application. I'm also hoping to stress-test the app a little to get a feel for how things scale. So far Gaelyk is running error free. I'm quite pleased with the simplicity of the framework and the price is right for App Engine hosting.

I doubt I'll get enough traffic to have to worry about paying for the service if the app gets that popular I'll have to figure out how to get it to pay for itself. Any ideas?

Domain Specific Languages @ TriJUG

This version of my DSL talk focused on architectural implications of Domain Specific Languages. Less time is spent on language details and I dive right into creating the language.

Speaking at TriJUG

I'll be giving a version of my DSL talk at TriJUG coming up March 15th.