New Year's Thoughts

If it takes ten years to get good at software development then I'm over due to get good by now I suppose. I am also going to have to take some calculated risks in the next year. So let's see what I was talking about last year and how things are shaping up so far.

Last year (a little in advance of the new year) I posted some thoughts I had been working on. In last year's "new year's thoughts" I said about software design:

Get to the core thought, the core idea, ignore as much minutiae as possible, get the problem solved. Keep the idea clean. Keep the core thought from being tangled in a mess of details that have nothing to do with the answer you are after.

... and I predicted that FP, Rules Engines, and DSL would become more important going forward ...

Functional programming, rules engines, and dynamic languages promise to free us from those mind-killing details that we don't really care about anymore as application developers. But they don't really address the need to create adaptive systems or systems that are able to span across computational mediums.

I had both identified the current trends and what I saw as the core problem. The fact of the matter is that the ideas behind the new fads in programming languages themselves won't really tackle the real problem in software engineering.

... so what is new in the last 365 days?

I recently found Kevin Kelly's the next 5,000 days TED talk. He has very keenly pointed out something that seems at once profound yet very simple. I can't imagine not having this idea to work with anymore.

If you want to really be relevant to the advance of technology today then you will position yourself to take advantage of Kevin Kelly's central point: There is only one machine and every screen in the world is a portal into that one machine. The reason the iPhone is a rampant success is that it really is a window into the one machine and acts like it.

The devices of the internet are approaching a complexity rivaling biology. To think we can understand them with our existing tools is a little foolish. We are yet to really develop the mental tools to deal with this new complex system. We will need these cognitive tools and we need them soon. Once we have them we will think it was odd they were so hard to conceive of. Much like the idea behind the number zero. It is hard to imagine how the first person conceived of the idea behind the number "zero" yet it is hard to imagine not having it.

This idea of "the one" is the guiding idea behind much of my own development efforts today. I didn't have a term for this before and I'm thankful to Kevin Kelly for coining a term for this idea. In many systems I confront there isn't an understanding of "the one" and data lives on barren islands of isolation.

In my new job I am very pleased to be allowed to participate in the development of interesting new technologies and I hope that I can find seemingly simple yet powerful changes to introduce that are like the idea of "zero" ... powerful, simple, and seemingly obvious once they are fully understood. That is the beauty of the truth when you have found it. It seems too simple and too obvious to have been missed.

There is a way to express UI, data, and logic that still is to be discovered. It will be obvious after we've found it. It will be profoundly different than what we have now. We won't recognize crossing into it until after we have already done so.

I expect that in 12 months we'll have better knowledge of which ideas are wrong and which ones more accurately reflect the emergent reality of the one machine. We will learn better the shape of the world it is carving out. That shape is not so much our invention anymore as it is a new undiscovered country which we are just now charting.

21st Century Network management

Oblong Industries

About once a year I see something like this: http://oblong.com/ and it renews the passion I have for technology changing the way we interact with our machine creations. I only dream that someday I can create a work that is as visceral in its impact. I hope systems like this will one day be everywhere and also not require any UI gadgetry such as VR gloves like the ones shown in the video.

New Year's Thoughts

According to Kurzweil your IT systems will be twice as powerful in 3 years. By 2013 or so we should be able to model a human brain in entirety on the world's most powerful machines. In time that power will hopefully trickle down to you and me. If you are like me, you write software, how are you going to use all that power? How are you going to produce all those features people need so rapidly?

If you've read my blog at all you'll know I'll tout the advance of the DSL and the dynamic language. I see these technologies on the JVM at a key inflection point between backwards compatibility and increased efficacy. They are transitional technologies at a key period in technological history.

The win in DSL, dynamic languages, and frameworks is always how succinctly you can get to your core thought or core idea. I don't really care about the ORM, the Ajax, the email most of the time. Most of the time there is a core idea I'm trying to work out.

In 24 months I need to be in the place where I can effectively divide the problems in front of me down into their key ideas. Keep the competing paradigms apart from each other. And then combine them at the right moments of confluence with a minimum of dissonance.

Get to the core thought, the core idea, ignore as much minutiae as possible, get the problem solved. Keep the idea clean. Keep the core thought from being tangled in a mess of details that have nothing to do with the answer you are after.

Advances in the way we create software will be key to the advance of the technologies of the future. If you are writing software today you will have to be prepared to leap a mighty chasm approaching us at break neck speed. That chasm will be paradoxically caused the increase of computational power available to us.

The problem of "traditional" programming that it is very procedural which can feel plodding at times. It takes forever to say anything meaningful. Yet before long we will have the ability to command computational power that could out strip our ability to speak into these systems with these sometimes unnecessarily verbose languages.

Consider that (in the video linked above) Kurzweil says that a scant 30 million bytes describes the whole of the human brain. The reason 30 million bytes could possibly hold the whole design for the human brain is that the brain's design is fractal and self organizing. A uniform underlying design is repeated with variation to create a whole. This is a similar goal to what we have in DRY (Don't Repeat Yourself) code that means we express an idea and leverage that idea again and again.

In the case of our brains, that assembled whole adapts to inputs to form intelligent variation leading to astounding complexity and abilities. The subtle variation introduced by input creates the mote of a soul... or at the very least all the personality traits we recognize as a living person. If only we could write software that could reorganize itself like that.

Functional programming, rules engines, and dynamic languages promise to free us from those mind-killing details that we don't really care about anymore as application developers. But they don't really address the need to create adaptive systems or systems that are able to span across computational mediums.

I predict that there will have to be new programming languages that run in new ways that allow for fractal designs like the human brain. I happen to think these still need to be languages because they are still conversations between a creator and the created system. I dream about these systems acting like holograms or fractals in their nature.

If you cut a hologram in half you get two identical but smaller and fuzzier holograms. If you look at one part of a fractal you see the whole. I think human designed software can have the same quality. The idea isn't new I've heard the opposite idea called "dirt" and perhaps that's a good word but in my mind these form knots ... asymmetric distributions of thought where one idea unrelated to another must intersect in incongruent ways. Perhaps "dirt" is the antithesis of "fractal" or "crystal" design.

I say "fractal" or "crystal" because "modular" isn't strong enough. You can make modules all day long but if they don't allow for cleanly intersecting concerns then you don't haven't done much more than take a messy room and shovel it's contents into random boxes. You haven't actually organized things you've just cut down on the size of the mess you have to sift through.

Good code focuses on one concern at a time and tries to ignore details that are incidental. It should be that in daily programming activities we can use tools like AOP and DSL to focus on the foremost thought ignoring details of execution. Good code is reusable because it encapsulates one concern in a way that is clean enough and simple enough to be easily lifted out and reused.

That's good stuff because if you're like me you really get paid for some high level thought and some high level idea and not the millions of tiny details you have to move through to get there.

The future...

I've spent a lot of time the last few months thinking about the future. Which technology should I invest my time in? Which framework will pay off the most? Where will I be in ten years?

Obviously, these are unanswerable questions. Everyone in the IT industry is asking these questions right now. With the second internet wave now fully upon us we are seeing a return of the technical uncertainty that IT outsiders didn't see in the first internet wave.

The fragmentation of the market is only natural and it is only natural to assume that the bigger players in the tool space will likely survive. But, which one will thrive? Who is getting the critical mind-share to prosper?

I think the next year or so will tell us which framework is really going to win. This time we know all the relevant business models for selling software. We know that free software is a vital on-ramp to small companies whether they are for profit or not. We know that the amateur programmer plays a vital role in the software ecosystem building the rough 1.0 for companies that don't have the backing to hire professionals.

Yet companies that stake their new businesses on amateurs only pay a huge technical debt later when they have to deal with success. I personally have seen two businesses fail due to huge technical debts that launched the company rapidly, but later swamped the same company when the software they had developed so cheaply could not scale to handle unbridled success. Their success bankrupted them as they could not handle the increased demand and earned bad reputations that caused their business to plummet.

In both cases I witnessed competitors saw that the new business idea prospered but failed due to technical debt. In both cases a branch company was formed and grew more slowly on a much more solid financial, process oriented, and technical base. The result was the second company does much better and destroys the original.

There needs to be a balance point in the software ecosystem. Businesses need to launch cheaply and quickly and still be able to refactor themselves for great successes. I think groovy has a good chance at meeting this need. It will be rapid to learn and should scale well... and when things get really tight you are ready to fall back to Java and the Enterprise stack.

Is there another programming language that offers an easy on-ramp from neophyte to journeyman out there? Is a tool like Ruby really going to make in roads into banks and cell phones? Is there another path?

I don't know for certain what will happen. But, I've made my choice.