Lucas Ward's Blog

On Git Command Line Usability

2011-12-14T22:29:00.000-08:00

Today I was reading about a new tool to use Git on a Mac called GitBox. I generally prefer the command-line, but as it was on sale and mentioned as one of the best clients for Mac, I figured I would at least take a look. When looking through the comments though, one part of a particular review stuck out to me:

This app generally is a relieve compared to the terribly complicated git command line

It's a common refrain from even a seasoned developer that has been using other tools for awhile, even those familiar with subversion on the command-line. To simplify the discussion, let's take one part of the obvious usability issues out of the general discussion of git usability: DVCS in general. If using something like Hg or Git after using a more centralized VCS, it's going to be more complicated by nature. With DVCS you're effectively managing your own repo and merging with another common repo shared with the rest of your team. With a more centralized system like SVN you're really just pushing change lists to a repo managed by someone else. I'm sure there's some power users at every company, but I'm betting a lot of SVN users aren't doing a lot of branch merging. Even looking at GitBox, which does look like a more simplified view of Git, the overall actions you're performing are more complicated than say, something like tortoiseSVN.

DVCS has it's advantages and disadvantages, but given that the overall model is more complex in nature for individual developers committing code, let's shelve that and focus solely on the question of whether the git command line itself is overly complicated. (as the review described)

My gut response to this question is 'it depends'. Perhaps I was a consultant for too long. Git to me seems Unixy. That's not really a word and doesn't mean much though. I suppose another way of saying it is: if you're used to working in unix like environments, it feels comfortable and familiar to you, and if you're not, it's never going to feel like anything other than overly complicated. If you've written a lot of bash scripts, or regularly tweaked your command line (i.e. bash settings), git feels natural. If you're not that kind of command-line user, git seems complex.

I would like to strenuously point out that I am in no way trying to offend users who don't like unix (or unix like operating system) or using the command-line in general. I have some strong personal feelings about it, but at the end of the day, it's personal preference. If you don't like using the command-line, then you don't like it. As an example, when talking through how to us a rest API with someone a month or so ago, I mentioned that they could just toss the URL into curl. I'm so used to reaching for the command-line as my first instinct, that I didn't even think that for someone not used to the command-line, it might seem complex. It's just curl -XGET http://someurl afterall.

But it's never that simple is it? The next question is, ok, how do I post data?

Hmm.....well, something like this:

curl -XPOST http://someurl -d "{"jsonstuff":"value"}"

Except, that won't work. You can't use double quotes like that. It has to be:

curl -XPOST http://someurl -d '{"jsonstuff":"value"}'

That will work. Well, maybe. You might need to set a json content header. If you're used to using a unix command, you just type man curl, look up the option and do:

curl -XPOST http://someurl -H "ContentType:application/json" -d '{"jsonstuff":"value"}'

But reading man pages can sometimes be an acquired skill. For those of us that are used to it, we might quickly notice the quote issue, or quickly find the option we're looking for with a quick man page search. However, if you just wanted to get something done quickly and you're not used to these types of tools, it makes a simple test of a rest API turn into an epic undertaking leaving you doing a google search for rest thick clients.

As another example, I recently wiped my mac machine. I hadn't used it in awhile, as I normally code on an Ubuntu machine. However, the stock git that came with the fresh install of OS X Lion was jarring. There was no colors set, and I didn't have my customery indicator in bash telling me what branch I was on. I needed to copy some of my bash config over from my other machine.

It's worth repeating, to effectively use git on the command line, I needed to modify my bash profile. (or bashrc, whatever you want to call it, depending on how you have things setup and distro, etc etc) For a regular command line user, it's not a problem. But I remember when I first was learning to use linux, and failing miserably to even add things to my PATH the first few times.

So I guess that's my main point. Git to me fits in well with all my other gnu/unix style command line tools. It feels familiar and as it follows these conventions, I can work with it. But as I have introduced it and other command line tools to fellow programmers, it becomes obvious to me: If you're not familiar with these conventions and with how to work with this type of command line, it's going to seem extremely complex.

Why developers sometimes do things 'the hard way'

2011-06-27T11:08:00.000-07:00

I don't like using the a word. I never want to catch myself saying anything that could be construed as: 'methodology X says we should do it this way'. Especially when it comes to engineering practices. These kinds of things should stand on their own really. In fact, I'm going to try and steer clear of any specifics in this post and speak to a larger trend.

When working through an issue or a new feature, regardless of what environment you're coding in or what language you're using, there is probably a 'hard way' to work through the issue/feature. I don't mean this in terms of level of effort. I mean it takes longer for whatever reason. It may even be an uglier solution. I'll give an example. Let's say you're adding a new feature to a web application. It's not a big feature, maybe just adding the ability to search off of a new set of values. The easier way is to find some kind of quick feedback. A way to implement the new search and find out quickly if there's an issue. I'll stop short of saying 'write a test' for the reasons mentioned above. However, instead you make the change and fire up the application server. You try the simple feature and it doesn't work. You forgot something simple. You fix it, bounce the server, and try it again. Still doesn't work. Rinse and repeat ad infinitum.

To the outside observer, this certainly is 'the hard way' to do it. Because of the amount of time it takes to restart the web server and click through to test the feature. It takes more effort than some other means with a shorter feedback cycle. But to the person using this approach is it really 'the hard way'? The way they are doing it requires less thinking, and while they're waiting on the server to bounce back up they can check reddit, failblog, or have an impromptu sword fight. To put it another way, it takes longer but requires less effort.

Of course, at this point someone could start throwing out the word 'lazy'. But let's take a more sympathetic approach. Why are they doing it this way? Why do they want to think less? Maybe they're burnt out? Maybe they feel the project is out of their control and there is no reason to put in a lot of effort. That its better to save their brain power for the new programming language or framework they're learning in their spare time. Maybe they aren't getting enough sleep. They probably know deep down that its not really the most efficient way to do it, but they're tired. Emotionally, physically, or even subconsciously. I've certainly felt that way before.

Maybe the approach isn't convincing your developers to do things the more efficient way. Maybe its better to be more sympathetic and find out why they're doing it the easy way.

Open Symphony, RIP (2000 - 2011)

2011-06-18T12:17:00.000-07:00

It looks like the folks over at open symphony have declared the project dead. The main projects that originated there and are still in use have their own separate sites now, but the rest hadn't been updated in awhile. I remember using OSCache awhile ago, but it seemed to have been completely replaced by EHCache.

The question in my mind is, do organization like OpenSymphony need to exist anymore? (perhaps the same could be said for something like CodeHaus as well) Given sites like github and bitbucket, and the ease of getting a project website up and running via various mechanisms, perhaps they aren't needed anymore? The one exception to that of course is community built around a particular project. Like plugins for Grails, or some of the Spring sub projects. They're tied to a particular solution and can benefit from some of the success of the solution as well. Although, in the case of Grails plugins, they're all hosted on GitHub, as is the code for grails itself.

Grails testing improvements

2011-06-08T08:08:00.000-07:00

Peter Ledbrook put up a new blog post detailing some of the improvements to come 1.4 for unit testing grails: http://blog.springsource.com/2011/06/07/countdown-to-grails-1-4-unit-testing/

There's some awesome stuff in there. I had read a bit about the new features in milestone 1. But it didn't show the kind of details that Peter has. I'm especially excited about these two features:

Full Gorm testing. It looks like they're using some kind of noSQL database in-memory. I'm interested to see if it will slow down the unit test at all, but the method of testing is much more intuitive. The whole mockDomain(Class,listOfInstances) felt a bit dirty, and it was kind of brittle if you used any kind of inheritance at all.
Improved JSON response testing. Grails now does the work in the mock response of converting the JSON response to a proper object hierarchy. In my own testing, I could get halfway there by converting the responseText to JSONObjects, but they were a bit painful to work with at times. (especially arrays)

Those two fixes represent for me the only real issues I have with grails unit testing. The annotations are nice, but I had given up on the test hierarchy long ago. You can use MockUtils easily in the same way that GrailsUnitTest does.

That leaves only one major complaint for me regarding testing and Grails: Integration testing. Its nice that they found ways to allow for using unit testing more often, but it doesn't solve the fundamental problem. Integration testing takes too long because you have to start the world. Lately I've been working a lot with ElasticSearch, and I refused to put it in the Integration test folder and rely on the bootstrap. In the test setup I brought up an in-memory ElasticSearch instance and test against it. Its definitely an integration test, but I can run it in about 6 seconds. It takes longer than that for Grails to resolve dependencies. (Which is something I hear they're addressing in 1.4) Its all perfectly fine for CI, but you can't use it to test on the fly. Its far too tempting to do a run-war and fiddle with it that way. It seems like it wouldn't be too difficult to bring up a persistent integration testing environment. In truth, it would probably still be some kind of headless container that you could keep running integration tests in without having to restart the container. I've seen a few other Grails bloggers mention it though, so perhaps its on the roadmap somewhere? Maybe post 1.4?

I'm speaking at Chicago Groovy User Group on June 14th

2011-06-06T13:59:00.000-07:00

Details can be found here: http://cgug.org/

And you can sign up here: http://cgug.eventbrite.com/?ref=enivtefor?amp;utm_source=eb_email&utm_media=email&utm_compaign=invitefor&utm_term=readmore

Fundspire is hiring

2011-05-24T18:01:00.000-07:00

The title says it all. Things are going very well here at Fundspire, and we're looking to expand. You can find more information here: http://www.fundspire.com/careers/

Just because I can be a bit more narrative here. We're using the following technologies: Grails, Ext Js, Spring Batch running on Amazon EC2. Of course, it would be great to find someone with experience in all of those technologies, but it never works out that way. A solid Java developer experienced with Spring and Hibernate would also be great. And of course the most important skill of all, a developer that tests the code they write, not with a debugger, but by writing tests.

If you think you might be interested (or know someone who might be) please apply via the link above, or send your resume to jobs@fundspire.com

Moving the Mysql Data Directory? Beware of AppArmor

2011-05-15T16:54:00.000-07:00

Recently, my 8 year old linux server I was running mysql on died. My new computer is more than capable of running mysql in the background, but I didn't want to mess with moving everything over. Now I'm forced to. It's extremely easy on Ubuntu to install mysql. However, I run my operating system and all development related applications off of a 64GB SSD. There isn't room, or a need to run it on the SSD. I have a second platter based hard drive as well. It should be fairly straightforward to store MySql data there. Most instructions for doing so will tell you to do the following:

Copy the current data directory to your new partition. (Current directory is usually /var/lib/mysql on Ubuntu)
Make sure it is still owned by the mysql user. 'chown -R mysql:mysql newdatadir'
Change the mysql configuration to point to the new directory. Inside of my.cnf is a value datadir, which is what needs to be changed to the new directory.

Sounds easy, right? It is, for the most part. If you try these steps on ubuntu, it won't work. MySql won't start up. If you're like me, the first thing you'll look is syslog, where you'll find something like:

[1064644.473299] type=1400 audit(1305502269.192:364):
apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/mysqld" 
name="/media/data1/mysql-data/" pid=24578 comm="mysqld" requested_mask="r" 
denied_mask="r"fsuid=1001 ouid=1001

Your first instinct is to check the permissions, but they're all fine. So, what's going on? AppArmor is preventing access. If you type 'ls -la /etc/apparmor.d' you'll see the following:

drwxr-xr-x   8 root root  4096 2011-05-15 19:18 ./
drwxr-xr-x 141 root root 12288 2011-05-15 18:20 ../
drwxr-xr-x   3 root root  4096 2011-04-11 18:38 abstractions/
drwxr-xr-x   2 root root  4096 2011-05-15 17:38 cache/
drwxr-xr-x   2 root root  4096 2010-10-07 11:06 disable/
drwxr-xr-x   2 root root  4096 2010-08-06 23:19 force-complain/
-rw-r--r--   1 root root   986 2010-09-13 03:07 gdm-guest-session
drwxr-xr-x   2 root root  4096 2011-05-15 17:38 local/
-rw-r--r--   1 root root  2052 2010-08-06 23:18 sbin.dhclient3
drwxr-xr-x   3 root root  4096 2011-04-11 18:38 tunables/
-rw-r--r--   1 root root  2052 2010-09-27 17:58 usr.bin.evince
-rw-r--r--   1 root root  4006 2011-04-29 04:00 usr.bin.firefox
-rw-r--r--   1 root root  4158 2010-10-01 04:58 usr.sbin.cupsd
-rw-r--r--   1 root root   989 2010-11-10 00:51 usr.sbin.mysqld
-rw-r--r--   1 root root  1172 2010-08-06 12:45 usr.sbin.tcpdump

If you open up usr.sbin.mysqld, and add the new directory to the list, your problems will be solved. Something like:

 /mysql-data/ r,
/mysql-data/** rwk,

Tips for connecting to Github private repositories with Hudson/Jenkins

2011-05-04T19:41:00.000-07:00

I recently created a private repository on GitHub, and wanted to create a build for it on my Jenkins server. I ran into some interesting problems that I couldn't find reasonable solutions to via google, so I thought I would write about my experience.

The first and most important thing to keep in mind is that there are no settings in Jenkins to control or manage SSH keys for Git. You might get confused when you see the SSH section in the general settings, thinking you can upload your git RSA key there and it will work. It won't. The only thing that will work is setting it in the .ssh directory of the user that is running Jenkins. If you're running it standalone, it will be the Jenkins user. In the case of a standard Ubuntu install, this was in /var/lib/jenkins/.ssh. If you are running Jenkins inside of a tomcat container, then it will be the tomcat user. (tomcat6 generally in Linux)

The second thing I learned is: If you're running Jenkins on Windows, it's going to be painful. Mind you, I'm not running it that way, but nearly every single Google result on this issue was coming back with issues for Windows. It's not really even Jenkins fault either. I'll stop short of bashing Windows, but if you've ever had to deal with running any type of service on Windows, you're already familiar with the pain anyway.

And the final lesson I learned? You can't use a passphrase in your RSA key file. I lost hours to this one. I thought I had some other issue with the key. Until I stumbled on this blog post. It makes sense when you think about it. When you're running git on your machine you might get prompted for access to a keychain, but after that its seamless so it can be easy to forget that its there. If you follow the default instructions for creating the key on github, it will guide you to create the passphrase. Ignore that and follow the advice on the previously mentioned post. Git should work without issue after that. I would still recommend using a passphrase for your personal account though.

Grails Envers Plugin

2011-04-22T10:55:00.000-07:00

I recently needed to add auditing to my grails application and decided to use envers, mostly for its ability to keep updates to multiple hibernate objects tied together in a way that is query-able afterwards. Unfortunately, because Grails exerts somewhat tight control of Hibernate, and because of how Envers was implemented, getting it to work with grails can be a bit tricky. I did a cursory glance around and couldn't find anyone that had created a plugin to help with Envers and Grails. So while implementing it in my application, I wrote the envers specific code as a plugin, which I have published on github:

https://github.com/lucaslward/grails-envers-plugin

Eventually, I need to write actual documentation for it, and try and get it in the official grails plugin repository. However, for now, I'll write up how to use it based on the integration tests.

If you're pulling it down from the git repo above, you need only do a simple 'grails package-plugin', which will create a zip file, which can be then be installed into your grails application via: 'grails install-plugin /path/to/zip/grails-envers-plugin-0.1.x.zip'. The plugin sets up envers via Hibernate event listeners, and provides gorm style dynamic methods that can be used to query for revisions.

In order to configure your domain objects to use Envers, you need to annotate them with @Audited. (I haven't written any code to try and make that work with static members a la gorm) I'll use the sample domain from the plugin:

@Audited
class Customer {

  String email
  String name
  Address address
  SortedSet orders = new TreeSet()

  static constraints = {
    address (blank: true, nullable: true)
    address component: true
  }

  static mapping = {
    orders cascade: "all,delete-orphan"
  }
}

@Audited
class Address {
  String city
  String zip
}

@Audited
class OrderEntry implements Comparable{

  Date date
  double amount
  int numberOfItems
  Customer customer

  static belongTo = [customer:Customer]

  @Override
  int compareTo(OrderEntry o) {
    date.compareTo(o.date)
  }
}

In this example, I have three domain classes: Customer, Address, and OrderEntry. Customer in this class is the main class, who has a one to one relation ship with an Address, and a collection of Orders. Now, let's assume that we create a customer with an address, then modify the customer and the address twice in separate transactions:

Customer customer
Customer.withTransaction {
  def address = new Address(city: "Chicago", zip: "60640")
  address.save()
  customer = new Customer(name: "PureGorm", email: "tester@gorm.org", address: address)
  customer.save(flush: true)
}

Customer.withTransaction {
  customer = Customer.findByName("PureGorm")
  customer.email = "tester2@gorm.org"
  customer.address.city = "New York"
  customer.save(flush: true)
}

Customer.withTransaction {
  customer = Customer.findByName("PureGorm")
  customer.email = "tester3@gorm.org"
  customer.address.zip = "10003"
  customer.save(flush: true)
}

Note: If you're trying to test envers in your application you have to turn off transactions for your test and wrap individual calls in transactions as above. (with an appropriate tear-down to clean up afterwards of course)

In this scenario, we should have 3 entries in the audit tables for Customer and Address (customer_aud and address_aud respectively). We can query for this methods with the findAllRevisions dynamic method:

def results = Customer.findAllRevisions()

assert results.size() == 3
def r = results[0]
assert r.name == "PureGorm"
assert r.email == "tester@gorm.org"
assert r.address.city == "Chicago"
assert r.address.zip == "60640"
assert r.revisionType == RevisionType.ADD
r = results[1]
assert r.email == "tester2@gorm.org"
assert r.address.city == "New York"
assert r.revisionType == RevisionType.MOD
r = results[2]
assert r.email == "tester3@gorm.org"
assert r.address.zip == "10003"
assert r.revisionType == RevisionType.MOD

As you can see, findAllRevisions returns an array of three results. The plugin will automatically take the envers RevisionEntity and RevisionType, and set them on the returned revisions, which is how revisionType is being checked. As you can see, all three revisions are present. By default its sorted by the earliest revision first. First is an add, followed by two 'mods' (update). The revision entity is also accessible via the revisionEntity property: r.revisionEntity.getRevisionDate(). It's also worth noting that even a custom envers RevisionEntity will be accessible this way. (See the envers documentation for more details on how to do this)

findAllRevisions also supports sorting in the same way as gorm queries. Assuming the same audit trail as above, you could query it this way:

def results = Customer.findAllRevisions(sort:"email",order:"asc")

assert results.size() == 3
assert results[0].email == "tester2@gorm.org"
UserRevisionEntity entity = results[0].revisionEntity
assert entity.getUserId() == currentUser.id
assert results[0].revisionEntity.getUserId() == currentUser.id
assert results[1].email == "tester3@gorm.org"
assert results[2].email == "tester@gorm.org"

You still get back 3 results. But this time the first result is the first update, followed by the second, following by the initial create. This is because its being sorted by email ascending.

As with sorting, you can also use max and offset:

    
def results = Customer.findAllRevisions(max:1)

assert results.size() == 1
assert results[0].email == "tester@gorm.org"

results = Customer.findAllRevisions(max:1,offset:1)

assert results[0].email == "tester2@gorm.org"

results = Customer.findAllRevisions(max:1,offset:2)

assert results[0].email == "tester3@gorm.org"

Besides sorting and paginating, you can also search for revision by property name:

Customer.findAllRevisionsById(customer.id)

Customer.findAllRevisionsByAddress(customer.address)

Customer.findAllRevisionsByName("PureGorm",[sort:"email",order:"asc", max:2])

The first case above is searching by id, and requires a long. The second is searching by the address, which requires an attached hibernate entity. The final is searching by a simple property name, in this case: "PureGorm", with some additional sorting, etc tacked on. As with gorm, you can use any property name on the object to query by. At this point, the plugin only supports querying by one property. If you need to query by more, you will need to use the Envers classes directly. (i.e. AuditReader)

The final method available statically is getCurrentRevision():

Customer.getCurrentRevision()

Which gets the current revision number. Meaning, what is the last revision of any customer?

There are also two methods that are applicable on domain class instances as well:

     
List revisions = customer.getRevisions()
assert revisions != null
assert revisions.size() == 3

Customer oldCustomer = customer.findAtRevision(revisions[1])
assert oldCustomer.email == "tester2@gorm.org"
assert oldCustomer.address.city == "New York"

The first method will return all revisions of a particular class. (The same as Customer.findAllRevisionById(customer.id)) And the second will find a particular customer at a particular revision.

There is certainly some functionality I'm missing here, as I'm using the integration tests as an example. I will add more blog entries on the subject as I discover the corner cases I have missed. Hopefully, I can also get time to write actual documentation, and get the plugin in the official repository as well.

When working with Ext JS, pay attention to the cookies

2011-04-08T12:04:00.001-07:00

We use Ext Js as our main Javascript library at Fundspire. There's a couple of really cool features we use a lot. One is the GridPanel. It gives you a nice looking 'table' with sorting, column resizing, etc. There are also some interesting ways to make multiple 'panes' in your application. One example being the border layout. (The layout browser shows all the layouts pretty well) A good example is the official 'forum example' from Ext Js: http://dev.sencha.com/deploy/dev/examples/forum/forum.html I'm still relatively new to the framework, but have been able to do some interesting things with it. After a couple more months I hope to write up a more detailed review.

I've been working with a lot of these components this week and was having a lot of trouble manipulating the column and border size defaults. My changes just didn't seem to have an impact. We use jawr, which renames the javascript when there's changes to force a reload, so that wasn't the issue. It turns out that the border and column size information is stored in a cookie on the user's browser! It's something that's easily setup via an Ext Js CookieProvider Once you have a cookie provider setup, your state across a lot of components is saved automatically. Which, if you think about it is pretty slick. If someone is using a smaller monitor they can change the sizes to be what they want and not have to worry about it again. However, I went down a lot of rabbit holes when my changes didn't appear to be having an affect, and I haven't seen this behavior documented anywhere in the Api docs for the components I'm using. I knew we were saving state in cookies, but didn't realize that it included these components automatically. So, if you're using Ext Js and your layout changes don't appear to be having an effect, make sure that it's not storing a cookie to save that information.

Accessing the identifier of a lazy loaded association in grails without another database call

2011-04-07T13:49:00.001-07:00

I ran into a bit of a weird scenario recently with grails. I have a couple of classes similar to those below:

class Foo {
Bar bar
}

class Bar {
static hasMany = [foos: Foo]
}

Essentially, I have a many to one relationship. Now, consider the following code:

Foo foo = Foo.get(1)
Bar bar = foo.bar

Assuming that there is actually a bar set on this particular foo, what will you get?

You'll get the actual Bar, completely instantiated. Now, imagine the exact same scenario in normal java with the exact same equivalent hibernate settings via annotations. (I'm too lazy to model what that would look like) What would you get back for a similar call? You would get back a proxy! Then, if you called something on the proxy, hibernate would go and fetch the actual object for you.

Gorm unwraps the proxy when the object itself is called via a property.

If you take a look at the HibernatePluginSupport class, which configures all of the dynamic Gorm methods (and the first place I go when I want to see how a particular method is actually working) you will see this in action:

static final LAZY_PROPERTY_HANDLER = { String propertyName ->
  def propertyValue = PropertyUtils.getProperty(delegate, propertyName)
  if (propertyValue instanceof HibernateProxy) {
      return GrailsHibernateUtil.unwrapProxy(propertyValue)
  }
  return propertyValue
}

/**
 * This method overrides a getter on a property that is a Hibernate proxy 
 * in order to make sure the initialized object is returned hence avoiding 
 * Hibernate proxy hell
 */
static void handleLazyProxy(GrailsDomainClass domainClass, 
                            GrailsDomainClassProperty property) {
    String propertyName = property.name
    def getterName = GrailsClassUtils.getGetterName(propertyName)
    def setterName = GrailsClassUtils.getSetterName(propertyName)
    domainClass.metaClass."${getterName}" = 
        LAZY_PROPERTY_HANDLER.curry(propertyName)
    domainClass.metaClass."${setterName}" = { 
        PropertyUtils.setProperty(delegate, propertyName, it)  
    }

    for (GrailsDomainClass sub in domainClass.subClasses) {
        handleLazyProxy(sub, sub.getPropertyByName(property.name))
    }
}

So, anytime you access your properties, the above code is going to unwrap it.

What whoever wrote this is trying to avoid is the 'proxy hell' mentioned in the comment. The hibernate docs have a good explanation of it:

http://docs.jboss.org/hibernate/core/3.3/reference/en/html/performance.html#performance-fetching-lazy

As does the Grails docs:

http://grails.org/doc/latest/guide/5.%20Object%20Relational%20Mapping%20%28GORM%29.html#5.5.2.8%20Eager%20and%20Lazy%20Fetching

They're more eagerly fetching the lazy relationships than Hibernate does to avoid some of the instanceof check issues outlined in the above links.

Now, back to my issue. What happens if you want to obtain the identifier of your relationship without actually resulting in a database call to get it? The proxy has the identifier already, that's how it loads it up when you call it. In pure hibernate there's a few ways to do it:

Session.getIdentifier
entity.getId() (assuming you are using property based configuration)
Cast it to a HibernateProxy, get the LazyLoadInitializer and get the Identifier from it

The problem with all of these solutions is that you have to get around the Gorm code above that unwraps any properties when you call a getter. You can't just add setBar and getBar. However, you can add another method that can get 'raw' access to the field. And before anyone posts a comment about it you can't use load either because it requires the id, and that's the whole point of this exercise (getting the id that is).

I didn't find this out until I had wasted quite a few hours, but apparently this was a known issue: http://jira.grails.org/browse/GRAILS-2570 I never saw this when googling for it originally, but you can essentially say:

foo.barId

And you will get the id of the bar without the proxy being unwrapped. I'm not sure this is referenced anywhere in the docs though.

New Desktop

2011-04-06T08:03:00.000-07:00

In my new gig at Fundspire, I generally work from home. For awhile this has been on my 4 year-old Macbook Pro. However, during my time at ThoughtWorks, I got extremely accustomed to using extremely powerful desktop machines running Ubuntu. In fact, I think the ones I used on my last project were running two xeon processors. Regardless, rather than get a new laptop, since I'm working from home I decided to build a new desktop. I've built a number of machines over the years, and it's always been a favorite hobby of mine. I enjoy spending time getting back up to speed with what all the major chipsets are, and the actual build can be a lot of fun too. Below, I'm going to list out everything I bought and why, with a total at the end. All of the prices and links are to NewEgg. I've been using them for years and have always been happy with their prices, selection and service.

Case

There's two areas I never scrimp on when building a new computer: The motherboard and the case. The case can make the build and future upgrades to the machine simple, or a nightmare. When looking for a case, three things are important to me:

Cable management. Besides the whole airflow thing, it makes it a lot easier to work inside of the case, or do simple upgrades when the cables aren't in a huge jumble in front of the motherboard. I can't count the number of times I accidentally knocked a cable loose and didn't realize it.
Sound proofing. Maybe I've just been using laptops too long, but I'm really over having a tower that sounds like a jet taking off when I turn it on.
Lots of space for additional drives, etc.

I have no intentions of overclocking, so I'm not particularly worried if it has things like water cooler hookups or whatever. And I really don't want a light show in my office. So, I went with a full tower by Fractal Design.

CPU

Since the CPU really determines the motherboard nowadays, I had to decide that next. However, since the new Intel Sandy Bridge processors came out a couple of months ago, it was a no-brainer. I went with the Core i7 Sandy Bridge 2600K. The K in this instance means the chip has an unlocked modifier, which is why all the overclockers you see tend to get either it or the i5. And while I'm not going to overclock it, there was a combo deal on my motherboard for the K version, which would normally be $20 more than the regular 2600 variant, making it the same price, so why not?

Motherboard

This was the most difficult spot for me. As I said above, it's probably the most important piece in the system, and completely affects how you can upgrade in the future, so it's important. The first thing I noticed when getting back into the latest hardware was how much it seems like every generation of new processors from either Intel or even AMD seems to have a completely different socket. Maybe they're more interchangeable than I think, but it at least seems pretty specific. For instance, take a look at this wikipedia page: http://en.wikipedia.org/wiki/List_of_Intel_Core_i7_microprocessors. Here's the sockets of all the different Intel i7 chips:

Lynnfield: LGA1156
Bloomfield/Gulftown: LGA 1366
Sandybridge: LGA1155

Obviously, I wanted to go with an LGA1155 socket motherboard, since that's the chip I got, but it's really easy to get confused. There's also a noticeable difference in the motherboards for Bloomfield processors. Most of the them are triple channel. While all the newer Sandy Bridge are all dual channel. It's a bit confusing at first, but SB represented a pretty big change in the memory controller. Namely, it's on the processor itself now and not the northbridge. That being the case, the only option for motherboards is whatever the chip has, which is dual channel. Also, from the numbers I've read, it seems that SB chipsets running 1333 DDR3 have higher memory thoroughputs than the previous generation triple channel setups, so it seems to work out.

I went with Gigabyte for my motherboard: http://www.newegg.com/Product/Product.aspx?Item=N82E16813128478

I really just wanted something that would support a lot of memory for the future, (This one goes up to 32 GB) SATA III (6.0 GBs) and USB 3.0 ports. I also tend to favor the 'enthusiast' motherboards a bit more, as their tendency to be used in overclocking setups means they have gratuitous amounts of heatsinks on all the chips. Having overheated more than one motherboard in the past, it's nice to have that even if you're not going to overclock.

Video Card

I'm going to be running Ubuntu, so the video card selection is extremely important. Video cards and linux have always had a checkered past, and even in 2011 you can easily pick up a card that doesn't work right with a particular kernel or something. (It happened to me last year with a Fedora setup I had) Personally, I think the only option is an NVidia setup. I used to use Radeons exclusively, but starting with something like the Geforce 8xxx chipsets, Nvidia added this thing called PureVideo. This has support for VDPAU (Video Decode and Presentation API for Unix) Which allows *nix machines to use hardware decoding of video on the video cards themselves. With that in mind, I bought a Geforce GT 430. It was introduced last fall, is extremely cheap and focuses on being able to push video, which is all I want. I watch mlb.tv on another screen while coding sometimes, without hardware video decoding, it would take up a lot of resources.

Everything else is pretty straightforward.

Powersupply: OCZ ModXStream Pro 700W You can now get 'modular' PSUs, which means that you only have the cables you plug into the unit, and don't have a tangle of wires for all the connections you're not currently using.

SSD: Crucial RealSSD 64GB Who in their right mind would build a system today and not use an SSD? Especially if you're a programmer. I went with 64 Gb to save money. It's enough to run ubuntu on the SSD, and any code projects. For everything else, I picked up a normal platter drive for $40.

Ram: 4 x 4GB DDR3 PC1333 (I did a quantity of 2 on the setup I linked to for a total of 16 GB) Most of the nicer SandyBridge setups support faster ram, but since the sandy bridge architecture is at 1333, I'm assuming they're getting those speeds by overclocking, and it wasn't something I wanted to deal with.

Everything else is pretty normal, I picked up a new DVD drive for $20, mostly because I needed one with a SATA hookup. (All my existing ones are PATA) I also picked up a heatsink, thermal paste, etc as well. They're generally cheap and worth it to make sure you don't fry your cpu. With all of that, the total including shipping was: $1,337.62

It's not too bad for what is essentially a top of the line system. The only way to go higher is to get into server grade territory, with two CPUs and 12 ram slots. You could reduce the cost on the setup I have by going i5 and halving the ram, but that only saves you $200, and as a programmer, most of my work is CPU and Ram bound so I think it's money well spent. You can also go cheaper on the motherboard and the case, but you'll always regret it. Of course, if you also want to play games on the computer, you'll probably want to dump way more money than I did in a video card.

Excluding jars from a Grails generated war file

2011-03-24T15:07:00.000-07:00

Yesterday, I posted about how to upgrade hibernate versions in grails. This solution worked great for me locally. (i.e. using run-app) But we were still getting weird issues in our staging environment. It's not uncommon for this kind of thing to happen, and I used my local tomcat setup to try and nail down the problem. When I cracked open the WAR, I found two version of hibernate, 3.3.1 and 3.3.2. Upon closer inspection of the problem, it turned out to be the same original issue I had upgraded Hibernate to fix, somewhat hidden by some additional 'noise' in the log file. I tried about 10 different things to get the war to generate correctly, until I stumbled onto this blog post:

http://www.anyware.co.uk/2005/2009/01/21/excluding-files-from-a-war-with-grails-the-right-way/

Grails gives you the option to declare a closure to modify the war resources. I added the following to the top of my buildConfig.groovy, and it fixed the issue:


grails.war.resources = { stagingDir ->
  delete(file:"${stagingDir}/WEB-INF/lib/hibernate-core-3.3.1.GA.jar")
}

Upgrading Grails 1.3.x to use Hibernate 3.3.2

2011-03-23T13:19:00.000-07:00

While using Hibernate Envers, I ran into an issue that was fixed in Hibernate 3.3.2. Unfortunately, Grails is using Hibernate 3.3.1. Luckily, it's fairly straightforward to exclude and declare a new dependency in your BuildConfig.groovy:


inherits("global") {
 excludes 'hibernate'
}
dependencies {
 compile ('org.hibernate:hibernate-core:3.3.2.GA'){
     excludes 'ehcache', 'xml-apis', 'commons-logging'
 }
}

Update: turns out it's not as straightforward as I thought. The above will work fine with 'grails run-app' but the 3.3.1 version will still be in the War file: http://www.lucasward.net/2011/03/excluding-jars-from-grails-generated.html You can fix that by adding the following to your BuildConfig.groovy:


grails.war.resources = { stagingDir ->
 delete(file:"${stagingDir}/WEB-INF/lib/hibernate-core-3.3.1.GA.jar")
}

Unfortunately, in the 3.3.2 release, "“org.hibernate.dialect.DialectFactory” was moved to “org.hibernate.dialect.resolve.DialectFactory”. Which was part of the following issue: http://opensource.atlassian.com/projects/hibernate/browse/HHH-2933

This is apparently an issue many have been seeing: http://grails.1312388.n4.nabble.com/update-hibernate-version-td3002449.html

While it may seem crazy that the Hibernate team moved something to a new package in a point release. I'm pretty sure they consider this class to be internal and not part of their API. The issue the change was made in was to expose to users a way to resolve their own dialects programmatically. Which, if you look at where the Grails Hibernate plugin is creating the offending spring bean, is the same thing Grails is doing:


if (ds && ds.dialect) {
  if (ds.dialect instanceof Class) {
      hibProps."hibernate.dialect" = ds.dialect.name
  }
  else {
      hibProps."hibernate.dialect" = ds.dialect.toString()
  }
}
else {
  dialectDetector(HibernateDialectDetectorFactoryBean) {
      dataSource = ref("dataSource")
      vendorNameDialectMappings = vendorToDialect
   }
   hibProps."hibernate.dialect" = dialectDetector
}

The above is taken from HibernatePluginSupport.groovy, line 128. (At least in Grails 1.3.6) Essentially they've written a Spring bean to programmatically determine the dialect by calling the DialectFactory themselves. You can bypass this issue completely by specifying your dialect, rather than having it detected. If you're using mysql, it looks something like:


dataSource {
driverClassName = "com.mysql.jdbc.Driver"
dialect = "org.hibernate.dialect.MySQLDialect"
}

The Programmer's Time Paradox

2011-03-17T21:12:00.001-07:00

At the beginning of this week I started work on a new feature. For the purposes of this post it doesn't matter what the feature was, but I started in on it on Monday and things went extremely well. I was in the zone. It's hard to describe, but I think if you're a programmer you'll understand the feeling. I started out with just the right amount of testing in the right places, built something simple and added on top of it, while refactoring periodically as I discovered new angles on the problem. Everytime I moved forward, I was able to run my tests and tell immediately that my last fix had broken something. I was moving forward in small steps and it was all falling together nicely.

And then I got greedy. I suppose it makes me a dork to admit it, but I was having fun. I had been somewhat behind on the feature because I was debating on a couple different paths, but now that I had chosen, I was moving forward with wreckless abandon. It's somewhat intoxicating to be in the zone. So, I pushed myself hard. I worked from around 9 AM until 1 or 2 in the morning every night for 3 days straight. (I work from home, so it's a bit easier to work later like that)

This morning I woke up begrudgingly. The comforter was very, very heavy. It was the weird programmer kind of fatigue, where you've done nothing but sit on your but all day, but your mind has been going too fast for too long, so you feel mentally exhausted but physically fine. When you hit that point, no amount of coffee is going to make it better. Now, I've been on some pretty hellish death marches before. Some of them spanned months, so it's not like working hard for 3 days is all that terrible. But the programming gods are vengeful. I might have been tired, but I was excited to get the feature working. I was in the home stretch. My sister is coming to town this weekend and it's her first time in Chicago since turning 21, so I wanted to get this thing done and party like it was...2003?

So I got sloppy. I ran into a weird bug. It was a NullPointerException with no stack trace. Which sometimes interchanged itself with an InovcationTargetException. When you see something like that happen, it's time to stop the production line. I should have stopped, immediately, and attempted to recreate the conditions in my testing setup. I had a perfect, isolated, test suite setup where I could have recreated this thing in a controlled environment, but I didn't do that. Instead, I fiddled with idea after idea. They were all well informed, reasonable things to be trying. But when you're doing this kind of thing inside your application, there can be too many factors. Production code is complicated, and there's just too many variables.

At around 6PM I stopped myself, and did what I should have done in the first place. In an integration test, I slowly recreated the problem, piece by piece. Within 15 minutes I had found the problem. I forgot to put an annotation on a particular class.

I. forgot. an. annotation.....$#$%#$%#$!!!!!!!!!!!

I suppose I could get mad at the guy who wrote the open source framework I was using. But I would be one heck of a hypocrite if I did. I can't imagine how many people have been in the same situation with Spring Batch, and probably felt the same way about me. I hit a scenario he didn't think of, so there wasn't specific error handling for it, and the results were nondeterministic.

No, I was tired. I pushed myself too hard and I made a stupid mistake, and the funny thing is, I lost all of the time I had saved by working so much! Awesome.

And when has that not been the case? I can't think of time where I worked too many hours and this exact same thing hasn't happened. I was tired and made a stupid mistake. A mistake that I would have easily caught if fully rested. And that mistake had cost all the time I had spent working late. And it's compounded by the number of people overworking themselves. The more people doing it, the worse the effect is. Because your stupid mistake made while trying to fix someone else's stupid mistake quickly snowballs.

But sometimes, it's just hard to grok. I don't think the human mind is fully capable of thinking that way. If you're running somewhere, running faster gets you there quicker, baring the limits of endurance. But overall, if you can run faster, running faster will get you there sooner. Or at least is seems that way when you're coding at 1 in the morning on an interesting problem. And I know all about sustainable pace, and I agree with it fully. And I wasn't under the spell of an evil project manager either. It's a small company, and I'm the CTO. The only drive I have is to get features done quicker to support our sales efforts. And I know, from personal experience, what happens when programmers work unsustainable paces.

You know what it's like? It's like white castle. At least once or twice a year I'll decide eating at White Castle is a good idea. I may or may not be sober when I make this decision, but I make it anyway. And I pay for it. The next day you wake up and you *know* you made a terrible mistake. And for days, maybe even weeks, you'll swear to never eat at white castle again. But then you'll be at your cousin's wedding and there just so happens to be a white castle within walking distance, and the terrible cycle repeats itself.

Grails Testing issue when rendering as 'text/json'

2011-03-09T11:40:00.001-08:00

I ran into this issue while unit testing my controllers (many of which only return JSON) in Grails. I've reported the issue to grails: http://jira.codehaus.org/browse/GRAILS-7339 but it's also worth noting here, so anyone else left scratching their head as to why the rendering doesn't appear to be working in their unit tests can find the workaround as well:

If you are writing a unit or integration test in grails and mocking the controller, which can be done either by extending ControllerUnitTestCase (or using mockController) you might run into this issue. As you probably know, there are multiple ways to render JSON in grails. One is to take a normal map and render as JSON:


def jsonMap = [success:true]
render jsonMap as JSON

This will work fine everywhere, because you're effectively passing it in as a converter. However, if you use the more verbose (and in many ways more powerful) rendering, you will run into issues:


render(contentType: "text/json") {
  success(true)
  results {
    result.each {
    def period = it.getPeriod() 
    b(id: it.id,name: it.name)
    }
  }
}

In my case, my test looks like this:


assert JSON.parse(this.controller.response.contentAsString)?.success == true

This won't work, because the content on the mock response will be null. With a little digging into how mockController() works, you will find the problem lies in MockUtils. Specifically, in the method that handles a map with render options and a closure:


clazz.metaClass.render = {Map map, Closure c ->
  renderArgs.putAll(map)

  switch(map["contentType"]) {
    case null:
    break

    case "application/xml":
    case "text/xml":
      def b = new StreamingMarkupBuilder()
      if (map["encoding"]) b.encoding = map["encoding"]

      def writable = b.bind(c)
      delegate.response.outputStream << writable
      break

    default:
      println "Nothing"
      break
  }
}

As you can see, the case of 'text/json' isn't handled. I was able to workaround this issue by creating a custom unit test case class to extend from the Grails ControllerUnitTestCase:


class CustomControllerTestCase extends ControllerUnitTestCase{

    protected void setUp() {
        super.setUp()
        controller.class.metaClass.render = {Map map, Closure c ->
            renderArgs.putAll(map)

            switch(map["contentType"]) {
            case null:
                break

            case "application/xml":
            case "text/xml":
                def b = new StreamingMarkupBuilder()
                if (map["encoding"]) b.encoding = map["encoding"]

                def writable = b.bind(c)
                delegate.response.outputStream << writable
                break

            case "text/json":
                new JSonBuilder(delegate.response).json(c)
                break
            default:
                println "Nothing"
                break
            }
        }
    }

}

Since it's groovy, you could pretty much replace this anywhere by overriding your mock controller's meta class.

Disabling Spring Security for grails integration testing

2011-03-09T08:09:00.001-08:00

One of the great things about Grails is how easy it is to do integration testing. When running the integration tests, the grails container will be started up, including an in-memory version of hsqldb. (and it looks to go to h2 in 1.4) When you combine that with controller mocking, you can easily test from you controller down to the database. However, because you're getting your entire system, minus only the web server, it can take a while to startup. It's not too bad for a CI build, but can get annoying when you're writing and running tests in your development environment. One of the big culprits in terms of time is spring security. I'm not sure why that is, but it is possible to disable it in your integration test environment. The first step is to set the active configuration property to false in Config.groovy:


test {
  grails.plugins.springsecurity.active = false        
}

Now, when you run 'grails test-app integration:' you should see something like this:


grails test-app integration:
Welcome to Grails 1.3.6 - http://grails.org/
Licensed under Apache Standard License 2.0
Resolving dependencies...
Dependencies resolved in 5123ms.
Environment set to test
Starting integration test phase ...
Spring Security is disabled, not loading

The important thing to notice is the last line: Spring Security is disabled, not loading

Of course, there's now a new problem, any code you have that is referencing springSecurityService will now fail. In general, the most likely method to cause an issue is getCurrentUser(). However, it's simple to create a quick stub for it:


class StubSpringSecurityService {

    def currentUser

    Object getCurrentUser() {
        return currentUser
    }

    String encodePassword(String password, salt = null) {
        return password
    }

}

In my case, I only needed to stub getCurrentUser() and encodePassword, however, you may need to add more, and the general approach can be applied to any additional methods you may need. It is also worth noting that I debated about using something like stubFor, but decided it was more straightforward to make a simple stub.

Now that you have a stub, you need to make sure that it's being set in the test environment. You can do this by adding the following to resources.groovy:


if (GrailsUtil.environment == "test"){
  springSecurityService(StubSpringSecurityService)
}

Now for the last step, setting up the current user in your bootstrap:


if (GrailsUtil.environment == "test") {
      def testUser = new User()
      testUser.save(failOnError: true)
      springSecurityService.currentUser = testUser;
}

Any code running during an integration test, will now get back the saved user if it calls getCurrentUser()

Javascript tip of the month

2011-02-28T12:52:00.001-08:00

I was reading an article I saw linked to from a blog I follow:

http://devblog.blackberry.com/2011/02/thanks-for-the-open-letter-to-rim-developer-relations/

Ironically, it's a response to an open letter to RIM about how difficult it is to use their PlayBook developer tools. I was doing other things when I noticed that page will still loading, so I took a quick peak into the html, and saw this:


<script src="http://code.jquery.com/jquery-latest.js"></script>
<script src="http://dev.jquery.com/view/trunk/plugins/validate/jquery.validate.js" type="text/javascript"></script>

This should go without saying, but don't do that! That's not the only offense, but certainly the most costly in terms of loading a page. There is actually another section that reference jquery locally, so I'm not sure how that got in there, but it has a pretty huge negative impact to have to download the unminified version of jquery from jquery.com.

Case Sensitivity: MySQL and Spring Batch

2011-02-10T09:25:00.000-08:00

I've been having some fun lately with multi-platform development. Personally, I develop on a Mac. However, I also have an Ubuntu workstation that runs a MySQL instance among other things that I can connect to at home. It's a nice setup, since production is ultimately running on ubuntu server and I have always advocated developing on an environment as close to production as possible. (Our build server is Ubuntu as well) However, one of the other devs uses Windows and has MySQL installed on his Windows machine as a development environment. This can create interesting case-sensitivity problems. Anyone using the default Mac setup (HSF+) or Windows have case-insensitive file systems. (Although, I run a case-sensitive variant of HSF+ on my mac) This can be important if you are also running MySQL on that system, as described below:

In MySQL, databases correspond to directories within the data directory. Each table within a database corresponds to at least one file within the database directory (and possibly more, depending on the storage engine). Consequently, the case sensitivity of the underlying operating system plays a part in the case sensitivity of database and table names. This means database and table names are not case sensitive in Windows, and case sensitive in most varieties of Unix.

This can be especially interesting in Spring Batch. By default, SB uses all uppercase table names. An example being BATCH_JOB_EXECUTION. If for example, your windows colleague creates the tables in lower case on his Windows system, they will work fine for him/her, since the underlying file system isn't case sensitive. However, if you take a data dump from his system and put it into your linux mysql instance, running batch jobs will result in the following error:


com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'dbmaster.BATCH_JOB_INSTANCE' doesn't exist

Or something similar. What could get even wackier is if you change the prefix of your Spring Batch tables. Setting the prefix to something like 'awesome', would result in SB querying for a table such as 'awesome_JOB_EXECUTION', which again, would work fine on Windows but blow up anywhere else. (Unless you happened to create the table in alternating caps)

There are some options, however. There is a property in MySQL called 'lower_case_table_names' that allows you to configure how MySQL deals with table name resolution:

0 - tables are stored on disk using the letter case defined in your create table statements and all checks are case sensitive. (This setting could be problematic on non-case sensitive systems)
1 - all table names are stored on disk as lowercase regardless of how you define them in your create statement, and any table names in queries will be converted to lower case
2 - table names are stored as defined in the create statement but mysql converts them to lowercase on lookup. Which obviously won't make a difference on a case-insensitive system.

Note: lower_case_table_names was introduced in MySQL 4.0.17

On my ubuntu install of MySQL lower_case_table_names is set to 0, which I assume to be the default. However, you can check your system easily by entering: 'show variables;' into mysql.

On my local ubuntu instance I create a file called: "/etc/mysql/conf.d/lower_case_table_names.cnf". The .cnf ending is important, as that is what my.cnf is setup to look for, at least in my setup. In the file is the following:


[mysqld]
lower_case_table_names=1

The [mysqld] is important as there are numerous engines which are configured in the same file. For example, you can all create settings for mysqldump specifically as well. Next time you bounce mysql, if you make another call to 'show variables' you should see the updated setting. After this change, even though Spring Batch is making SQL statements in all caps and the tables are actually stored in lower case, the queries will still work.

Amazon RDS

It's worth noting that if you use Amazon's RDS service that you must use their web console (or command line tools) to modify this setting. It's under parameter groups. The following guide from amazon should help you: http://aws.amazon.com/articles/2935

Sharing overload?

2011-02-03T21:57:00.000-08:00

First: Watch this hilarious VW spot on YouTube: http://www.youtube.com/watch?v=R55e-uHQna0

Second: Press the share button below the clip.

On the top we've got how many people shared this link on twitter, facebook or buzz. Do we really care about that?

Then the most important bit, the actual link, which we can shorten. Although I don't know why, doing so cuts the size down by 25%. I've always thought YouTube links were some of the simplest out there: youtube.com?watch?v=SomeRandomNumber. "Shortening" it turns it into: youtu.be/SomeRandomNumber. Go Belgium I guess.

And then there's no less than 9 places I can share it. Half of them I haven't heard of. And really, Myspace?

Can I just have a link?

Speculative Generality

2011-02-01T13:11:00.000-08:00

In my last post I talked about Velocity Anchors, and how applying metaphors such as technical debt takes away from the real problem, that developers want to get stuff done, and there's things slowing them down. I also mentioned a couple of causes, WTF and Code Jenga. WTF representing code that developers waste time having to figure out before they can get stuff done, and Code Jenga being the fear that even the smallest change will cause cascading failures and a lot of wasted time figuring out the problem. Time that could better be spent getting stuff done. To continue the series, I'd like to talk about one of the biggest causes of WTFs: Speculative Generality*. What speculative generality means is the inclination to speculate that some piece of functionality will be generally useful. I found a great illustration of this concept recently, even if it wasn't named that, from the chaosinmotion blog: How (not) to write factorial in java. Its a great post that uses some solid coding examples to illustrate the point, and I think his conclusion really nails it:

The biggest complaint I have with many Java developers is that they develop a whole bunch of really bad habits. Specifications are unclear, or they think someday the code may need to be extended into a different direction. So they write a whole bunch of overblown architectural nonsense, sight unseen, thinking that the additional crap someday will help out and make things easier. And Java as a language lends itself to doing this very quickly and easily, so that (as the theory goes) it’s easy for us to build architectures that will someday make it easier on us in the future.

But the future never gets easier, does it?

Later, in responding to comments, he presents his theory on why this tends to happen so often in programming:

Java is simple to program. With a good IDE like Eclipse, Java is a pleasure to write programs in. It’s also quite easy to write some very complicated algorithms, and building a client/server stack is absolutely trivial.

But programmers like complexity. We like things that are complicated: the flashing complicated lights on the control panels on Star Trek, or the joys of figuring out a complicated calculator with nearly a hundred buttons or playing with cell phones that have dozens of modes and menus and buttons. Programmers are puzzle solvers, and we love puzzles. We love taking things apart, putting things together again: many of us are the ones who were punished at 8 for disassembling the family computer or television set or radio, despite the fact that we nearly put it together in working order.

We love complexity.

His essential theory is that Java is easy to crank out code and that deep down developers love complexity. There's probably something to that. However, with the exception of assembler, there hasn't been a real restriction in programming that prevents developers from writing too much code, at least programmatically. Perhaps years ago when you had to code in punch cards and mail them off to be compiled, or wait for cpu cycles on a mainframe, then your development environment was limiting you. Nowadays, I think any language could be guilty of making it easy to slap around a lot of architecture or abstractions you may not need. I also agree fundamentally with his argument about certain programmers loving complexity. There is certainly some nerd pride there, and in some cases legitimate attempts by developers to make themselves too necessary to fire. However, for the sake of argument, let's assume all developers are acting with integrity. That they're not doing something malicious to guarantee themselves a job, but are honestly doing what they think is right.

In my experience, what has caused the majority of the type of code he describes in his blog is the statement I've heard every-time I've questioned overly complex code: 'I know that we're going to need to do X in the future' There's always something that the developer just knows for 100% sure will need to be done. Next month we'll need to be able to swap in a new implementation of this algorithm. There might be a performance issue, so we'll definitely need to be able to swap out this strategy. I know there could be a lot of references in response to this that: You ain't gonna need it. However, I don't think that gets to the heart of the issue. I think the more important question is what's causing them to think that they are going to need it? Why aren't they thinking about the complexity cost they're putting into the code? The technical debt they're adding. Or WTF count, or whatever. The fact that the next developer in 6 months has to figure out why you made something simple into a pluggable architecture, figure out how it works, and then try and bolt code onto it. Why?

Architecture Astronauts

One motivation certainly comes from the architecture astronaut camp. You can spot these types in a few ways. Sometimes they'll come out and say: "I prefer providing frameworks, not mucking around with business rules". They'll usually also exist in some kind of special architecture group that delivers 'frameworks' to 'business groups'. They're also at varying levels of the organization. Some 'enterprise architects' make decisions to use things like SoA or to split your web application into 10 war files or a host of other decisions that lead to complexity. Their heart is usually in the right place, and they honestly believe they're making things better, but they're usually so separated from delivering actual functionality, to actual end users, that they don't realize how much pain and complexity they're causing. They don't have to deal with the outcomes. There are also architects actually writing framework code. But when you're an architect, architecture is the solution to every problem. You look for additional places to add architecture. You also almost never have to actually use it, so you don't feel the pain. And finally, there's the cause I've heard most often: 'The average developers here aren't capable enough to work with [insert technology here] so we need to provide an architecture to make it easier for them". Even though it sounds like it, in most cases they aren't trying to belittle their coworkers. In many organizations architects are the senior developers and 'application developers' are the junior developers, at least from an experience perspective. There could even be a case where architects are on-shore and application developers are off-shore, which is a whole other level of organizational pain. However, my main point is that they live outside of delivering business value. Their job is to deliver complex frameworks. I'll leave out the organizational challenges that often lead to these positions being created, but in my experience its one of the biggest causes of overly complex code. I could talk for a long time about this particular subject, but I'll try and stay on topic and move on to the next cause of speculative generality.

Inability to effectively refactor the codebase

Even when working with an application developer who is focused on actually delivering working code to end users and truly wants to deliver it quickly, you will find they have implemented a crazy abstraction for no good reason. As I mentioned earlier in this post, the common reasoning is: 'I know I'm going to need feature x, so I created the strategyfatoryplugabblewidget'. Ok then you say, why can't you add that new feature next month when you need it? In my experience, the true reason is generally because they know they'll be afraid to next month. The codebase is finicky, there aren't a lot of tests, the feature they're working on right now will have been signed off on, whatever. Bottom line is, they won't want to change it in the future for fear of breaking the existing functionality. The overall system makes it difficult to refactor any code at all. So, anytime they put in any new feature, they try and future proof all of it, so they just need to add one new class, or extend one abstract class or implement some interface and wire it in with spring. Whatever it is, they never change or improve the existing codebase. Their coding is just one long exercise in bolting in new classes and fixing bugs in existing code. Sometimes the motivation can be worthwhile. They don't want to waste time 'refactoring', they want to provide new functionality. Of course, this generally leads to complex and buggy code, which eventually catches up with them, causing work to grind to a halt, but they don't want to think about that now. And in their defense, without reasonable automated testing or an army of manual testers, refactoring code is really really hard. And the longer you let things go, the harder it is to refactor. It's a vicious cycle.

So what's the conclusion here? Of course, keeping hammering on the yagni principles. Continue to push to make sure you're doing the simplest thing that could possibly work. However, make sure you code is reasonably easy to refactor. Encourage refactoring in whatever way you can. Although, it's probably a separate post in and of itself about how to work refactoring into a general feature development cycle, but it can be done. And ditch the architects. Everyone is delivering business value, everyone is feeling the same pain. And don't accept complexity just because the code works. It has it's own cost.

* Credit where credit is due, I got this term from Ram Singaram, a former colleague at ThoughtWorks.

Velocity Anchors

2011-01-26T11:37:00.000-08:00

A former colleague of mine from my ThoughtWorks days, Aaron Erickson recently wrote a blog post about Engineering practices and how they impact a project's velocity. The post is good, and talks about some common engineering practices to help improve velocity. However, the problem I've found on numerous projects, regardless of their level of engineering maturity is that they often don't see that they even have any velocity issues. The closest most projects get to is: 'We want to release by date x, with feature-set y and at our current velocity that isn't possible'. Aaron's post is a an example of positive ways to improve velocity, rather than the unsustainable practices of throwing more people at it or making everyone work 100+ hours a week. But what really is the proverbial anchor slowing the velocity ship down?

Technical Debt?

Martin Fowler wrote a recent blog post related to software craftsmanship where he talked about how much he hates using metaphors to describe software development. Technical debt is an often used metaphor for describing what is essentially causing velocity anchors in the codebase. This is of course completely separate from process issues. (7 design meetings and signoff sessions before even starting to code will kill the velocity of even the cleanest codebase) It's a tempting metaphor to use. Everyone understands the concept of debt from their own lives. Your income is severely handicapped if every month a large percentage has to go to paying down debt. The same can be applied mentally too, paying down debt every-time you code. However, there's a number of problems with using this metaphor.

People don't understand debt

Earlier I said "Everyone understands the concept of debt from their own lives" No, no they don't. Especially not in America. In many ways our attitude towards debt in our personal lives mirrors the problems in codebases. Can't afford the new TV? It's cool, Best Buy has financing. In fact, in many ways the concept of "I'm taking on acceptable debt now to get a feature out, and I'll totally going pay it off next quarter" is what gets most companies into trouble in the first place.

The code fairies don't send you a bill

A credit card bill is tangible. They mail you a piece of paper that says: "You have to pay this much right now or bad stuff is going to happen." And you know exactly what that bad stuff is. Everyone knows what the end of this path is, and speaking as someone who has recently watched family go through it, bankruptcy sucks. There is no such obviously painful end point to accumulating too much debt in your code. The only thing I can tell you is: you'll know it when you see it. When you've spent years working on a codebase, and have a small percentage of the features you want done, and your retention rate is falling rapidly, you'll know you have too much debt. However, its really hard to make someone understand that when they're staring at all the screens in best buy.

Software development is subjective

Is that piece of code technical debt or a really clever architecture? I bet you could take any random snippet from your codebase and ask two developers whether its technical debt or not and get two completely different answers. Usually because one of them thinks you'll "definitely need to let users add their own custom widgets in the future". (I'll save the speculative generality rant for another day) My point is that you could spend a lot of time trying to define what is debt vs. 'good architecture' and end up being one of those people who hasn't actually written code in years and has a calendar where every hour is triple booked.

Its not really that hard of a concept really. I'm not sure it even needs a metaphor. Your developers are trying to get stuff done. There's a bunch of crap in the way making that take longer than it rightfully should. If you care about developer productivity, you try and hunt it down and fix it. If you don't, then stop reading this and get back to your gant chart. I bet if you put one more developer on that task the earn/burn ratio will look way better. (Another rant for another day though)

So, what are some of the things getting in the way of getting stuff done?

WTF?

That's the sound your developer makes when they are looking at a piece of code and can't figure out what it's doing. it might be a bug fix or a new feature or whatever. The bottom line is that they're trying to get something done, but instead they spend hours digging through file after file trying to figure out why there are 7 variables named 'chart' that all mean completely different things depending upon which part of the application they're used in. It's a simple equation. The number of WTFs is inversely proportional to developer productivity. We could probably even make a Sonar widget for it.

Code Jenga

Have you ever touched a piece of code to make what seemed like a simple bug fix only to realize that you just committed an atrocity? One variable changes, and now the login page has no text? It makes developers afraid to do anymore than the bare minimum required. And if anything more is required, it takes forever, because they now have to climb the mountain of WTFs to accomplish the feature/bug fix.

I'm sure I could come up with some more if I kept thinking about it. But then I've seen plenty of teams brought to their knees because they spend the entire time arguing about which design pattern to use, or what's the most productive IDE, or which BBQ restaurant is best.

The only thing I've ever seen work is empowering developers* to make the right decision when the time comes. They know the codebase, they know what's slowing them down, let them make the decision. Yes, stuff still has to get done. Yes, releases still need to get out. But by and large, if you let developers code and focus on solving business problems, most of the time they'll make the right call.

*Architects aren't developers. If they aren't solving business problems, they're not a developer. I know, a bit hypocritical coming from someone who wrote a framework.

Unless you have a team full of developers who don't realize all the little things out of their day that's killing their velocity. 15 minutes to do a build? 7 properties files to update for each change? 10 Different components that all inter-depend upon each other and require a certain build order for no good reason? People get used to the craziest things, and unless someone points it out to them, won't even notice. And even if you do, they might just shrug and say: 'I don't mind'. Then again, telling them they have too much technical debt doesn't help much either.

And maybe that's my point really. Regardless of all the engineering practices or programming language, or IDEs, there's no guarantees. Switching to Ruby won't make your project ship on time. Using TDD doesn't mean you won't have quality issues. Meticulously planning every second of every day of the project from now until completion doesn't mean you'll make that date either. The only thing you can do is focus on getting stuff done, hire the best, most passionate people you can, and hope for the best.

New Job

2011-01-26T11:28:00.000-08:00

This post should have been made a month ago, so I apologize for the delay. However, last month I left ThoughtWorks to pursue an opportunity a start-up called Fundspire. I have accepted the position of CTO there and will be leading the technical direction of the software. It's a small company, so that's really a fancy way of saying that I'll be the one coding. I plan to continue blogging and contributing to open source, with the same blessing from Fundspire as I received from ThoughtWorks. The Fundspire solution has been implemented in Grails on Amazon EC2, so there should be more posts in the future around those technologies.

Groovy Puzzler

2010-12-14T17:52:00.000-08:00

Consider the following code snippet:


def A = "A"
def B = "B"

def map = ["${A}":1, (B):2, C:3]

println map
println map[A]
println map[B]
println map["C"]
println map[C]

What will be output?

The first thing to consider is, what will groovy think the map is, given the 3 different ways entry are created. However, printing it will return:


[A:1, B:2, C:3]

Looks like it all ended up being the same thing, no?

Now consider what attempting to print a, b and c will give you:


null
2
3
Exception thrown

groovy.lang.MissingPropertyException: No such property: C for class: ConsoleScript27
 at ConsoleScript27.run(ConsoleScript27:10)

Now, let's think about what we've done here. With A, we've added it using the normal syntax for dereferencing an object inside of a string, by surround it with ${} inside the string itself, somewhat similar to JSTL. This is returning the string 'A' correctly, but when attempting to reference it, it's not found, thus returning null.

With B, the correct way to dereference and object was used, by surrounding it with a parens. This works as expected, and the referring object can even be used as a key.

C used the string property approach. That is, groovy treated it as the literal string 'C'. This is why referencing it as a ["C"] worked and [C] caused an exception.

The following groovy page contains all you need to know about map manipulation in groovy: http://groovy.codehaus.org/JN1035-Maps

Next, let's consider some of the very useful operations that groovy adds to normal java collections. In this case, max(). What will the following return?

def someList = ["75", 32, 3.2]

println someList.max {item ->
 return null
}

The answer is "75". This is because it's the first item in the list. If the closure passed to the max method returns null, the first item in the collection will be returned.

Now, imagine the following scenario. You mess up how you reference the map, inadvertently causing null to be returned for all calls. You then use this map in a closure for the max method. You don't catch any of this because in your unit test the first element should be the max. Now, imagine the list you're operating on was obtained from a database via GORM, which means the order will vary from call to call, since there is no order guarantee without an orderby clause, which isn't there. So, on the webpage you'll see the value change everytime you call it, but when you debug, the map will look correct. Hours later, you'll discover that you didn't create the Map correctly, even though inspecting it in debug, and printing it out in unit testing will show that it looks correct.

While I don't know Groovy that well yet, I can understand at least some of the mechanisms that would lead to that behavior. However, I wonder two things:

Why does the map print out the way it does? I need to do some more digging into the Groovy Map implementation to figure out what underlying mechanism is causing it, but it seems like there could be some kind of way of determining how it's being stored. Meaning, there's obvious differences between the 3 entries, but you won't see it visually when printing, and you won't be able to determine it easily in a debugger. (Although, I may have missed something important when looking)
Should the closure returning null for every element behave this way? I suppose there isn't much way around it, when you consider a scenario where null is returned in some but not all cases.

Groovy MetaProgramming

2010-12-10T09:09:00.000-08:00

While working on a Grails project I started investigating groovy meta-programming. I wanted to be able to add one method per class for every Enum in the containing class. (I'll have an example below)

The official groovy documentation on the topic is here: http://groovy.codehaus.org/ExpandoMetaClass

Most examples show something like the following:


//Add new method call printHello
SomeClass.metaClass.printHello = { print "Hello" }
def instance = new SomeClass()

Which of course when run will print "Hello".

However, I didn't know the method was called at runtime, I just wanted to add one for every entry in an enum. It's not hard, but there isn't an example out there, after a little playing in the groovy console, I figured it out:


enum Status {A,B,C}

class User{
   def status
}

Status.each{ statusEnum ->
   User.metaClass."is${statusEnum.name()}" = {
       return status == statusEnum
   }
}

def user = new User()
println user.isA()
user.status = Status.A
println user.isA()
println user.isB()
user.status = Status.B
println user.isB()

While will output:


false
true
false
true

Of course, this isn't exactly complex, and I may find that I don't like how this turns out, but I couldn't find an example of this type of Groovy meta-programming example, so I thought I would share.

Maven and Continuous Delivery

2010-11-09T21:30:00.000-08:00

There has been an interesting discussion going on recently in the Maven user group list about Continuous Delivery:

http://maven.40175.n5.nabble.com/Continuous-Delivery-and-Maven-td3245370.html

My colleague here at ThoughtWorks, Jez Humble, contends that the SNAPSHOT model that Maven uses is prohibitive to many of the principles espoused in his book. And I have to say, I largely agree with him.

Before diving into why I agree with him, it's worthwhile to add a bit of context. I'm one of a minority of people in ThoughtWorks that prefer Maven to Ant. That's not to say that we don't deliver projects using Maven, but if given a choice, most ThoughtWorkers prefer Ant. Personally, I have had a long history of using Maven, starting with my work with Spring Batch. That's not to say that Maven doesn't have it's problems, but if given a choice, I'll start a project with Maven. (Why that's the case would certainly be worthy of another blog post) I mention all of this because I am not bringing up issues with the release model of Maven to bash it because I hate the tool. I bring it up because I think there can be improvements in how Maven thinks about and deals with releases.

I remember the first time I had to cut a release with Maven oh so many years ago. I had my project working, everything was in exactly the state I wanted it to be into for release. All I needed to do was create the final cut. Like a good Maven user, I went to check out the documentation for how Maven approached releases. I quickly found the section on the release plugin and thought 'Aha, I just need to use this plugin, and my release will be painless, thanks Maven!' Of course, reality is never that simple. The maven release project has two 'phases' prepare, and then the actual release. Below is a list of what Maven does during the prepare phase, shamelessly pulled from the official plugin site:

Check that there are no uncommitted changes in the sources
Check that there are no SNAPSHOT dependencies
Change the version in the POMs from x-SNAPSHOT to a new version (you will be prompted for the versions to use)
Transform the SCM information in the POM to include the final destination of the tag
Run the project tests against the modified POMs to confirm everything is in working order
Commit the modified POMs
Tag the code in the SCM with a version name (this will be prompted for)
Bump the version in the POMs to a new value y-SNAPSHOT (these values will also be prompted for)
Commit the modified POMs

The SCM related bits make sense. Of course you need to create a tag that coincides with the release. However, the majority of the remaining bullets are around modifications to the POM files themselves, and checking that other POM files aren't in a 'SNAPSHOT' state.

I didn't understand that process years ago, and I still don't understand it today. I have my codebase, I've tested it thoroughly, both with unit, integration, automated, and manually testing. I just want to take the 'binary' that I've been using, and promote it to production while also taging the SCM revision that created it. Why does a snapshot state even matter? I suppose with external dependencies there are some concerns about APIs changing underneath you. However, what does it matter, you have working code, right now, surely the changing API is a problem for tomorrow, not the release of today?

In my time at ThoughtWorks, and other consulting experience, I have always prognosticated that the same build that a developer makes on his machine, and ultimately what will come out of CI (because he or she commits that revision) should look as identical as possible to what will be in production. Why add an extra step to change anything at all?

In short, I agree with Jez, why can't every artifact created from mvn deploy be potentially a release into production? The only value I've seen the snapshot concept add is away to describe a dependency as 'whatever the latest CI has produced is'. It seems to me its just as easy to say: 'any release greater than x.y' and call it a day, which is something Maven supports now as well, at least from a dependency declaration perspective.

Spring Batch Deployment Example

2010-07-15T14:14:00.001-07:00

Deployment has always been a tricky part of Batch Processing. Unlike a Web application, there is no standardized deployment, and there are a variety of environments that could be deployed to. You could actually deploy a Batch application within a web container, or as a standalone java app to be started by one of many available schedulers. For this reason, everyone's environment is different, and there can be no one example that someone can use as a starting point. However, I have done enough standalone deployments for Linux using Bash to share a simple example.

The job itself matters little for the purposes of this post. A simple one tasklet job suffice:

   <job id="sampleJob" job-repository="jobRepository">
       <step id="simpleStep">
           <tasklet ref="tasklet" />
       </step>
   </job>

   <beans:bean id="jobRepository" 
   class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"/>

   <beans:bean id="transactionManager" 
   class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />

   <beans:bean id="jobLauncher" 
   class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
       <beans:property name="jobRepository" ref="jobRepository" />
   </beans:bean>

   <beans:bean id="tasklet" class="net.lucasward.sample.SampleTasklet" />

(namespace removed for readability)

The Tasklet simply prints 'Hello World'

public class SampleTasklet implements Tasklet {

   public RepeatStatus execute(StepContribution contribution, 
                               ChunkContext chunkContext) throws Exception {
       System.out.println("Hello World");
       return FINISHED;
   }
}

Simple enough, right? Now for the hard part. If we want a scheduler to be able to launch this from the command line, what do we do? The first problem to solve is, what does our deployment look like? In my mind, there's three necessary components:

The jars themselves, which need to be on the classpath
A Script that can be called
The xml and/or .properties files that will be used for configuration

Personally, I prefer to separate the XML files from the jars, to allow for tweaks to the Job. This is especially useful for Spring Batch job definitions, although may be less so for normal spring configuration files. My preferred layout is to have three directories: bin, lib, and resources. Obviously, the scripts go in /bin, jars in /lib, and xml/properties in /resources. It doesn't really matter how you break yours up, but it's the format I'll be using. In order to create this layout, I'll use Maven and the assembly plugin:


          <plugin>
               <artifactId>maven-assembly-plugin</artifactId>
               <version>2.2-beta-2</version>
               <configuration>
                   <descriptors>
                       <descriptor>src/main/assembly/descriptor.xml</descriptor>
                   </descriptors>
               </configuration>
               <executions>
                   <execution>
                       <id>make-distribution</id>
                       <phase>package</phase>
                       <goals>
                           <goal>single</goal>
                       </goals>
                   </execution>
               </executions>
           </plugin>

And the Descriptor:

<assembly>
   <id>distribution</id>
   <formats>
       <format>tar.gz</format>
   </formats>
   <includeBaseDirectory>false</includeBaseDirectory>
   <fileSets>
       <fileSet>
           <directory>src/main/scripts</directory>
           <outputDirectory>bin</outputDirectory>
           <useDefaultExcludes>true</useDefaultExcludes>
       </fileSet>
       <fileSet>
           <directory>src/main/resources</directory>
           <outputDirectory>resources</outputDirectory>
           <useDefaultExcludes>true</useDefaultExcludes>
           <filtered>true</filtered>
       </fileSet>
   </fileSets>

   <dependencySets>
       <dependencySet>
           <outputDirectory>lib</outputDirectory>
       </dependencySet>
   </dependencySets>
</assembly>

The format I'm using is tar.gz, since I'm targeting linux, but there are many more available.

The two fileset entries describes both the bin and resources directory. (I haven't talked about the script yet, but I will below). The 'DependencySet' reference is to transfer all the dependencies that Maven is managing into the lib directory, including the created jar itself. When you run 'mvn install' there will be two artifacts created: The normal jar, and another file with the same name, ending in -distribution.tar.gz. In the case of my example that is: batch-deploy-sample-1.0-SNAPSHOT.jar and batch-deploy-sample-1.0-SNAPSHOT-distribution.tar.gz. In my example, unzipping gave me the following:

./bin:
sampleJob.sh

./lib:
aopalliance-1.0.jar                           spring-batch-core-2.1.1.RELEASE.jar           spring-tx-2.5.6.jar
batch-deploy-sample-1.0-SNAPSHOT.jar          spring-batch-infrastructure-2.1.1.RELEASE.jar stax-1.2.0.jar
commons-logging-1.1.1.jar                     spring-beans-2.5.6.jar                        stax-api-1.0.1.jar
jettison-1.1.jar                              spring-context-2.5.6.jar                      xpp3_min-1.1.4c.jar
spring-aop-2.5.6.jar                          spring-core-2.5.6.jar                         xstream-1.3.jar

./resources:
jobs

./resources/jobs:
sampleJob.xml

All we need now is a simple script to actually run the job:

#!/bin/bash

CP=resources/

LIB=lib/*
for f in $LIB
do
CP=$CP:$f
done

java -cp $CP org.springframework.batch.core.launch.support.CommandLineJobRunner \
jobs/sampleJob.xml sampleJob

I'm probably not going to win any awards for my bash scripting skills anytime soon, but it gets the job done and isn't quite as archaic as a more concise version would be. Essentially, I'm creating a classpath from all the jar files in /lib by looping through the files and separating them with a colon. Once that is done, I can start a java process, using the CommandLineJobRunner as the Main method. As described in the documentation, all I need to pass to the job runner is the xml file to run the job, and the job name. (It's worth noting that normally you would also need JobParameters, but since I'm using the MapRepository, it isn't necessary)

You can download the running example from my github account: batch-deploy-sample.

Test code quality

2010-05-24T21:41:00.001-07:00

I've been meaning to write about this for awhile, but it always seems so contentious for some reason. I'm sure we've all seen the worst examples before. I once opened up a 5,000 line unit test of a Spring MVC controller. It got this bad because every bit of setup was cut and paste into each new test function. Of course, it should have been a sign to the original author that refactoring was needed when there was that much setup required in the first place. But it still begs the question: What level of quality should be expected of test code?

I've worked with lots of great developers that would see so much as a line of duplication and take it upon themselves to drop everything, including the donut in their hand, to remove it from code. However, when it comes to test code, that same developer would shrug their shoulders and say: 'It's only test code'. I just don't get this. If lack of reuse and duplicate code would cause an aneurysm in normal code, why is it acceptable in test code? For that matter, why would the way you treat test code be any different at all?

When we write unit tests, it's code we intend to last just as long as the class it's testing, or at least until the assumptions we made about the code have changed. And given how slowly some companies could change even those basic assumptions, that could mean never. Testcode still needs to be easy to maintain, and easy to understand.

Here's some of the reasoning I've heard:

I don't want to make the unit test complicated, it should be easy to understand the intention

If abstractions and code reuse done properly can make code easier to understand, why would the same not apply to test code? How would duplication between two unit test classes aid in the clarity of the individual test?

Unit tests should be completely isolated from each other

I'll agree to this in principle. But I think it can be taken too far. Unit tests should be able to be run side effect free. You should be able to run each individual test by itself, or with any number of other unit tests without the existence or absence of such tests causing failures. However, this has nothing to do with sharing abstractions with each other, or reusing the same setup data if more than one class under test uses the same domain objects. As long as you're creating new instances in your test classes, and not accessing them through stateful static calls, there should be no issue.

Modifying one test to pass shouldn't cause another to break

This is a tricky one that has bitten me numerous times before. If you create some kind of abstraction, such as some type of a builder to create a set of test data for multiple tests, you run the risk of breaking a lot of tests if you change something in the builder to make one pass. It can be extremely disheartening to watch the unit test you were working with pass, only to see hundreds of test failures when you run the entire build.

However, despite the pain this has caused me in the past, doesn't this seem like a normal coding problem? Maybe it's coupling/cohesion, or demeters law, or any other random law. There's probably a perfectly well known refactoring mechanism to make this test code easier to modify without side affects. There is no reason not to apply them to your unit test code as well.

Now though, there comes the dirty part, that only an 'Enterprise' developer will know the unfortunate reality of. What if your test code looks nothing like your real code, but there's nothing really wrong with it except it breaks many of the largely nonsensical 'coding standards' put in place by some type of 'governing body'? What if *gasp* its better as a result?

Boiler Plate code and closures in Java

2010-05-19T12:22:00.000-07:00

While working on a project recently, I ran into the following code:


public SomethingFromDatabase getSomethingFromDatabase(){
  Element element = getDataFromCache();
  return (SomethingFromDatabase)element.getValue();
}

private Element getDataFromCache(){
  Element element = cache.get(KEY);
  if(element == null){
    element = new Element(KEY, dao.getSomethingFromDatabase());
    cache.put(element);
  }
  return element;
}

Assuming cache is an instance of net.sf.Ehcache and the dao is your run of the mill Dao.

There's some obvious problems with this code, the one that stood out the most to me is the boilerplate code. The solution I used to solve the problem, which is common, is to use closures:


public  T get(String cacheKey, CacheDataClosure closure){
  Element element = cache.get(cacheKey);
  @SuppressWarnings("unchecked")
  T cachedObject = element == null ? null : (T) element.getValue();
  if(cachedObject == null){
    cachedObject = closure.getData();
    cache.put(new Element(cacheKey, cachedObject));
  }
  return cachedObject;
}

public interface CacheDataClosure {
  T getData();
}

A similar approach is used by Spring in it's *Template classes. Refactoring the original code to use the Facade, would look something like:


private Foo getDataFromCache(){
  return cacheFacade.get(KEY, new CacheDataClosure(){
     Foo getData(){ return dao.getFooFromDatabase(); }
   });
}

I'm sure to some people reading this blog, the solution seems obvious, and almost not worth mentioning. However, over the years I have seen many interesting solutions to the same problem, especially in the database world, but elsewhere as well. You would be surprised how many people solve this problem with inheritance by instinct. Even those who have used frameworks that require a closure in certain scenarios, don't necessarily think about what it is they're using, and don't think to use it when a problem arises later. Despite the fact that closures are ugly in Java, they are still the best solution for allowing for separation of concerns in a lot of scenarios, and reducing boilerplate code.

SCM Continued

2010-02-19T16:27:00.000-08:00

In my last blog post I talked about a maturity model for source control systems. Martin Fowler has recently posted a blog entry that covers the same topic:

http://martinfowler.com/bliki/VersionControlTools.html

One of the reasons that Paul and I thought a maturity model for SCM was interesting, was that it brought objectivity to the debate.(or at least tried to) It seems far too subjective (and easy) to break SCM systems down into two categories: usable and unusable. For me personally, I would list ClearCase in the 'unusable' category. However, despite it's shortcomings, thousands of programmers use it on a daily basis. Therefore, it can't be 'unusable'. What it boils down to though, is that for me personally, given what I find important when developing, ClearCase is a tool that I feel prevents me from coding effectively. If someone else doesn't hold these same values to be important, perhaps it changes what they view to be usable.

Martin takes an interesting twist with his article, in that while he still uses two groups, his dividing line is on 'recommendability'. This is still a somewhat subjective measurement, especially since his focus group was us ThoughtWorkers. But after reading the responses to my last blog post, a common theme seemed to come from those using ClearCase. They were willing to defend it as 'usable', but not a single one that I read would recommend it. This mirrors some of my project experience as well. I've even known some ClearCase admins that would argue with me at length about how ClearCase isn't that bad, as long as you 'used it right', and yet even they wouldn't recommend it as a source control product. Perhaps it's much more useful to ask a client if they would recommend their SCM to others, rather than arguing about why you don't think it's the best solution. Afterall, if they wouldn't recommend it to others, why are they still using it?

A Maturity Model for Source Control (SCMM)

2010-02-03T20:52:00.000-08:00

Most enterprise developers are already familiar with the Capability Maturity Model. Those of you with Agile tendencies may have also heard of the Agile Maturity Model. The purpose of these models is to objectively assess an organization's maturity in a particular methodology. Despite any feelings you may have on CMM or waterfall in general, having an agreed upon way to assess the basic principles of a methodology can be quite useful. One area where this type of assessment typically falls under the radar is SCM. Many companies look at a particular SCM solution, and believe it covers every need. However, when objectively analyzing the various uses and features of SCM systems, a type of maturity model begins to emerge.

As with other maturity models, we can use a similar numbering system to delineate how 'mature' an organization or tool is, with a higher number being better. As you can imagine, zero corresponds to a complete lack of source control at all, or perhaps a shared file system and a roofing shingle. The highest level is reserved for the most advanced tools such as Git or Mercurial, that allow for advanced SCM capabilities such as zero-cost branching and merge through rename.

In doing this analysis, some common feature-sets emerge:

Atomicity: If you check in 5 files, and there is an issue with one, none of the five should be committed.
Revision Tagging: The ability to take a snapshot of the state of your codebase at a particular moment in time, and refer back to it later.
Branching: The ability to have more than one version of the codebase in active development, seperate from each other.
Merging: Taking different versions of the same codebase and combining those changes. This may mean changes from a local copy or between two branches, with a complete history of how these various pieces have been merged.
Merge through Rename: A more advanced SCM feature for dealing with merging files that have been renamed. Consider file A in branch 1 and 2 that is then renamed to B in branch 2, but not in 1. Then consider what would happen when A's contents are changed in branch 1. SCM systems that support merge through rename allow for this and more, all others blow up.
Repository Navigation: The means of accessing the contents of the repository. Subversion, for example, has a web interface. (among many other third party tools) While Clearcase has thick client access.

Lack of SCM Bravery

Most folks in enterprise computing as of 2003, had experienced two SCM tools. It does not matter which two. The important point is that they remebered which of the two was the better one, and which was the OH-MY-GOD-I-NEVER-WANT-TO-USE-THAT-AGAIN one. They had taken that lesson, and were going to make damn sure they were never going to use the lame one again. Sadly that meant there were risk averse in respect of a third or fourth choice. Perhaps also as all the tools chains and workflows are different, there was caution based on that too.

Much has changed since then with experience of half a dozen much more common place now. However, if you walk into most enterprises today, you're likely to see only one of two SCMs: Clearcase or Subversion. For those of us who have used many of the SCM systems below, (especially us consultants) it is hard to understand why someone would continue to use some of the SCM systems that are lower on the maturity model below. It can be easy to think that it is imposed upon them by the powers that be, however, in many cases they will tell you at length why their SCM is better, or at least why they don't feel the need to switch. This stems from two fundamental issues:

They're pervasive. Like an IDE, they're there every second of every day, everything we do as programmers short of documentation needs to be checked in. It has a huge impact upon our day to day lives.
Steep learning curve. Regardless of the solution, and any third party tools helping to make it easier, source control in and of itself is complex. Switching requires that you accept a drop in productivity, and no one wants to become a newb again.

Five maturity Levels

Shadowing the CMM, we're aiming at five levels above zero.

Level Zero - 'No SCM'

No source control solution at all, or a shared file system with periodic backups. One developer, or a few at most share source without tools, and as such run a number of risks:

Source may not be compilable at any moment
Source may be lost because of developer error
Extremely easy for developers to overwrite each others changes.

There is nothing to redeem in this level, or build on for future ones.

Level One - 'First Foray'

Developers have workspace on network and cannot work offline
Run a build means go to a long lunch
Refactoring - if it works at all, is deathly slow
Checkouts slow enough to be done over night
Checkins slow
Non-atomic commits
Branching and tagging expensive
Personal/local branching means second checkout
Centralized rather than distributed
Unusable or slow merge point tracking
Merge through rename - merge, rename not understood by tool
Repository can corrupt on occasion, high administrator/expert to developer ratio (1:10)

Tools with basic ability to checkout, version and lock files. Usually implies developers are working on the same code. Synchronization to head my be problematic depending on locked status of individual source files. Tools in this space may have scaling issues, and not work well over long distances. Renaming of resources may be hard to impossible. Branching and tagging may require permissions on triplicate stationary, and with a slaughtered lamb or incense stick or two (they take a while and eat disk).

Examples:

Visual Source Safe

The core of VSS's problem is that it is not client/server and has not moved forward much in a number of years. Developers can hurt each other with locking files and even marking files as shared (an esoteric feature).

Clearcase's Dynamic mode of operation

Though an enterprise tool that sold widely into enterprises, it is like wading through molasses (treacle for Brits) to use. It mounts a network share or two for you, and thats what you edit on, and compile against. Both IDEs and builds are slow as a consequence. Dynamic checkouts are also precarious as you may have more than one developer active whithin it, and for periods of time, you may observe the codeline to be non-compilable. Furthermore, it does not support atomic commits. Though three way merge is actually advanced, nothing else about Clearcase in dynamic mode (including UCM mode) makes you want to particularly recommend it to anyone for any purpose. Merges from one active branch to another can actually take longer than the time spent making the original commits to the donor branch. There is a inverse square law at work with these installs of Clearcase, in that capacity drops off exponentially as you add more staff to it, and attempt to get busier (more throughput) with it. Clearcase sales folks and consultants recommend this mode of operation. Anomalous for Clearcase (dynamic mode) when rated against the bullets above, is that it can merge through rename, which is the preserve of more advanced tools (see later).

Level Two - 'Clunky'

Developers have local copies and can work offline
Local file system means fast(er) builds
Refactoring - will go through; make a cup of tea
Checkouts will trickle past as if in bullet time
Checkins potentially still slow
Non-atomic commits
Branching and tagging potentially expensive
Personal/local branching means second checkout
Centralized rather than distributed
Unusable or slow merge point tracking
Merge through rename not working, requires extensive follow up conflict resolution before commit
Repository can corrupt on occasion, administrator to developer ratio sub optimal (1:20)

Examples:

CVS

For the longest time, CVS was the default internet-community SCM server. It has a variety of network protocols, and runs at a predictable speed over an arbitrary distance. It uses optimistic locking (it kinda introduced this), and developers find working offline a breeze. Commits back can still be like pulling teeth, in that every directory in a source tree is analyzed for differences with the server version via a wire call (just a hash but even so). Sadly, branching and tagging are still costly. Some folks still prefer CVS over others listed below.

TFS

Microsoft have tried to make something in Perforce's image (they historically ran a private fork of Perforce from the 90's called Source Depot), but do not have all of the features and performance of Perforce. They rammed in too many non-source control features and it is falling short of the mark in terms of installation and administration costs. Its pretty much wedded to Windows developers, and correspondingly leverages a ton of other MS server side pieces. Day to day integrated operations in Visual Studio are where you feel forcibly slowed down versus not using SCM. Agile folks approach checkout/checkin/conflict resolution with dread. This tool should be higher ranked based on features, but is a let down in implementation.

Clearcase static mode of operation

This is where Clearcase checks out a branch to your C:\ drive. Much like CVS, but a little faster. Builds run as fast as is possible for a local IDE drive (a gazillion times faster than a overloaded 10-base-T ethernet network with 50 devs on it). You're not going to get hosed by some other developer rendering the branch non-compilable (phew!). At least not unless they checked in that broken state. You don't automatically keep abreast of communal efforts - you have to sync/update periodically. Clearcase sales folks and consultants do not recommend this mode of operation, for some strange reason. Anomalous for Clearcase (static mode) when rated against the bullets above, is that it can merge through rename, which is the preserve of more advanced tools (see later).

Level Three - 'Basic'

Developers have local copies and can work offline
Local file system means fast builds
Refactoring - speedy; smile at your neighbors for a second
Checkouts might finish before you die of old age
Checkins very fast
Atomic Commits
Lightweight (cost free) tagging and branches
Rudimentary branching and merging
Personal/local branching means second checkout
Centralized rather than distributed
Rudimentary merge point tracking
Merge through rename not working, requires extensive follow up conflict resolution before commit
Repository can corrupt on occasion, low administrator to developer ratio (1:100)

Examples:

Subversion

This is now the defacto standard install for enterprises, and since 2003 has been viable enough for the majority of open source SCM portals to install it, or at least have plans to. Compared to CVS its a definite advance. There is much improved speed. That is for normal checkin checkout operations and branching and tagging. It is a comfortable place for Agile developers to feel that their love for throughput is requited. Just don't try to do frantic parallel development on more than one branch.

Level Four - 'Effective and Reliable'

Developers have local copies and can work offline
Local file system means fast builds.
Refactoring - speedy; smile at your neighbors for a second
Checkouts reasonably fast.Checkins reasonably fast
No-op sync/update very fast
Atomic Commits
Lightweight (cost free) tagging and branches
Advanced branching and merging
Personal/local branching means second checkoutCentralized rather than distributed
Sophisticated merge point tracking
Merge through rename only possible with configured branch mappings, otherwise a fix-up before commit is required
Repository corruptions very rare, very low administrator to developer ratio (1:1000)

Examples:

Perforce

Fast is what this ten year old tool was built to be. In some operations nothing beats it. Back in the day, Perforce heralded a number of firsts in terms of capability (atom commits, lightweight branching/tagging, three-way merges).

This is bound to be controversial at 'level 4' as P4 uses read-only locks extensively.

Some Agileists are going to violently object as this mandates good tool support (Intellij juggles the read-only flags via perforce, but Vim does not). This severely limits the ability to work offline (making it anomalous versus the definition for level 4). You can still do it, you have to blast away the read only flags, and do a revert-unchanged when you reconnect later, but don't try to do renames while offline - it'll end in tears. Also refactorings (when there is tool support) will work, but will be slightly slower than for the likes of Subversion as network IO is happening per changed file.

Level Five - 'Speedy, Invisible, and Highly Capable'

Developers have local copies and can work offline
Local file system means fast builds
Refactoring - speedy; crick your knuckles momentarily
Checkouts reasonably fast
Checkins reasonably fast
No-op sync/update very fast
Atomic Commits
Lightweight (cost free) tagging and branches
Advanced branching and merging
Highly efficient local/personal branches
Distributed rather than centralized, full audit on consumed contributions from distributed sources
Seamless merge through rename - no configuration needed
Sophisticated merge point tracking
Repository corruptions very rare, almost invisible administrator to developer ratio (1:10000)

Examples:

GIT and Mercurial

These two are very similar, and both have their fans. It is difficult to tell which will win out over the other in time. For both, the killer capability for Agile folks who treat previously committed code like wet-paint, is merge through rename. It works so well that you feel this is how Fowler intended refactoring to feel and that all other SCMs fall short on that vision. In short Git and Mercurial have the speed of Perforce, plus easy local branching, plus distributed operation, plus merge through rename.

Level Minus One - 'Death Wish'

There is a special place in hell for PVCS Dimensions. We don't know about the latest version which is rumored to have atomic commits, but we'e pretty sure that the turn of the millenium version was the worst of Clearcase but with a bucket full of suck thrown into the mix. For example, person A doing a sync/update in the morning would take 45 mins, but if person B started their sync/update after person A, then 1.5 hours would be the reality (even if nothing changed). Sometimes things were so bad, an ad-hoc distributed mode would leap into being (devs putting sets of changed files on network mounts, floppy disks or emailing). As far as the authors can recall, there is 100% correlation between PVCS-Dimensions use and failed projects.

Anomalies

Subversion with Git as a front end. Git has built in support for Subversion servers. Given some time, Git can clone a whole Subversion repository to a local workspace. From there you get the quick branch juggling capability, as well as local commits that can be sent back to Subversion later via 'dcommit'. Speedy branch local juggling (a level 5 gain) is possible just like for Git proper. However, Git as a front end for Subversion cannot participate in Subversion merge point tracking (yet), thus we cannot back-implement another level 5 feature - merge through rename. Lastly, Git fronting Subversion is still mostly tied to a single server, so cannot claim to be distributed in the text book sense of the word. Where should it be placed ? 3.5 or 4 perhaps in terms of maturity, but merge lets it down.

Seamless merge through rename is the high bar

You're most likely to be using Clearcase in the enterprise today, and may have heard that a shift to Subvesion will be more productive (and a lot cheaper). Though Subversion is enterprise approved now and taking over from Clearcase as #1 soon enough, it is time to look for new tools. Git and Mercurial mark the high bar now in many ways. However one feature stands out for Agile teams chasing high throughput - seamless merge through rename. If Perforce wants some of its old empire back, it is going to have to do this feature which could well be hard for them. The Subversion team already has it scheduled.

This blog post was the result of much discussion and pair blogging with Paul Hammant.

Spring Batch and DDD

2008-11-02T18:11:00.001-08:00

I use Google blog search (http://blogsearch.google.com/) to keep track of anyone blogging about Spring Batch. Usually, blog posts are 'getting started guides', with simple examples, which is understandable given the way our samples are organized. Our samples are indented to test nearly every framework feature in a real batch job, and serve as a form of functional test. This is good for developers that understand the framework and want to know how to use feature X, but not so good if you're new to the framework. I always intended on creating a simple example, but I think the blogging community has been so good about creating them, I'm not sure we could add much value.

I'm slightly digressing though. I recently ran into a much different blog post about Spring Batch:

http://java-chimaera.blogspot.com/2008/11/spring-batch-domain-driven-design-in.html

The main theme of the post is that Spring Batch serves as almost a reference implementation for DDD:

"By applying DDD in Spring Batch, we now have a realistic and elaborate example of concepts like ‘repository’, ‘factory’ or again the ‘ubiquitous language’. What’s also interesting in this implementation is that, by being a technical domain, developers can easily comprehend the concepts and identify them more easily."

In nearly every presentation Dave or I has given about Spring Batch, we usually start describing the domain model, and express that Eric Evans' book was highly influential in how we designed Spring Batch. I can't begin to tell you how much this model changed from when we first start until 1.0. In the original model, Steps were called StepControllers, and so on. As we continued through development, they naturally began to evolve. We were very conscious of Evans' concept of Ubiquitous Language, and as we noticed ourselves continuing to refer to a concept as something, we usually decided that the class should be called that, which ultimately let to the simple Job and Step in the framework today.

I could talk for days about the evolution of the domain model in Spring Batch, but in the interest of keeping this post reasonably sized, I'll simply summarize by saying that DDD was a big driver in the creation of Spring Batch, and it's interesting to me that others unrelated to the project can see this by simply inspecting the code. I think it speaks volumes for the approach.