Code Quality

1 09 2008

What do you do if you have no idea who might use your code, and you want to be able to sleep at night ? Simple, you make certain that all the lines of code you write for all the situations you can think of are covered by a unit test, so you dont get a 4 am call out to something that you broke. Fortunately open source code doesn’t come with a Blackberry you have to answer or a pager thats always on, but that doesn’t mean that there isn’t a sense of responsibility to others that use your code. Even if what you write disappears into a corporation and supports perhaps millions of users, I still feel a sense that I should make every effort to make it as solid and reliable as I can.

A level of testing that allows me to sleep can’t be achieved by just writing unit tests. There needs to be a number of tools to help, and a number of additional concepts. At the simple level, unit tests generate code coverage and high levels of coverage like Shindig’s social API see http://people.apache.org/~ieb/shindig/shindig-parent/shindig-social-api/cobertura/index.html  generate increasing quality. The code is know to work before it hits a server environment and patches don’t introduce quite bugs that only become noisy in production. Cobertura or any other coverage reporting tool is vital in identifying lines of code that haven’t been tested or scenarios that are not covered. 

The second tool to generate quality is FindBugs, that will tell the developer about bad or dangerous code patterns. Having reports running as part of the build gets these reports close to the developer, and once you have added FindBugs annotations the developer gains control over false positives.

To a lesser extent Check Style reports like http://people.apache.org/~ieb/shindig/shindig-parent/shindig-social-api/checkstyle.html  can also identify overly complex or potentially buggy code.

All of these tools and reports are great at improving code quality, but there is one increasingly important area where they fail. Multicore processors are encouraging more and more use of parallelism within our applications. Scaleable web applications are starting to consider overlaying network wait times within a single request to minimize the request latency. With this inprocessor parallelism, single JVM testing cannot simulate or test the real randomness of an application under load, as the JVM is inherently synchronized with the testing framework. To achieve realistic testing I would argue that there needs to be some randomness in the testing sequence and profile. Only with this randomness, can a testing framework expect to expose those race conditions that only happen when networked machine are communicating over unpredictable connections in a truly disconnected environment. 

Now I can sleep.





Shindig SPI Implementation

1 09 2008

A while back I started a Service Provider Implementation for Shindig aiming to store the data necessary to support a the OpenSocial API in a simple DB. This was before there was a SPI, now there is and it makes quite a bit of sense. There is a push to release Shindig 1.0, but the end of the month, and although a sample implementation of the SPI may not be in the core 1.0 release, I think it would make sense to have something done by then. Not least because it will keep us clean on maintaining the SPI, if we have an implementation of the SPI to break.

The Open Social Spec has grown in recent months, as more eyes have been over it. 0.7 was a relatively simple social model, but in 0.8.1 that has just been released, you can see the influence of input from the likes of MySpace and LinkedIn. All of this makes sense, and certainly makes the model more complete, but at the same time makes it all the more complicated. 

So how to implement the SPI. I looked at pure JDBC, but decided, due to the size of the model, this was going to take quite a time to make work well. I have used Hibernate in the past, but often found that it leads to a great object model, but really bad DB model that dies in production. There are more modern standards like JPA, but some of these have the same potential as older versions of hibernate to make a mess of the DB. A while back I used Apache Cayenne on another reasonably large scale project. What struck me at the time was how if made the developer focus on the construction of a great DB model that would perform at scale, and then drove the connection of that DB model into a Object model. 

For Shindig, I am using Apache Cayenne 2. I did  a v0.7 implementation with this stack and I am now updating this to 0.8.1. The big thing that strikes me is that I am designing the database from an entity model, and then performing Relational to Object Mapping, rather than building a Object model and finding a way to store it. The second thing, that strikes me, is that even though there are significant additions and changes between 0.7 ad 0.8.1 Cayenne is quick and easy to work with, leaving me free to make the optimizations I want to in the ORM conversions without demanding me to jump through hoops. 

What will be really interesting to see, is how the implementation process impacts the database design and scale up in production. At the moment, it looks nice, but as always with a DB, looking nice needs to be checked with lots of data and load at scale.