~ryanlowe/blog: Agile Data Serialization Management

Agile Data Serialization Management

Admittedly I don't know much about databases, so I'm looking for some input in this area. I do know a bit about agility though, and using a database for AudioMan concerns me on that level. I'm looking specifically for pointers to somelike like agile database development concepts.

Databases are a persistence mechanism, like regular files, and are basically a form of object serialization. You take an object and put it into permanent storage in some known format. Then you can go back to the storage and retrieve the object you put in from that known format.

The problem with agile systems is that the objects are changing much more often than in a system made with a non-agile big design up front (BDUF) process like waterfall. When you serialize objects they must be in a format that can be understood by some parser to deserialize them. If you serialize in format A and deserialize in format A' after a refactor in an agile process, you have breakage. So the serialization mechanism also needs to be agile.

Databases can be the same way. If you add or remove columns in a table the code can be adversely affected and break. You can detect these breakages with good unit testing -- that's not much of a problem. But the issue is creating a database schema that promotes agility. You don't want to be prevented from adding new features or refactoring when you need to because you are hamstringed by a restrictive database schema.

There is also the issue of updating the storage data on the user's machine, whether it be a database or a file format. In an agile system you need to be able to convert an old iteration's serialization or database format to the new one fairly painlessly. This is all part of maintenance and allowing for agility. If you cannot easily convert the formats, updating them will be too much of a pain for the users and the users will actively resist change. How can you get their feedback in a timely mannor if they constantly resist change? In an agile process, you want your users to actively encourage change based on their feedback! Having a painless and transparent update to the serialization format is ideal.

There are probably many agile projects that have to manage this issue but one that I'm familiar with is the Eclipse project and its workspace directory. Eclipse has a one-way conversion tool that runs whenever you install a new version of Eclipse and it converts the old workspace data into the new serialization format used by the new version. You can't go back to the old format from new format but that is rarely an issue. You could always just back up the old workspace before you update Eclipse and the restore it from backup if you needed it.

The Eclipse workspace directory contains all of the projects you're working on and all of your preferences, as well as local change history and lots of other working data. Given the size and scope of the preferences and other features that use the workspace directory, it seems like the Eclipse project would be a good case study in agile data serialization management. They seem to have it figured out pretty well.

The workspace stuff seems to be in the Eclipse "core" component.

Posted at March 24, 2004 at 05:30 AM EST
Last updated March 24, 2004 at 05:30 AM EST

Comments

Andrew reponded on his blog:

http://www.beernut.ca/andrew/archives/000701.html
http://www.beernut.ca/andrew/archives/000702.html

» Posted by: Ryan at March 24, 2004 07:03 AM

One way to not be bound to any database schema is to tear down and build your tables in a "all tests" suite. We have our scripts in a .sql file, run those, run the tests, ...

That way you only have the columns / tables you need at the time. AND everything is in source control, so you can roll back to an older version if you had to.

The thing that I like is that it's all in the same tool, and that you're still just looking for the green bar. ;-)

» Posted by: Jim at March 24, 2004 08:44 AM

Using databases can definitely be a problem when dealing with agile applications. Something really important is to make sure you keep your database in normal form. Think back to that database course we took. I think level 3 or 4 is all that's necessary. This will help when trying to extend your database to hold more information about something. make sure that if you think you may want something to be a foreign key in the future, that you make it one at the beginning, even if the table it's point to has only one useful field in it. It might really help with extending your database later on.

Also, remember to build a good database abstraction layer. you don't want to have to run all over your code, trying to find every place you accessed the database from. Ensure you have an API that doesn't have to change too much if you cange the database schema.

Also, you may want to look into object oriented databases. I'm not sure if these would be any good for you. I'm not really too familiar with them myself. Although I think it releives some of the frustration with serialization of data, as data is stored as objects. Last time I checked, OO DB's were pretty new, and weren't that popular. But things may have changed by now.

» Posted by: Kibbee at March 24, 2004 05:22 PM

Just for my own fun, I started doing a little research on OO Databases, and I found something that might help out. Apache Xindice is an XML database. AudioMan is currently using XML to store data, right? well, this is pretty much a database that stores XML. Might make it easier to manage stuff. Not sure how well it would work with AudioMan though. Here's a link to it:

http://xml.apache.org/xindice/

Let me know if it helps

» Posted by: Kibbee at March 24, 2004 05:40 PM

Good suggestions. The repository/database is already well abstracted. When I get recordsets I would convert them into objects before returning them (again, more deserialization/parsing) to the layer above.

There are object persistence mechanisms in .NET that do this automagically but I'm not sure about Java and SQL/JDBC.

Using XPath to query in Xindice worries me a little bit because I've never used it. Could take a while to learn how to use it properly, but it's worth looking into.

Other lightweight Java databases like hsqldb use SQL as the query format, and there's much less of a learning curve there.

Multiplatform installation is also a big issue that AudioMan's database will have to handle gracefully.

» Posted by: Ryan at March 24, 2004 05:54 PM

In Java, you can use entity EJB's and they will keep the data object and sql logic together. This is more of building apps with components quickly. You don't even have to have the EJB's on the same machine 'cause they reference themselves similar to CORBA. But that won't help you here 'cause they run in a container and are really for web apps (since they are part of J2EE spec. I believe).

A lot of enterprise Java that I've read about is web stuff, so that's where those come in.

Mind you, I could have things mixed up a bit since I've only read about them, and not actually used them.

» Posted by: Jim at March 24, 2004 08:59 PM

Jim: EJBs seem to be a standard format for objects so that they can be read by third party apps (via Java reflection). They do seem to have built in persistence support but it might be a stretch to expect "lightweight" databases to support them. I can hope though! :)

http://en.wikipedia.org/wiki/EJB

» Posted by: Ryan at March 25, 2004 07:46 AM