|« February 2005||April 2005 »|
I'm Ryan Lowe, a Software Engineering graduate living in Ottawa, Canada. I like agile software development and Ruby on Rails.
I write this blog in Canadian English and don't use a spell checker. Typos happen.
» Full-time Ruby on Rails freelancer
» Full-time with Rails since May 2005
» Former committer for RadRails (now Aptana)
» I also have a few Rails side-projects in development:
1. wheretogoinTO.com Toronto nightlife
2. Hey Heads Up! TODO list and sharing
3. Layered Genealogy family history research
4. foos for foosball scoring
5. fanconcert for music fans (on hold)
Hiring Rails developers? I can telecommute by the hour from Ottawa, Canada
»» Email: rails AT ryanlowe DOT ca
Now hosted on Hey! Heads Up -- check it out!
Derek Lowe's (Ryan's older brother) words at Ryan's funeral
email@example.com no more
Forging Email Headers: Good, Bad or Ugly?
Sarcastic Dictionary (Part 1 of Many)
Twisting Rails is Risky Business
Risky Business? My Take on Early Alphas
Whoa, it's August 2007
A Postscript to "Growth at the grassroots"
»» All Blog Posts
David Heinemeier Hansson
James Duncan Davidson
Signal vs. Noise
Amy Hoy: (24)slash7
Luis de la Rosa
# Eclipse: How to Run All Tests in Multiple Projects?
Here's a question to the Planet Eclipse audience...
I have 23 Eclipse projects (most of them plugin projects) for one product. Is it possible to run all of the JUnit Test Cases in those projects at once in Eclipse? In the run configuration all I can see is the option to run either a single Test Case or all of the Test Cases in a single project.
I suppose I could maintain a giant JUnit Test Suite class but I'd rather not have it span projects. My fallback right now is to run my product Ant build script and read the JUnit report with the consolidated results of all projects. The downside is that Ant test running is not integrated into Eclipse's test runner view so it is more mouse clicks to see the test results.
How have other people worked around this?
Maybe a good enhancement would be to run all JUnit Test Cases in a working set.
# Software Engineering and Feedback
Managing the AudioMan software project has reinforced an opinion I've had of software engineering for a few years: software project management is all about control. I'm not talking about control in the sense of social power, I'm talking about project control with feedback loops.
Feedback is what the other engineering professions use to control processes, and so should software engineering. I had that realisation while I was still taking electrical engineering classes and studying signal feedback loops.
There are two aspects to feedback: measuring and responding. You can't respond well if you don't measure well. It's actually kind of silly that we learned how to respond to software management problems better than we learned how to measure. Why? Because measuring a software project is still fairly new, and responding without measuring -- even incorrectly -- is not that new.
Software project management has a close relationship to business project management. Business projects have balance sheets -- a business project consumes and generates cash. The overall success or health of a business project could be determined largely by looking at cashflow over a period of time.
There's a business project wrapped around software projects to sell the software and pay the developers. Software projects consume resources (person hours) and generate code. The health of a software project is more directly related to the health of the code. The business project around it could very well succeed with an unhealthy software project inside of it but it is more likely to fail. It all depends on how each are managed and coupled together -- but let's just talk about software project measuring.
Unit tests are a good way to ensure that a project stays healthy. If you break an expected behaviour a unit test fails and you know about it. How quickly you know about the breakage depends on how often you build and test the project. See continuous integration and JUnit.
Code coverage analysis tools let you see weakly tested areas of code in the software project. Weakly tested areas won't catch breakages in the code as well as strongly tested areas. To make sure you get all of the benefits of unit testing throughout your project, you can monitor code coverage numbers. See tools like EMMA and Clover.
Sometimes it's good to know when people check-in code to the repository and what that code was. Maybe someone modified an area that can't be unit tested (ie. user interface code). A tester could identify that area as recently changed and optimize their overall testing effort by concentrating on this changed area of code for a while. Code that has recently changed is sometimes more likely to have unfound defects. See tools like CVSspam.
Monitoring tools have an interesting affect on developers: when developers know they are being watched, they are more careful. They don't want to embarrass themselves by breaking the build or holding the rest of the team up. This is a variation of the broken window theory. Tools make developers accountable for their actions and that's a good thing. Another benefit of close monitoring is that developers are more likely to learn from their mistakes if they get quick feedback instead of a bug report months later ... or never.
Monitoring tools enable flexibility, refactoring and ultimately development freedom. Tools like unit testing allow the development team to take bigger risks that are not practical without unit tests from a code stability standpoint.
When the project is monitored closely developers might get the sense that they have less freedom. This is the kind of thing that has to be disproved over time -- the tools proving themselves to be freedom-enhancers. It doesn't seem to be enough to give the benefits of project monitoring anecdotally because it is a different development paradigm.
Another reason? Personally accountability scares some people. How can that be worked out?
# Week 05 Status Report
This is the week 05 status report for AudioMan and its subprojects.
What was done last week
A few numbers: there were 117 CVS check-ins last week. There are now 23 Eclipse projects: 3 for platform, 5 for build, 11 plugins, 2 features and 2 branding. There are over 400 passing unit tests, most of them inherited from the Java ID3 code I wrote last summer.
Half of the work I did in week 05 involved branding, building and packaging but I also did some development work on the Durham Quick Editor (QE). QE now reads file, MPEG and ID3 metadata and also writes ID3 metadata! Here's a screenshot:
You add files to the list on the left, select a file and then its metadata is editable on the right. QE will be a good way to test Durham plugins for new metadata types and that's why I started on it first. QE will also be a good place to use non-audio metadata plugins so if anyone is interested in doing that it's almost ready to do.
AudioMan and QE will share quite a bit of code so I made two projects to put that shared code:
On the building and packaging side, ALL of the plugin projects are built on a default build. A product build file chooses the correct plugins for a specific product and asks the packager build file to package it for a specific platform. I'm considering releasing my AudioMan build files as a separate subproject - RCP-RelEng - so people can use them to build their own RCP apps.
Platform integration for both products on the Mac is almost done. The last thing to to is package all of the product files in the .app bundle and then dmg it. I'm very close.
On Windows the user will just unzip the directory anywhere they want and run the EXE. That's good enough for now but later we'll have to make a real Windows installer.
What got bumped?
New things to do
# Ant Mangles Eclipse RCP Mac OS X .app Launcher?
It appears as though Apache Ant doesn't play nice with Mac OS X *.app launcher directories.
I'm copying the Eclipse RCP *.app launcher directory with my AudioMan Ant packaging script to build the Mac OS X Carbon versions of AudioMan2 and the Durham Quick Editor. I'm just using the
**NOTE: I had to put a _ line continuation in that long line. It's not in the original code.
The packager generates a completely assembled RCP application for the platform in the
In Mac OS X I click on the launcher in the assembled RCP application's directory and nothing happens. The
If I delete the launcher that my Ant packaging script copied and use the Finder to copy over a fresh launcher from the RCP binary
Update 10:50 AM: Ant <copy> Removes Permissions!
Eclipse.app is a directory -- you can see its contents if you open a Terminal and
If you look in
I guess this is why Eclipse for Mac OS X comes in a gzip'd tar file instead of a ZIP file. ZIP archives do not preserve file permissions and tar archives do. Why are the Linux Eclipse builds ZIP'd then? I guess they expect Linux users to be able to
Anyway, at the bottom of Ant's documentation for the <copy> task they note that Unix file permissions are NOT preserved by the <copy> task. D'oh!
My build machine is on a Linux box so that probably helps things -- I can set permissions before I build the distributable (I'll also have to verify permissions aren't modified by CVS -- Update 11:40 AM: CVS preserves permissions). But it looks like my Mac OS X distributable will have to be a *.tar.gz file until I figure out how to make a nice self-contained Mac package.
While I'm at it I should make my Linux distributables gzip'd and tar'd as well until I figure out how to make something more friendly like an RPM. Linux users don't need their hand held but they'd probably rather not have to
I'd like the AudioMan products to just work right out of the box on every platform. I don't want to lose a user because the install is a PITA.
Many thanks to my anonymous commenter for the solid lead.
# AudioMan Screenshots: With and Without View Titles
I've laid out some initial AudioMan and Durham Quick Editor user interfaces. Since both products use the Eclipse Rich Client Platform (RCP) I decided to use perspectives and views. Unfortunately views come with tabs (Eclipse calls them view titles) that I don't need or want.
Here are some screenshots:
I may have to modify the
Update 9:55AM: Here's the code I'm using to make the AudioMan Browsing Perspective that has tabs:
Update 10:12AM: I figured it out. The IFolderLayout distracted me from the solution -- you can put the view IDs right in the
Now the GUIs look like the second set of screenshots. Woohoo.
# Week 04 Status Report
This is the week 04 status report for AudioMan and its subprojects.
What was done last week
The build machine Ant scripts are ready to go. They will be modified as time goes on, so it's certainly not done but it's ready for the first iteration. I'm very happy with the results after being inspired by Andrew's build ideas, thanks buddy!
The build machine is building and packaging two RCP applications: AudioMan2 and the Durham Quick Editor; for four platforms: Windows, Mac OS X, Linux GTK and Linux motif. All of the packages are ZIP files around 5 MB each. I love being able to build native-looking applications for four(+) platforms at once with the same code! This should be awesome.
The Windows packages work, the Mac OS X ones do but with a small problem (see below). Roy is going to test my Linux builds when he gets back from Cuba. Thanks!
I'll probably wait until after the first iteration to write up how AudioMan's Ant build scripts work to make sure that they actually do work well in development and on the build machine after some inevitable tweaking. This is the fourth rewrite of the build scripts for AudioMan so I have a good amount of confidence in my Ant skillz.
So what about the monster that is PDE BUILD? I took a look at the docs and decided it wasn't worth learning how to integrate some tools (like EMMA) and partial builds (just compile code, just run tests, etc) into it. It seems like PDEBUILD would be great for a build machine, just not my build machine.
This week I also talked to my excellent web host Epicworks about hosting the AudioMan source code repository and build machine on their servers instead of a Fedora Core 3 box in my apartment under my desk. A hosted box has several advantanges over the home machine:
After getting the build machine ready I dug into the AudioMan code. I took the standard RCP template application and modified the perspective, view and action names to match AudioMan. We're all ready to go now, it's off to the races.
What got bumped?
New things to do
That's quite a bit to do -- and will keep me busy over the next two weeks or so. The long weekend this week will definitely help. If you are interested in writing some code in the next two weeks just let me know and we'll figure out a way to collaborate.
# How this Developer sees the Business of Software
I don't want to give you the impression that I write open source software because I'm an altruistic software engineer. I have selfish reasons, definitely. I want to make lots of money as well -- I have no problem admitting I'm a capitalist. Mostly I want to prove I can manage a software project.
What better way to do that than transparently? The disadvantage of giving away my development time for free is offset by the investment in a project that might get me into the project manager hot seat. Open source projects also make fantastic discussion topics for interviews -- which I found out last year -- because commercial companies won't let you talk about your work in detail with other people. I could talk about my open source project in detail because it is completely open.
Business people might get the impression that software developers are a bit too idealistic about their work -- their art -- to realize where their paycheque comes from. Software engineers might think that business people don't understand software processes well enough to get them working for them instead of against them. This post is an exploration of these thoughts and hopefully the start of a small discussion with my readers on the business of software, as seen by a software developer.
There are lots of ways for a software business to sell software to another business but here are quick rundowns of four major ways.
Note that this list does not include internally developed software or commercially-developed open source software (like Eclipse). That kind of software, while developed in a similar way, does not have similar business-to-business concerns because the development is internal to one business.
Secure a business contract to make software for a customer. Design the whole thing on paper up front, gathering requirements from your customer. Write one or many large specification documents (note the root word specific). Over the course of many months implement the contents of those documents. Spend a few weeks testing and then deliver the product.
**BDUF is software lingo for By Design Up Front.
Quickly hack together a prototype to show people some neat software technology that they've never seen before. Listen to the oooohs and aaaahs at your demos. Continue to add features to this prototype until you make a sale. Once you have a customer with a chequebook in hand, spend some time cleaning up the prototype until it is ready for use in the Real World instead of in a controlled demo.
Secure a business contract to make software for a customer. Pear the requirements down to the smallest amount of work that can produce an initial working product. Build that small product in a short BDUF-style iteration. After the iteration deliver the product, take comments from your customer along with new feature suggestions and incorporate them into the next iteration. Repeat over and over again until all of the customer's requirements are met.
** Agile is a classification of a group of software processes. An example is eXtreme Programming (XP).
Identify a market that needs software to perform some task or is underserved by current products. Secure funding to develop this software from an investor (angel), bank (debt financing) or from your own pocket (equity financing). Develop the product with requirements gathered from prospective customers without giving your secrets away to competitors. After months of development and testing release the product to much fanfair, recoup your investment and make a profit.
** COTS more software lingo; it stands for Commercial Off The Shelf. It refers to software designed for a market instead of a specific customer, like Microsoft Office.
Each of these approaches has software development and business advantages and disadvantages. Here's my list. Feel free to add to it in my comments.
 There are a few variations to the 80/20 rule, actually. This version of the 80/20 rule states that 80% of the "work" takes 20% of the time. The last 20% of the work takes 80% of the time. It all depends on how you define "work" -- but in this case it's the difference between visible work (what the demo does) and under-the-covers work. Without the under-the-covers work the product is not ready to ship, but the customer probably doesn't know this.
That's it for now, but I'll probably be adding to this in the upcoming week. Comments?
# Week 03 Status Report
What was done last week
That's almost all of the things I wanted to do this week. I'm starting to get a better idea of what I can accomplish in a week in my free time and setting reasonable and attainable goals -- that's a Good Thing.
For open source projects working on a weekly time scale seems to work well, since you never know when you'll actually have free time. Any finer grain planning than that probably wouldn't work. For larger open source projects the smallest grain would probably be a month or more -- you never know when others will have free time either.
This week I made an AudioMan Release Engineering mailing list. I must stress that this mailing list is not for most people and volume will be relatively high. I'm still experimenting with tools that send emails to this list so there's a chance you may get extra emails from that as well.
Right now the build machine sends a check-in email to the releng mailing list every time someone makes a check-in. See this example of a check-in email, it's pretty neat. I will also be sending the results of every build to this mailing list as well. If that doesn't sound like your cup of tea, I wouldn't recommend joining the list.
What got bumped?
New things to do
I'm going to spend the next week laying out just a bit of the basic UI for AudioMan in the Rich Client Platform (RCP) because I want to have the top end AudioMan projects there to make sure the project dependencies work with the build machine, which will be my main focus.
I was really impressed by what Andrew is doing with his build process. I hope he doesn't mind but I'm going to steal a few of his great ideas and incorporate them into AudioMan's build process. I hope to make building the project flexible for developers as well as rock-solid for the build machine -- and use as much of the same Ant code for both as possible.
Like I said last week, I really want to get the tools and the build system in place before I start doing major work on AudioMan. Then my development rhythym won't get tripped up because I have to fix a something major with the build machine. I want to get it to the it just works stage and be able to forget about it for long periods of time.
The most interesting part of the AudioMan build machine is building RCP plugin projects and how I've centralized all of the Ant code that does it. I'd like to write a document about how I do the Ant builds because I think it could save people some time -- and they would be free to use the code because it's under the EPL. I'm not sure how much easier my way is than Eclipse's PDE BUILD but it seems to me like it would be an alternative with a smaller learning curve.
# Would People Pay for AudioMan?
Here's a question: do you think people would pay for AudioMan? I don't mean how AudioMan was or is right now, I mean a piece of software that would help you organize your collection, point out missing tags, fix tags with Internet services, manage backup discs, etc.
In other words, most of the features in the Draft Functional Specification. Do you think there's a market for it? You won't hurt my feelings by giving me your opinion -- it will honestly help me.
The price point I have in mind is around $20 US ($25 CAD at the current exchange rate). A new major version of AudioMan with new features would be released approximately on a yearly basis. In between major releases there would be maintenance for the existing major version, which would have bug fixes but not new features.
This question only applies to the AudioMan. Durham would continue to be open source under the EPL. The ID3 library, jid3v2, would be open source as well. I would continue to develop AudioMan with an open process but the AudioMan code itself would be "closed".
# Use XML for Metadata Type Description?
Besides the build machine, I'm working on the Durham properties code at the moment. Properties are the name-value pairs that make up metadata, except that Durham has to handle names and values that are more complex than just one Object.
We also need to describe each of these value objects because AudioMan needs to know how to create them. For example, ID3v2 tags have a Track Number property. We need to describe this property so that AudioMan can prompt the user for it and verify the value. Here are some sample restrictions:
Originally I was thinking of making metadata type plugin writers describe all of their properties by creating the proper objects. What a drag that would be! Not only for them, but I'd also have to verify everything they gave me. It also could be very buggy doing it this way because the definition for a metadata type plugin could be pages and pages of code.
A better format might be XML. Something like:
The Key for a simple property like Track Number would just be the PropertyDescription id: "TRCK". The for an ID3v2 Comment the Key is more complex so we have to specify the legal values that the user can input.
If someone is making a new Comment in AudioMan2, with that description we'll know how to prompt them for the key and the value and if what they fill out is within the proper bounds. Note that in the UI we won't say "please enter the key and value" we'll probably say something like "please choose the language, enter a short description and the text of the Comment".
For the Language enumeration the user might be presented with a dropdown box of all of the available languages. OR we could just have English as the default initially and not present them with a choice until Unicode support is in. That's a detail we'll figure out later -- I'm getting off track. But supplying a default for a key value is also an option.
To go back to the architecture diagram I drew last week, the communication between Durham and the metadata type plugins could be XML like this.
Then Durham parses the description XML and gives AudioMan description objects so that AudioMan can validate the values that the user enters, and Durham can also validate them before it stores the values in the cache (ie. a persistent database).
This is a pretty rough idea right now. What do you think?
# Week 02 Status Report
Last week was dominated by documentation but I did still manage to get the AudioMan build machine set up. One Eclipse project --
Here are the artifacts I worked on in the past week:
This week there's more work to do on the build machine:
Sure, I could do this build machine stuff later but why? Some people are so anxious to get into the code that they don't set tools up to help them. I already know these tools and processes work for me so I want them right now.
I added an AudioMan Release Engineering section to the AudioMan website to describe what's going on. I don't want release engineering to be some magical thing working under the covers -- I want it front and center so people can check it out. All of the release engineering code will be open source under the EPL too.
There's one other artifact I was working on last week: sequence diagrams for Durham's core logic. I have hand-drawn drafts that still need another revision to consolidate some common things, then I'll do them up in Visio and post them.
Why do this up front work? Well, it's good to get it all down finally. People are wondering what this Durham thing is supposed to do. I already figured out a lot of the logic when I worked on AudioMan1 so the work is done, it's just locked in my brain. I just have to strip it down into the basics and get it really well tested for the first few iterations of AudioMan2.
Another reason? Just like the ID3 File I/O use cases, Durham's core logic isn't going to change much. Is the metadata stale? OK, update it. The rest is passing bits to the right components. Durham is the basic basic stuff and that's why I wanted to rip it right out of AudioMan1. Then people can use Durham for other things.
The goal is to get these first iterations of AudioMan2 out as quickly as possible but still balancing the quality factor and doing proper release engineering. That means stripping down what AudioMan2 does to a flying gas can. Less to code, less to test, less to release. Then we'll turn the can with wings into a F-117A iteratively, week after week.
Having the patience to get this all set up is probably the hardest part. It would be so easy to get undisciplined and write a bunch of code but as the PM I have to think long term. If AudioMan1 taught me anything, it's that.
# ID3 File Reading and Writing Use Cases
I got a little side tracked this week with AudioMan2 when Luis asked me for some use cases for ID3 metadata editing. So here's the initial stab at it: ID3 Tag File I/O Use Cases, weighing in just under 4,000 words.
Writing it out was actually a great exercise because now I have it straight in my head and down in a document that other people can read. Usually I'm not too keen on making software project artifacts but this area is different than most of the rest of the project. These use cases will not be changing much and I wrote them in a programming language-independent type of way so they are useful to anyone needing to read or write ID3 metadata.
Given how well this went I'll probably also write use cases for Durham's metadata synchronization and persistence algorithms. That's another important area that won't change much and could use some peer review.
# Draft AudioMan2 Architecture Diagram
I finally got around to making that AudioMan2 architecture diagram so that I can communicate my thoughts to people pictorially. Take a look at the big version and let me know what you think and we'll go from there.
Personally I'm a little concerned that every component from the GUI at the top to the metadata plugins at the bottom use the same object:
I suppose I could use a different object between Durham and the metadata plugins than between Durham and AudioMan2 but what does that buy me? I'm not really sure ... thoughts?
This is more of what AudioMan2 will look like at the end of the first stable release (version 2.0 -- there never was an AudioMan 1.0). We'll be iterating many times before a release, and the first few iterations of AudioMan2 may not have a persistent database for example.
When we're talking artifacts like diagrams I suppose it's better to do them per-release, not per iteration. Again, feedback will determine the need.
# Running Plugins or RCP Application with Older Eclipse Platform
So you're developing an Eclipse 3.0 plugin or Rich Client Platform (RCP) application and want to take advantage of the new whiz-bang development features in the latest Eclipse 3.1 milestones (or final release, when it comes).
You fret because you know that by default Eclipse uses the same platform plugins to run your plugin or RCP application as the development environment. So you have a problem: your application is supposed to be designed for 3.0 but the development environment is using Eclipse 3.1 platform plugins to run it. D'oh!
Luckily this is easy to fix and doesn't take much work. I'm going to describe the way I did it and then you can modify it to suit your needs. For the sake of my example consider I'm making an RCP application called WhizBang.
First I created a new Simple Project in Eclipse and gave it a name like
I went back to Eclipse and into the Preferences dialog under
Even though everything works just fine there are probably a few plugins you can remove because they aren't being used by your plugin or application.
The easiest way to see which plugins you need is to open the run configuration and go to the Plugins tab. Then hit Deselect All and add only the Workspace Plug-ins that you need for your project -- don't add any External Plug-ins yourself. Then press the Add Required Plug-ins button and Eclipse will figure out which platform plugins are necessary. Rerun your configuration to make sure it still works.
The unchecked projects in the platform could be removed from the platform project but if you think you're going to need them later you might as well keep them in there.
What's the other advantage of having a platform project like this? Yes, you guessed it: the build machine needs a copy of the required Eclipse platform to make your distribution ZIP or installer.
Update 2005.03.24: Plugin Templates
If you removed the
You can delete the source JAR files from the source plugin to decrease its size if you only need the schema files.