|« November 2004||January 2005 »|
I'm Ryan Lowe, a Software Engineering graduate living in Ottawa, Canada. I like agile software development and Ruby on Rails.
I write this blog in Canadian English and don't use a spell checker. Typos happen.
» Full-time Ruby on Rails freelancer
» Full-time with Rails since May 2005
» Former committer for RadRails (now Aptana)
» I also have a few Rails side-projects in development:
1. wheretogoinTO.com Toronto nightlife
2. Hey Heads Up! TODO list and sharing
3. Layered Genealogy family history research
4. foos for foosball scoring
5. fanconcert for music fans (on hold)
Hiring Rails developers? I can telecommute by the hour from Ottawa, Canada
»» Email: rails AT ryanlowe DOT ca
Now hosted on Hey! Heads Up -- check it out!
Derek Lowe's (Ryan's older brother) words at Ryan's funeral
email@example.com no more
Forging Email Headers: Good, Bad or Ugly?
Sarcastic Dictionary (Part 1 of Many)
Twisting Rails is Risky Business
Risky Business? My Take on Early Alphas
Whoa, it's August 2007
A Postscript to "Growth at the grassroots"
»» All Blog Posts
David Heinemeier Hansson
James Duncan Davidson
Signal vs. Noise
Amy Hoy: (24)slash7
Luis de la Rosa
# An Expanding Text Box for Property Values
I'd like a custom widget for Durham, to be able to handle arbitrary property values.
A file's metadata is name/value pairs, or properties. The names are usually short but the values can be arbitrarily long. In a form to edit metadata there may be text widgets a few lines high for entering values that are expected to be long.
But what if you don't know what to expect? I want to make a form for arbitrary metadata from any file type. I don't want to have to go through each of the properties and say which could be long.
An easy solution would be to make every text box a few lines high -- but that would be ugly and would take up too much space. The form will be in a scrolling view, but scrolling is a bitch. Plus when you expect a short value a regular one line text box just looks better.
It would be nice to have a standard one-line-high textbox widget that can adjust its height either manually or automatically. The manual way involves something like a button beside it representing "switch between one line and multiline". That doesn't seem difficult.
The automatic way could increase the height of the text box widget as needed. If you get to the end of the text box it will add another line as you type. There would also be a maximum height for the widget (ie. 4 lines) after which a vertical scrollbar would appear inside it. Would that be too confusing or weird to use?
# Mysterious Presents
This morning a complete stranger in a car ahead of me at the Tim Hortons drive thru in Renfrew (population 6000) paid for my coffee and bagel. It was a small and simple gift but I still can't stop smiling about it.
# Ask: Why do you want to do that?
People often ask "how do i do this?" I'm getting into the habit of first asking them right back "why do you want to do that?"
Sometimes the solution they are asking about is not suited to the problem they are trying to solve. The bad thing about that is they may not know enough about the solution to know it's unsuited.
So I find that a good thing to do is to dig dig dig until you get all the way down to the intent. Once you know intent you are in a better position to give your opinion. Intent itself is unbiased by a solution, which gives you more creativity and flexibility in your own suggestion for a solution.
This relates to software development in many ways but a common one is user feedback. A customer/user may not be very technically inclined but asks you to implement some specific feature or enhancement.
They fall into the common feedback trap and phrase the request in the form of a solution rather than a problem. Consider their suggested solution but also dig down to the intent of the problem, so you can integrate an intent-based solution into the current software architecture. You are responsible for maintaining the architecture and making sure everything fits, not them.
# In Through the Out Door
Sound will get into every nook and cranny it possibly can. That's a pretty cool property but sometimes it can be annoying.
Take my apartment door: there's a fairly even 1/4 inch gap around the whole door for sound to travel through. At least the door frame juts inward on the outside of the door to block direct noise but the gap around the door is preserved by three small rubber stoppers (marked in red).
Their function is probably to stop the metal door from banging on the metal door frame but they also hold the gap there, allowing sound to travel in from the outside and out from the inside.
It would be nice if I could fix this gap and cut the noise travel so I don't disturb my neighbors when I play my John Tesh records at full volume. By the same token I'd rather not be woken up by my neighbor across the hall jingling his keys at 6AM.
So I'm thinking of replacing the three stoppers with a rubber strip of the same width, going completely around the door frame (B). If that doesn't block most of the noise I could also add a small piece of plastic or rubber to the inside of the door (A).
This may be a little silly, sure. :) But a little bit of work could cut down on a lot of noise so I don't disturb my neighbors and the hallway noise doesn't disturb me.
# OK I'll Admit It: Software Isn't Perfect
There's an interesting discussion going on over at Mary Hodder's blog, pointed to by Dave Winer. Mary was upset because she paid for a beta version of NetNewsWire that ended up corrupting her data. She also went into what she thought of user-developer relations.
Most of the conversation seems to be around the word 'beta' and how several developers/companies have been using it for different purposes (like Google uses it to get around legal issues), clouding the meaning. As a user this is a completely fair complaint. Mary even paid for this beta version.
But let's look at this from the other end. How much quality does a beta release have? How much quality do you need to have before you can stop calling your software beta?
I look at this problem as a software engineer and the word "beta" seems as insignificant and arbitrary as version numbers. There's no universal quality measuring stick you must satisfy to get out of beta, it's just an arbitrary developer-defined state meaning "less than perfect". Well guess what? The final release will be less than perfect too. All beta means is that it's less than less than perfect.
That's the stumbling block that users need to get over when it comes to software. All software is imperfect and there's nothing software developers can do about it. It would be impractical to make software perfect because it would cost far too much and take too long to release. Just ask NASA how much work it takes to get defect-free software. Even after all of the checks and balances NASA has, they still have disasters caused by software. The solution for software companies is to release less than perfect software that does a pretty good job.
The downside is that occassionally this less than perfect software will have a catastrophic defect that ruins data. The fact that the software is being called beta, or 1.0 or even 5.0 is pretty irrelevant at that point. So is the fact that the user is paying for it. The only thing that matters is that the software screwed up.
I don't think it's the users' fault they have come to the unrealistic expectation that software is perfect, it's years of denial by the software industry. Admitting your software has bugs was bad business! So users have come to expect perfect software even though they were never actually getting it in the first place.
Fortunately things are changing. Software is getting so complex and often has so many changing dependencies these days that updating frequently (and usually for no cost) is becoming the new status quo. Users are getting used to seeing more problems with software and realising that software has defects, especially on web sites (which are software-based). Users are also used to updating products like Windows on a monthly basis but we still have a long way to go.
The only thing users aren't used to is regularly backing up their critical data. Usually they only do this after they've been stung by a bad defect or a power outage, fire or natural disaster. This is where the software companies need to do better: educating the public about the expectations they should have about the quality of their software as well as the need to back up important data.
It's not that easy to back data up either, or keep backups organised. We need to do a much better job here.
Sometimes defects are out of the software developer's hands, like an operating system error, a network error (ie. web pages), a hardware failure or a power outage. We can't expect users to understand all of the technical issues, but their quality expectations need to change and the software industry needs to do it's part to help out.
# Iron Ring Firmly in Place
After watching a few successful open source projects and seeing what kinds of things work for them it will be nice to try a project of my own and see if I can capitalize on the same techniques. It was instructive seeing the AudioMan project's lack of momentum as an open source project, noted in its post-mortem.
A secondary goal of Durham will be to examine how open source techniques work, what level of quality open source software delivers (and how) and investigate if any open source development techniques may be used in closed source development.
Open source software will not only be a major business competitor to closed source software in the future, it will also be a major (and free) source of bleeding edge development techniques and idea sharing among colleagues.
It would be a big mistake to dismiss successful open source projects as a fluke or an anomaly -- they could one day cut into programmer and software engineers' livelihoods. If we pay attention to why open source succeeds we can learn, adjust and even benefit greatly from it as software professionals.
I already use several open source projects for my work and I don't know where I'd be without them.
# Baby Steps to the ...
This is convenient for deployment because users don't need to keep checking your website and mess with installers to be up to date. I'd really like to use this update mechanism to release updates as often as weekly, which is a lot faster than the version numbers in Eclipse change.
The first step is to get the absolute bare minimum amount of stuff working so I can make the first release and start the ball rolling. The application itself will be a scaled down version of the old AudioMan project. I still haven't figured out what to call the application that does audio files -- only the platform is called Durham.
The user interface will be a two paned window, with a table on the left and a form on the right. The user can add audio files to the table and then select one of them and view and edit its metadata in the form. There's no storage, the table is cleared every time the application is closed. For now I'm going to call this GUI the Quick Edit Perspective.
# Google: Protect PageRank Against Blog Spammers
James Robertson blogs about the blog spam problem and how software homogeneity doesn't help. At the end he says:
I don't have a solution, other than suggesting that you take "the road less traveled" in selecting blog server software. The trouble is, that mostly requires a level of technical literacy beyond the reach and/or interest of the vast majority of people.
...and that's exactly it. The "road less travelled" is usually too complicated for most people. MovableType (MT) is fairly straightforward to install even if you know almost nothing about Linux or the Apache web server you host it on. You get your hand held the whole way through the MT install and it's why I use it. The popular tools are often popular exactly because while solving the user's problem they are also relatively easy to install and use.
I agree that the popularity of a piece of software makes it a more attractive target and in the early stages of being a target that sometimes makes the software unusable or a PITA to maintain. But MT and MT-Blacklist (protects against blog spam) are at the level of maturity now that if you have MT-Blacklist installed you are fairly well covered. Sure, this takes education and maintenance of the blacklist on the MT admin's part, but it's getting much better than it was just six months ago.
It's amazing that with all of the talk of blog spam not many people mention the *real problem* that blog comment spams exploit: Google rankings. If Google didn't hold blog spams in such high regard there would be less benefit to blog spamming. Yes, people can also click on links in blog comment spams but they are usually on old less-travelled entries anyway. The main goal of blog spam is to improve the linked site's Google PageRank by being associated with a page that already has a high PageRank.
I've noticed this on my own site. The entries that get spammed the most are the ones that I get the most Google referrals from. It's almost as if the blog spammers do a Google search for my blog and take the top of the pile. Being referred by a page with a high PageRank is better, and blog spammers know this.
What can Google do about it? Recognize URLs that are used in blog spams and penalize the site's Google ranking accordingly, just like search engines used to do when pages overused META tags and keywords years ago. Google could maintain their own human-moderated black list (or penalty list) for this purpose, just like MT-Blacklist does.
What's the benefit to Google? It improves the integrity of their search results because people will have one less way of exploiting the PageRank algorithms. Google has a vested interest in returning search results that haven't been exploited for monetary gain. That's the kind of honest ("don't be evil") technology Google has built its business on, and why people trust Google at all. Here I'm asking Google: don't let other people be evil either.
# Durham Courts Us
So I've created some brainstorming notes for the successor to AudioMan, which I've given the codename Durham for now (ie. until someone sics their lawyers on me). The notes are probably riddled with spelling mistakes and logic errors but let me know what you think of the idea.
# AudioMan Post Mortem
After a little while to digest my learning experience, I think it's time to look back and go over some problems and lessons learned.
Problem #1: Successful open source and closed source projects have different rules
I was using the AudioMan project to learn about Java, Eclipse, threading and project management, not ship a product. Then I wondered why no one was interested. :) I shouldn't have been that surprised, really.
I was scratching my own itch, but few people had the same itch. This isn't so important in open source software unless you are trying to get noticed. In the closed source world obviously you need to identify a market that has a problem where your sofware is the solution. Then you can make a buck.
This isn't so important for open source software, sure. But the more users I have, the more feedback I'll get and the more bug reports I'll get. All of that improves the product and it's about opening channels of communication and listening to them.
Problem #2: It was difficult to contribute
I made it almost impossible to contribute to this project which made it a one man show. While I had control this way it just doesn't scale. The more restrictions you have the more you have to communicate to people who want to contribute and then they lose interest.
Tear down the barriers to contribution and people will be more eager to send you code, good or bad. Adjust your process and release plan to accomodate this.
Another barrier to contribution was that the code was not well modularized. Even if people wanted to work on an isolated area, they couldn't. A plugin-style architecture could solve this problem.
Problem #3: Ship first, learn second
After doing this project, I feel that learning by itself is not a strong enough motivator. If you have users always pushing you to better your product, that seems like a better way to get your ass in gear. While learning new things is good, make it a secondary goal after learning to ship a product and gather a userbase.
Problem #4: Quality standards were too high
To get feedback you need to ship a product. To ship a product you need decent enough quality that people can use the product. You don't need hundreds of unit tests. You don't need 100% line code coverage.
Even though a completely green coverage report is neat to look at, it doesn't mean much. Line coverage is a tricky metric and it can make you feel warm and fuzzy when you probably shouldn't. It's false comfort, beware.
It was interesting to see how much work it took to get close to 100% code coverage, but then it was also a great learning experience to see how it wasn't worth all of that work. The trick seems to be setting a reasonable quality standard but still having a way to verify that your product satisfies that standard. Line coverage is a good guideline, but it's only accurate as a quality standard to about 25-40%.
Also very important is not denying the existence of defects in your software and having a plan to deal with them as they are found. Users will be much more impressed if you can patch a problem and release an updated version of product quickly than if you say your software is perfect.
The notion of defect free or perfect software is an absolute falacy. The quicker you admit this to yourself, the better. The general public is starting to notice this too -- a lot of mainstream software needs constant updating. All software is in a perpetual state of brokenness. The only thing that matters is that it's not broken in important places.
As well, that kind of perfection goal gets in the way of forward feature progress. You'll spend too much time testing and once you get to your perfect state you'll be too scared to make major changes. Then you can't respond to your user's changing needs -- you're stuck admiring your perfect project while it goes stale.
Problem #5: No one was interested
It might be interesting to ask people why they weren't interested in AudioMan but would they be able to tell me? :) Bottom line was that only a few people cared about AudioMan.
First, you have to make this a goal if you want it to happen. The project won't be able to sell itself. You have to do legwork to get the product noticed and into people's hands.
Second, you have to make users your priority so they will stick with you. If a user requests a feature and you deliver it quicky they will feel empowered. If they give you a bug report and you fix it quickly, they'll know you care about their needs.
Problem #6: The problem is bigger than audio files
Metadata isn't exclusive to music files and only targetting the audio subset was a mistake. While narrowing it down simplified the user interface it limited the extensibility of the application, which in turn limited the possible market and possible user base.
As well, people like tools they can hack and extend. Just look at all of the games that have benefitted from this, as well as more traditional applications like Firefox and its extensions, and developer tools like Eclipse and its plugins.
Applications like Firefox also benefit from the extension model because then they can stay simple. If someone wants a specific feature they can download or make an extension that does it. That feature doesn't creep into Firefox itself and so it stays lightweight for the majority that don't a lot of specific features.
Problem #7: The problem needed focus
Was AudioMan a collection browser, a collection organizer, a metadata editor or a backup tracking tool? Good question -- I didn't really know either. Sometimes leaving yourself flexible can lead you into a new market you hadn't anticipated, to the benefit of your project. Eric Raymond explores this effect in The Cathedral and the Bazaar.
But other times being flexible can make the project a hard sell, leaving people confused about it actually does. It should be realy easy to explain what the product does. If it isn't you might want to figure out why. Then you'll be able to make a quick elevator pitch to people without having to think about it.
Problem #8: Leverage the platform
The plugin architecture is a good example of an established technology with help manuals and tutorials that plugin developers already know. I can leverage this knowledge for a new product. While developing an extension to this new product, the developers can be learning how to make and use Eclipse plugins, fragments and extensions. These are transferrable skills they can use for other projects because they use the Eclipse plugin architecture.
Other RCP goodies include: update manager, job progress, workbench UI and more.
Problem #9: Shipping a project is less than 25% programming
That was a good lesson to learn for sure. There's a lot of communication necessary to ship a successful product, as well as a lot of administration work. You have to be well organized, and you need to be motivated enough to do tasks you don't like to do.
This is especially true on open source projects where people are contributing in their spare time, and they only want to do things they are interested in. As a project leader, you should encourage this but it leaves you with all of the garbage work. If you are a programmer, maybe being the head of an open source project isn't the goal you should be aiming for. Instead, be a major contributor and leave the cleanup work for the project lead.
This is why I always cringe when I hear people say that the leader of project X is a great programmer or hacker. That may not, and probably isn't, necessarily the case. These guys at the top (head committers and/or project managers) know how to coordinate people and communicate well. They could be terrible programmers. The essential skills are to be able to recognize good quality, seize opportunities, ship a product regularly and communicate with the community you help built around the product.
Will I start a new project? Probably. When? I'm not sure. I'd really like to learn (secondarily, of course) more about Eclipse RCP and J2SE 5 and all of the new language features as well as the new Eclipse 3.1 features. A project like this would be a good excuse to get into that.
What do I have in mind? How about an extensible platform built on Eclipse RCP and the plugin architecture targetted at the file metadata editing market? Then I'll add a few plugins of my own to deal with audio files, leaving the door wide open for other file formats to see who steps in.
# The State of Metadata/Search
Interesting things are happening in the metadata/search space these days. The reason why I refer to metadata is that search is reaching a limitation: it's only good at dealing with text file formats.
When we want to start searching binary files, there needs to be some textual information describing the file that we can search on, called metadata. Moreover, search engines need to support at least reading this metadata to build their indexes, and do scoring.
Then other applications need to be able to modify this metadata to make searching useful, somewhat of a chicken and egg problem. That metadata also needs to travel with the file from machine to machine, not only exist in a local database.
This signals that they are probably working on a more comprehensive change to the Windows file system that would, of course, take more time but be more revolutionary. It also indirectly points to a lack of fear that Google Desktop Search (GDS) will steal a lot of hearts and minds in the meantime.
Microsoft probably has a stop-gap competitor to GDS up their sleeve. They could release it as part of MSN Search and/or Windows Update but it wouldn't make them much money. It would, however, put them in a position to sell WinFS to people by creating more demand for better desktop search, which most people don't really care about today.
Update: Here's the stop-gap ... MSN Desktop Search.
Update 2: Yahoo joins Apple, Google and Microsoft in the already-crowded desktop search space. This got interesting really quickly...
So in the future, metadata will play a more important role than it does today. This creates demand for applications that can modify metadata for different file formats (which may have to be custom to the type of file, like images/audio/video, etc). It also creates a demand for standards for metadata so that third party apps can actually write it and search engines can read it.
# Sedentary Full Time
I didn't expect that the hardest thing about a full time job would be sitting for 8 to 10 hours a day. I don't know about you guys but the work has been really hard on my body. Shoulders, neck, back, legs, most of it. Not good if I plan to be doing this for the next thirty years!
I'm seeing a great massage therapist for the neck/shoulder/back pain, but only so much of that is covered by extended health care here and it returns quickly. There's a gym at the apartment building I just moved into that I'm anxious to start using, so maybe that will help too.
My mind really wants to do other projects outside of work, but the body is disagreeing. It's probably smarter to listen to my body on this one but damn, I'd really like to try some new programming stuff. Frustrating.
# Overzealous Comment Blocking
If you've tried to comment on my blog in the last few days, you could have been blocked. Sorry 'bout that, I had MT-Blacklist set up wrong and it was blocking ALL URLs. Oops. :)
# Software Project Monitoring Can Facilitate Creativity
What separates software engineers from programmers? I'm not talking about the job title here (because some people have hijacked it), I'm talking about a person that thinks about software from an engineering point of view. What's different between how he thinks of a solution to a customer's problem and how a programmer might think of a solution to the same problem?
Software engineers like my classmates and I were trained (at least given a solid introduction) in software project management and quality assurance. Honestly, these two subjects can and do take the fun out of programming but they also give valuable perspective to a software engineer. As far as your customer is concerned software isn't fun, it's actually costing them a lot of money. They want you to be effective, maybe happy and then you can have fun if you have time between milestones.
This, I think, is somewhat difficult for programmers to swallow. A lot of these programmers started out writing code in their spare time for fun and now they have to do the same type of work in a clinical, bug-sterilized and heavily monitored environment. It's completely different than their hacking at home. It's not much fun, it's hierarchial. It's responsible.
A software project manager worth his salt will try to monitor the heck out of his project because you can't control what you aren't measuring. That's a pretty basic control engineering concept I learned back in one of my electrical engineering classes.
The software engineer may understand the reasons for all of this monitoring better, though it may not make him a happy programmer per se. People want control over their work, room to be creative or innovative and most of all: they want to be trusted. They want responsibility as well as a challenge. Monitoring the unit test suite and blaming someone when the build breaks is not that trusting, to be honest. But it's engineering and this is the way the profession is heading.
Bridge building one thousand years ago may have been a casual affair but today it's taken quite seriously and is probably a monotonous and well-scripted task with expected checks and double-checks and triple-checks and balances. There are still people going into civil engineering even though they expect this, so it can't be all that bad of a future for software engineering.
How do you keep free-spirited programmers happier in an environment that needs high quality and thus, heavy monitoring? The trick as a software project manager may be to examine the project as a whole and divide it into pieces based on quality requirements and/or code complexity and/or testability. Then monitor (very important) the code quality using unit testing results, code coverage tools or profilers, etc. The bug-reporting database should also be monitored to see which of the pieces gets the most bug reports in an iteration, which is most fragile, which changes the most (code turnover can mean more defects) and factor those numbers in as well.
Then the project manager will know where the critical pieces of the software are and what state of health they are in at any given time. He'll also know where the less-critical areas are, and give people more freedom in those areas. If a piece suddenly becomes very unhealthy, the project manager should know right away from his monitoring and start some corrective action. Allowing for a certain degree of experimentation and creativity can solve problems in new ways, which is good too. Knowing where to allow creativity and flexibility is important.
Strictly from an engineering point of view though, monitoring the health of your project can lower your overall project risk and also put you in a position to take further risks (like large refactorings or whole component replacements) as well. But first you have to be testing your application in order to gather numbers to monitor project health.
You are testing, aren't you?
# Bell Mobility Off the Hook
After entire evening of going through my cell phone and VISA bills and loading the data into a spreadsheet to figure out what the hell happened, I can conclude this: Bell Mobility and I are financially even.
Nevermind the fact that I've had to waste my time worrying about this for the last four months but it's been relatively minor. Bell Mobility has my VISA information and could effectively bill whatever they wanted. Apparently VISA won't let you block Bell's charges either, you'd have to cancel the card. Some people are being overbilled hundreds of dollars. I'm not sure what percentage of people actually had large billing errors.
In a nutshell, I was double-billed for a month and the credited for the overbilling later. They also sent me a mysterious refund cheque which they then charged me for on the next bill, effectively cancelling it out. Combine all of that together and it was a big mess. Six months later it's all sorted out again.
Bell Mobility really needs to get whacked with a customer-care cluestick. It's not cool to leave your customers in the dark when you're having billing problems. I don't want to have to hear about it on CBC news, I want to hear it from the company. I want to know as soon as possible, and when you expect the problem will be dealt with. In other words I want to be treated like a human being instead of an account number. I didn't get one letter from Bell explaining the billing errors, I had to call the service number to find out. Of course they probably aren't contractually obligated to notify me at all, but that's what separates great companies from letter-of-the-contract-following companies. I really appreciate when companies go the extra mile.
The good thing about all of this is that I entered the minutes I used each month into the spreadsheet and I realised I'm definitely not on the right cell phone plan. I'll probably get a new phone and a new plan, and carefully consider whether or not I want to stick with Bell Mobility. I've heard the other cell phone service companies aren't much better...
As software engineers, we can look at this train wreck of a deployment and learn from it. This is obviously not the way to do it. Bell's lucky they could recover from the billing errors (or have they?) but nevertheless a botched software deployment can really hurt your business' reputation. They have my sympathy as a software engineer in that I can appreciate how complicated an upgrade like this would have been, but as a customer I felt alienated.