« July 2004 September 2004 »
blog header image
# Super Size Me

I saw the documentary Super Size Me recently. It's about a guy who decides to eat nothing about McDonalds food for 30 days. At the end of it he had gained 25 pounds, had elevated cholesterol and was seriously damaging his liver.

The documentary also covered generally the obesity problem in the United States -- relating to diet and exercise -- to complement a man's story of destroying his body.

Was this documentary extreme? Yes, but it illustrates a point. Even the man's doctors were surprised about the amount of damage this diet did to him. Even though no one in their right mind -- or without the prospect of a high-grossing documentary -- would eat McDonalds every day, it gives you an idea of how dangerous the food is. It gives doctors a better idea as well.

I stopped eating McDonalds when I went to university for a pretty pragmatic reason: I was downtown Ottawa with no car, the nearest McDonalds was six blocks away and wasn't as good as the one in Renfrew, the town I went to high school. I also had a cafeteria meal plan. Paying for food outside of that didn't make much sense.

High school, right ... did I mention I ate McDonalds every day for an entire school year? Ironically enough, I was sick of the cafeteria food. In retrospect it wasn't too healthy ... but what did I know? I was a seemingly indestructible kid and McDonalds tasted good -- until you got the feeling we affectionately called Post-McDonalds Syndrome. That kicked in about an hour later. I won't get into that...

When I got a car, there was McDonalds again: available, easy, quick. So is Subway, but it's not open at 3 AM. Which brings up ...

Lifestyle. Yes, the computer geek lifestyle ... what is it? We sit at desks for most of the work day. That by itself isn't healthy. Some of us keep strange hours, I know I do. Sleep? All over the place. Some of us eat poorly, though I don't think that's isolated at all to the computer industry.

There are some that have taken the same intelligence, logic (and sense of humour) they use in their computer work and apply it to their diets. In fact I've met some computer people that are downright health nuts (ever heard of bottled water with chlorophyll? it's green!). It depends on how interested we are in our own health.

Given the nature of the industry we are in we should be. At the very least, we're sitting a desk 4, 6, 8, 10, 12 hours a day -- we may want to counteract the negative aspects of that inactivity.

Which doesn't even consider what kind of implications our overall style of work has on our diet. It's very easy to say you'll change your diet but it's not so easy to do. It's not that much fun either. Subway over McDonalds is an easy choice and for lunch it's the least I can do. But there's a lot more.

What do some of you guys do?

posted at August 29, 2004 at 01:10 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (11)

# jid3rL Will Continue

I'm not going to stop development on jid3rL even though I'm working again. I'm determined to get it done, albeit slower than if I were still working on it ten hours a day.

The project has been punctuated by a few major refactors as I learn the domain of the problem. This is often the case in software development, and why the iterative/agile methodology centered on enabling refactoring is so powerful: I don't need to know the whole domain to start, because I'm not designing up front.

But don't forget: to refactor with confidence you need unit tests. Otherwise you will just accumulate regressions you have no idea about. jid3rL already has four hundred unit tests in less than a month. Crazy.

Speaking of unit tests, I've made an additional unit testing suite for jid3rL that tests against real mp3 files. It's up on the build page now. The original unit test suite just unit tests against the API with no mp3 files at all, and is what the jcoverage code coverage metric is based on. This is so the code coverage metric isn't skewed by tests involving files.

There are over forty frame types in the various versions of id3v2 and each has its own arrangement of bytes in the payload after the frame header. I have to parse these bytes to read the frame and also generate the bytes from a frame object to construct a tag for writing. In jid3rL that's done individually in each frame right now, which is a terrible administration headache already (I have less than a dozen frame types implemented) and could be a source of quality problems in parsing and building. Not to mention that unit testing each of these frame types is a major effort and should be centralized.

I was looking through the id3lib C++ source code recently and saw something interesting: all of the frame types are specified as constant structs, and the parser and builder are generalized to take these constants and parse and construct frames based on them. I like this approach and I think I'm going to mimic it, but in jid3rL the frame types will still be classes and not constants.

There are some complicated frame types in id3v2 to watch out for. Some use one field value to determine the lengths of another field. Some have fields a few bits in length appended together which do not fall on byte boundaries. Rather than complicate the generalized parser and builder with these advanced frames I think I will parse/build them separately. We'll see which direction the refactoring takes.

AudioMan itself is temporarily on hold until jid3rL is in a usable state and is relatively stable. I'm not going to take a chance on it because with a serious enough defect jid3rL could seriously damage people's mp3 files. That would be bad. jid3rL is going to be the most highly tested piece of software I've ever written.

posted at August 27, 2004 at 04:53 PM EST
last updated December 5-, 2005 at 05: 1 PM EST

»» permalink | comments (1)

# Scoble's Link Blog

James Robertson, who never seems to be afraid to tell it like it is, gets critical of Scoble's Link Blog. I'm going to have to agree with him on this one.

I've looked at Scoble's link blog a few times and it's just a big list of partially quoted posts. I've never subscribed to it. While it might be more convenient for him to dump a bunch of links quickly like that, I think it's having an important negative effect on the quality of his original blog.

Before I get into any criticisms of my own I think it's important to say that I value a person's right to choose about the content of their blog. If someone doesn't like your blog, they shouldn't read it -- it's that simple.

But Scoble's blog is on another level: it's almost a community on its own. I'm not saying he has more responsibility to his readers than any other blogger, but when he preaches about community building and doing that by linking to others, he should be careful to walk the walk to protect his credibility.

That's where the link blog fails. People enjoy being linked from Scoble's main blog because they get lots of traffic and discussion. I doubt they feel the same way about being linked from the link blog. It just doesn't have the same impact, the same community building force.

When you look at the pace he was going at though, he doesn't have much of a choice. Who on earth tries to personally aggregate 2000 blogs? People just don't scale like that, and it was bound to catch up to him and his blog. Is there an easy solution? No. Is Scoble's link blog a good solution? It might be for others, but it's not for me. I won't be subscribing, and that's unfortunate because Scoble often finds interesting stuff reading feeds and through his contacts.

I have a link blog too: my bullet blog on the right. I put only links there, and only stuff I really want people to notice. It goes in spurts, but I might add five links a week. Notice they are undated. Mark Pilgrim also has a link blog on the same page as his main blog.

I like this approach better because it's a solid connection between the main blog and the link blog. There's a credibility aspect to that, and the links are still in a high profile enough spot to be noticed by your blog traffic. Unfortunately this solution would probably not scale to Scoble's needs.

Update Sat 12:29 Scoble linked to this post on that link blog and I got less than a dozen referrals from it ... maybe my post title wasn't catchy enough? Maybe his link blog doesn't have many readers? Maybe many of his link blog readers are gone for the weekend? Many he just posted too many links in one day? Could be one of many reasons ... but any way you slice it, it seems that a link on the link blog doesn't have nearly the same community-building impact as a link on the Scobleizer.

Why am I picking on Robert? I'm not. :) I just find his blog and blogging to be an interesting case study. What he blogs is viewed by many times more people than most blogs, so he has special issues he has to deal with. Who knows, we may all be able to blog with the volume he does in the future but easier because we have better tools and procedures. We'll get better at blogging by watching extreme bloggers like Robert Scoble while thinking about and trying out ways to make reaching and communicating with communities easier. Blogging is still young -- there's lots of room for improvement on both ends of the channel; blogging and feed aggregation.

posted at August 27, 2004 at 11:20 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (0)

# Updated to MovableType 3

OK, I've transferred everything over to the new MovableType. If you have any problems with this blog now please let me know ... it should be all good now.

It's funny, even with no front web site the old MovableType installation got spammed 500 times since I moved it over. The spammers didn't even go through the web site to spam me, they just spammed the MT API. Tricky. This one hasn't been spammed yet, though I think I'll owe that to changing the file names to the post titles.

Update: it doesn't need the web site to spam comments because the CGI file that handles comments is in the MT install directory, not where the blog is. Once I remove the blog from the MT install, the entries won't exist and comments can't be added to them.

Update 2: I'm noticing a lot of referrals in my hit counter from unknown, which sometimes is an RSS feed reader opening up the post. Are my RSS feeds OK? The Movable Type default is to publish excerpts and I changed it -- my intent is to publish the full post.

Also, found some great summary and some tips about avoiding comment spam.

posted at August 26, 2004 at 12:31 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (4)

# Just Pile it On

Just got a copy of Visual Studio 2005 Beta 1 in the mail today, which I'm looking forward to checking out to see how much has changed since I compared the older Visual Studio .NET and Eclipse. Notice that VS 2005 doesn't explicitly mention .NET.

So naturally I'm looking for a "real" project to do in it. If a great idea for a project doesn't fall out of the sky, I'll probably write Tetris (TM) in .NET again. The first time I wrote it a hard drive crash erased the source code. :S Though I do still have the compiled version somewhere.

Update: I'll check out the .NET options for reading id3 tags, and if they are lacking I might port jid3rL to the .NET framework.

posted at August 25, 2004 at 01:28 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (2)

# Back to Work

After a little down time I've taken a full time job at Agnovi, a software company here in Ottawa. I won't be talking about the details of my job here, but I'm looking forward to it.

I'm also looking forward to moving back to downtown Ottawa. It's a renter's market now apparently too, which is sweet. If you know of any nice buildings downtown that I should check out, let me know. I'm looking for a one bedroom place.

PS> still working on jid3rL ... after mapping out all of the frame types, I have an even better grasp of the domain. Doing these "frames by reference" has proved tricky though.

posted at August 24, 2004 at 02:32 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (8)

# Comment Spam Blocking

I've been getting a lot of spam on this blog lately, so I've finally updated to Movable Type 3.0 and the MT-Blacklist plugin.

I also changed the URLs of the posts to correspond to the post title so they aren't as predictable.

I'll be transferring over my templates soon. Let me know if I forget anything. :)

Update Sunday 10:58pm Fun times! Not really. There are plenty of little glitches to fix. I moved my Bullet Blog over to MT3 too, because it was being spammed as well. It will probably take a week to tweak this site back to goodness again.

posted at August 22, 2004 at 01:47 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (2)

# Gmail Notifier

Google has released an official gmail notifier, which is good. There were a few unofficial ones bouncing around, but I couldn't get any of them to work well so I gave up. I'd rather have one that won't break anyway, straight from the source.

Instead of folders, gmail has labels. You can configure filters to automatically label a new email when it arrives based on subject, sender, etc. I use this for mailing list emails. Unfortunately gmail notifier does not see new labelled emails, only new unlabelled ones that appear in your "inbox". Why is that?

posted at August 20, 2004 at 03:48 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (1)

# Frame Fields Chart

To help me get an idea of the frames I have to parse in id3v2, I've made up a frame fields chart.

Remember that frames hierarchy I was talking about the other day? It's a recursive map of maps to store frames to ensures that I don't write over a frame with another with the same id.

About half of the frame types in the spec allow more than one frame with the same frame id. You have to parse the beginning of the frame to get to these fields to differentiate between them. I've marked those fields in the chart too.

The chart is almost complete, but there are a few question marks.

Update 1:05pm chart is done now.

posted at August 20, 2004 at 11:42 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (0)

# Leaning Towards Frames By Reference

If you managed to get though my babbling about big id3v2 frames (not just once, but twice) you might be asking yourself "why?" And that's a great question, because I could be trying to solve the wrong problem.

Tag2 (as well as SimpleTag2) has a method called writeToFile(File). The idea there is that even though most of the time you'll want to write the tag back to the original file sometimes you may want to read a tag from one file, modify it slightly and then write to it another file.

Let's say you keep track of large frames by reference; you keep track of the File, offset and length to find the frame. Then you read the tag on Monday, delete the original file on Tuesday and try to write the tag to another file on Friday. All of those frames-by-reference will be gone because the original file is gone.

But is this such a big deal? The tag still contains a lot of data: all of the frames that are small enough to be parsed to regular Java Objects. You could still write this data on Friday, and maybe tell the user of the library "hey, by the way: everything by reference wasn't written because the file is gone." That's the worst case.

More often though, you'll be writing to the same file the tag was read from. That file will still has all of the frames-by-reference, right? Yeah, probably. Maybe another program deleted one of them, but you can just skip that missing reference now because it's already gone.

Update 12:49: File, offset and length are not good references into a file to find a frame. Frame ordering can change over time, and the position of frames can be bumped slightly if previous frames increase in size. It's impractical to keep track of the offset changes that could happen in other applications.

A better reference is a pointer to the File, and the List of Objects I use as a frames hierarchy key. Then it won't matter if the position of the frame changes within the file, I can find the frame based on the jid3rL frame hierarchy I made up.

posted at August 19, 2004 at 12:11 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (0)

# jid3rL Frame Storage Types

So to summarize an earlier post about big frame sizes: id3v2 tags can be up to 256MB long, so technically speaking an id3v2 tag could have a single frame that's also 256MB long.

You may have read the comments on a post I wrote early in in jid3rL development. I explained that when a tag was bigger than the previous one, the whole file including the music had to be rewritten because of the way that file systems layed out files. To counter this, most tags are padded with zeros to allow a tag with more data to fit without needing to rewrite the whole file over again.

In the case of large frames, rewriting a file could turn into a hard drive space problem. If a tag is too long to fit over the old tag for a file F, I do the following.

  1. create a temporary file T
  2. write the new tag to T
  3. skip the size of the old tag in F and then write the rest of F to T
  4. delete F
  5. rename T to F

So just before step 4, I have two copies of the file -- F with the old tag and T with the new tag. If the tags are 256MB then that's at least 512MB, nevermind the size of the music. :) If I use the first approach I explained yesterday where I copy frame data to a temp file so I always have it, then that's another 256MB I need.

Using this extreme example you can see the jid3rL library will need to set a limit on the maximum supported frame size, after which the library could just refuse to modify the file. It could also just ignore gigantic frames and not rewrite them the next time the tag is written. Hard to say ...

This makes three different types of frame storage:

  1. If the frame is small enough (ie. size < 1024 kilobytes), its information is parsed and stored as Java Objects.
  2. If the frame is large (ie. 1024 kilobytes < size < 1 Megabyte), the frame is stored as a temp file or a pointer to a file, offset and length.
  3. If the frame is gigantic (ie. size > 1 Megabyte), the library gives an error.

Since jid3rL is an open source library, the threshold settings for these storage types could just be constants in a Java class. They won't need to change during program execution, but software developers using the library might want to tweak these thresholds themselves and recompile the JAR.

It would be nice if the first two frame storage types looked the same from the outside with a good abstraction. When I'm writing a tag to a file, I don't want to have to worry if a frame is coming an array of bytes or a file, I just want to take the list of frames and write them out one after the other.

Roy and I were also talking about frame ordering within the tag the other day and its implications. I'll have to blog about that next.

posted at August 19, 2004 at 10:03 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (2)

# id3v2 Frame Gigantism

Now that I'm done doing simple id3v2 tags with jid3v2, a small hurdle has come up. Images. Up until now, I've been storing the bytes of frames in byte arrays. When you get to images, which can be 100 kilobytes or more, then you start chewing up RAM with that kind of code.

Images aren't the only frames with this problem. Technically in id3v2 the frames have very high length limits, which are specified in the frame header and differ by id3 version. 2.2 uses three bytes, 2.3 uses four bytes and 2.4 uses four sync-safe bytes. An effective library should be able to handle these limits.

2.2: 3x8 = 24 bits, a range of 0 to 16,777,216
2.3: 4x8 = 32 bits, a range of 0 to 4,294,967,296**
2.4: 4x7 = 28 bits, a range of 0 to 268,435,456

**Note that the tag length is also a syncsafe 4 byte number (28 bits), which is the total length of all of the frames. So a frame with a length over 2^28 bytes isn't possible.

Those are some big maximum frame sizes. The way I see it, I have two options for these large frames. The first is to make a copy of these long frames and store as files somewhere; maybe somewhere in the user's temp directory, maybe a directory that you specify to the library that's in your application's directory. Then if you need to write that frame to a tag, you copy that frame's file over to the new tag. An obvious disadvantage of this approach is that it takes time to copy bytes like this.

The second option is to make a reference to the frame: the file, the starting position (offset) and the length. Then you don't have to copy the data in the frame to a temp directory in between. The disadvantage is that you can't move the source file, or the frame reference points to a missing file and the frame can't be written -- d'oh.

This might make you question the lifecycle of a Tag2 object produced from the jid3rL library. Will it be used right away, or will it have to stay around for a long time? In the case of AudioMan, the data is transferred to another object almost immediately, and the Tag2 is discarded. But as a library writer I don't think I can assume that will always be the case, which seems to make the temp file option a better choice. Thoughts?

posted at August 18, 2004 at 08:48 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (7)

# Automatic Object Test Generation

Something that's becoming apparent after working on jid3rL is that it's nice to have well tested objects but it's a pain to test them. If you start from the bottom with very well tested objects, then you'll have less defects when you integrate them together and use them with other code.

The problem is actually testing them, making them bulletproof. The equals() and hashCode() methods have very specific "contract" conditions that are easy to test. But writing the tests is tedious, especially for simple immutable objects like the ones I use in jid3rL.

It would be nice if I could generate these tests automatically and save a lot of work. It could be an Eclipse plugin ...

You mark the member variables that you want to count when determining object equality for that class, and then the tests are generated automatically. It could generate far more tests than you would normal do manyally, since it wouldn't take long to regenerate them again if you added a new member variable.

Speaking of that, if you did add a new member variable the Eclipse plugin could pop up an error/warning in the Problems view telling you to regenerate the tests again.

posted at August 17, 2004 at 11:38 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (5)

# jid3rL Wrapped in SimpleTag2

Ironically enough, after blogging about how the full jid3rL library might be "a bit much for general use", I ran into that problem integrating jid3rL into AudioMan. It was just a lot more than I needed.

I thought about the easiest way it could be presented if the user only needed to read/modify/create a few basic frame types from/in an MP3 file. I made a class called SimpleTag2 which wraps around a full Tag2 object and has only a few methods:

public SimpleTag2(Version version) //constructor
protected SimpleTag2(Tag2 tag) // constructor only used by parse()

public String get(SimpleFrameId id)
public void set(SimpleFrameId id, String value)

public static SimpleTag2 parse(File f) throws IOException
public void writeToFile(File f) throws IOException

The SimpleFrameId is another one of those Java enumeration classes, and contains a few of the popular frame ids abstracted out:


Then if you want to read a file's tag, it looks like this:

File f = new File("soundgarden - superunknown.mp3");
SimpleTag2 tag = SimpleTag2.parse(f);

System.out.println("track name: " + tag.get(SimpleFrameId.TRACK_NAME));

You can modify that file by continuing...

tag.set(SimpleFrameId.TRACK_NAME, "Superunknown");

You can also create a tag from scratch and write it to any file:

SimpleTag2 tag = new SimpleTag2(Version.v2_3);
tag.set(SimpleFrameId.TRACK_NAME, "Transatlanticism");
tag.set(SimpleFrameId.ARTIST_NAME, "Death Cab for Cutie");

File f = new File("trans.mp3");

In exchange for a powerful API, you get something that's pretty easy to use and extensible. Thoughts?

posted at August 17, 2004 at 10:22 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (1)

# Two Tiered Approach for a Library

Implementing the id3v2 spec has been a lot more fun than I thought it would be. Yes, truly geeky fun ... but fun nonetheless. I'd really like to implement the whole spec, and I don't think it would be that difficult.

(You can download jid3rL, a Java implementation of id3v2 in progress or take a look at the feature checklist)

The only problem with implementing the whole id3v2 spec is that it's about three times more than the average programmer really needs/wants to read and write id3v2 tags. Most of the id3v2 libraries I've seen are far from complete, most only reading and writing the basic tags: Text Information, Comments and maybe Attached Picture and a few others.

The applications that use id3v2 are equally simplistic with it. WinAmp and iTunes, two of the more popular MP3 players use only the most basic frame types for their tags.

So the API for a library that implements the complete spec might be a bit much for general use. It might be more useful to have a simplified API built on top of the main library that exposes much less functionality. Even better would be a library which exposes a simpler API but allows the library user to go into the more complex API if they need to.

This idea for this came from JFace, a Java GUI toolkit library built on top of SWT. JFace uses SWT to create an easier to use API for common SWT widgets, like wrapping an SWT Table with a JFace TableViewer. This saves JFace/SWT library users a lot of setup work and exposes easier to manipulate object models. Also, while using JFace, you can still get access to the underlying SWT objects to read and manipulate them.

Completely hiding complex details in a simplified API sounds like it would be much easier for me than allowing the user to dig deeper, though it may not be. If course the programmer would also have the option of just using the more complex API as well. I haven't given much thought on the specific implementation details, I'm just trying to empathize with the users of this library and what they'd want/need from it. Thoughts?

posted at August 16, 2004 at 10:06 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (1)

# When is a Feature "Complete"?

My previous post about the jid3rL feature checklist indirectly brings up a good point: when is a feature "complete"?

In relation to a standards specification like id3v2, you might say that a specific feature is complete when it implements its part of the spec.

Depending on the granularity/hierarchy of the feature, the completeness of it could also come into question. For example, the id3v2 tag header has an unsynchronization flag, meaning that the tag following the tag header is encoded to be unsynchronized. If I read the tag header and understand it - including the unsynchronization flag - am I then complete? Or is the "tag header" feature complete only when I support the unsynchronization feature itself? These dependencies are up to the project manager, but should be noted somewhere.

For that situation I've separated reading the tag header from support for the features dictated by the header. I might want to make this more explicit in the checklist.

Another issue you have to be concerned about is regressions, or functionality that once worked but has been broken. Once you indicate that a feature is "complete" in a document you'll probably want to maintain that completeness. It is not uncommon for dependencies between features to reveal themselves during development, and a dependency could break one complete feature while you're implementing another.

To ensure that this does not go unnoticed, a suite of regression tests should be used to confirm that completed features are still functioning as new features are added. It would also be easier to verify the effectiveness of this regression test suite if the tests were organized to correspond to the specification document. Then it's much easier for people to independently verify that the library functions properly.

Note that this level of "completeness" is often unique to libraries which implement specifications or projects following a more spec-driven waterfall software development style. Most software is dynamic, changing over time with changing user demands. If features are changing, dependencies between the features change and the definition of "complete" changes for those features.

This is when more concrete definitions of features for a specific version of the software are necessary, to leave little room open to interpretation and allow confirmation of completeness. The project manager will also need to have some way of keeping track of all of the testing results and project implementation status data; either a speadsheet or custom software.

The project manager then needs to ensure that the regression tests are up to date with the current state of "complete" for each feature as the project changes, otherwise the existing regression tests may given him a false sense of security that everything is OK. Code coverage analysis may be able to help identify weakly tested areas.

posted at August 14, 2004 at 01:18 AM EST
last updated December 5-, 2005 at 05: 1 PM EST

»» permalink | comments (0)

# Tracking the Progress of an Implementation of a Standard

Software applications are often moving targets. Customer priorities change over time, so the feature demands change over time. Maybe you're making a COTS software product and you want to bring it into a new market. There are lots of reasons.

Software libraries implementing standards, however, are not moving targets. They either implement the standard at a given version number or they do not [1]. The standard for a specific version number will never change.

This comes into play for jid3rL, which aims to implement the id3v2 standard for the Java platform. The id3v2 standard was published in the following versions:

2.2 in Mar 1998
2.3 in Feb 1999
2.4 in Nov 2000

So to track the progress of jid3rL I've laid out a matrix of the features in each version of id3v2 and whether or not jid3rL implements them.

[1] If the standard is specific enough. If the standard leaves room for interpretation then whether or not a given software product actually implements the standard can be called into question.

posted at August 13, 2004 at 11:59 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (5)

# jid3rL is Building Nightly

I'm not quite ready to release jid3rL 0.1 yet, but it's building nightly from CVS now. If you haven't been reading this blog lately, jid3rL is the LGPL Java id3v2 library I've been working on so AudioMan can have id3v2 read and write support for MP3 files.

Just like AudioMan, you can check out the code coverage of the unit testing against the library. Right now it's not so good. :) Not as much TDD as I would like...

There are a lot of unsupported/unparsed frame types but the major ones are there. One of my main goals with this project is making an easy to use API, so I'm looking forward to feedback there. Good JavaDocs are high on my priority list, as is some sort of manual -- even in the form of API-use examples that people can just cut and paste.

I'm pretty happy with my progress in just 10 days. It will be strange switching gears back to AudioMan again.

posted at August 12, 2004 at 05:46 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (1)

# ClassCastExceptions are Useless?

James Robertson spotted something in comp.lang.smalltalk:

Just curious, but how many people out there using static languages actually try to pass invalid types to methods? How many Java users commonly get ClassCastExceptions at runtime?

My answers are:

* I never pass invalid types to methods.
* I have no memory of ever getting a ClassCastException in Java.

After using dynamic languages for a while I have come to the conclusion that static typing solves a problem that doesn't exist.

When using Java Collections I get ClassCastExceptions at runtime periodically, but it's usually because I'm wrapping the collection inside another class that has methods returning a specific type instead of just Object. When I retrieve the object from the collection I cast it to that type.

What's the real reason I get ClassCastExceptions? Because I refactor a lot. After a refactoring the wrapper class to use B instead of A you might be putting a type B into the collection and incorrectly casting it to the old A in a wrapper class method[1].

This problem will go away when generics are introduced and I can specify the type I want a collection to contain. The compiler will tell me I'm trying to put a B into collection<A> and I'll be forced to refactor it to collection<B>. Then the compiler will tell me I'm casting an item from collection<B> to an A (the place where the ClassCastException formerly occured for me). Actually Eclipse will tell me these things as I type, instead of at compile time[2].

As I understand it, and I could be way wrong as usual, languages like Smalltalk don't have this problem because they don't enforce type, so you never cast an object coming out of a collection (I assume there is a complementary group of classes to Java Collections in Smalltalk). If I were refactoring the same problem in Smalltalk, without the ClassCastException I may never know that I'm returning an object of the wrong type after the refactor (A instead of B). To catch this problem, you'll need good unit tests and you'll need to change them all from using A to B. You could always just nuke the A class, and the compiler will tell you it's invalid. With static typing in Java and generics, you don't have to nuke A.

[1] Note: this is different from putting in new type A' and incorrectly casting it to the old version of A. A refactoring IDE like Eclipse would change all of the references of A to A' and you wouldn't get ClassCastExceptions because A doesn't exist any more.

[2] Eclipse is compiling the code in the background as I type so I don't need to compile it after.

Update 11:41 am James responds:

You don't have this problem in Smalltalk because it's not the sort of problem you tend to get yourself into, period. Say I had a collection holding Foos. If I refactor, and I end up having a collection holding Bars (completely incompatible), then I have bigger problems. Even so, I would have had to refactor all the surrounding code that accesses the collection elements - and if I didn't have tests under those circumstances, I'm in trouble whether I have static typing or dynamic typing. To be brutal, if you trust the compiler to solve this for you, then you shouldn't be writing code.

I agree, testing is critical. If you're making up a new class you have to unit test it, and tests will catch refactoring errors from B to A in Java or Smalltalk.

I wouldn't trust the compiler to test the result of a refactoring for me any farther than I could throw it, but with static types and generics it will be a lot more obvious when I break something.

This probably seems like type hand-holding, but it's nice to see something break immediately as you type it, just like you expect it to -- even before you run the unit test suite. It's good immediate feedback.

If you were unable to unit test (as stupid as that sounds, most people still don't) would you rather use a dynamically or statically typed language? Would static typing help? Assume for this question you are equally familiar with Java and Smalltalk and neither has an advantage in that respect.

Personally I'm still in Java land because it's what I need to know -- at the very least -- to get a job. Java is in more places than Smalltalk or Python. I'll move on to Python or Smalltalk in a bit, and if I've used Java and know it well I'll be better informed about the differences between the languages. I still have faith that I will "see the light" in regards to dynamically typed languages. I was just stating a case for ClassCastException in Java.

Update 1:34 pm Ian Bicking says on his blog:

A tool can analyze statically typed code and say with some confidence exactly where a class or method is used -- in Smalltalk, Python, or other dynamically typed languages, refactoring is just a string match. Good naming practices can make that string search more reliable, but it's still just strings, not a fully type-annotated source.

This is one of those unfortunate places where you can't have it both ways. Dynamic typing and late binding is resistant to static analysis, and static analysis can be used for good things. (And Python is actually more resistant than most dynamic languages.)


posted at August 10, 2004 at 09:01 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (12)

# jid3rL with FramesGroup

Here's an update to my earlier post, An Intro to jid3rL. The classes and constructors have changed to:

Tag2(Version version)
Tag2Header(Version version, Tag2HeaderFlags flags, int tagLength)
Tag2HeaderFlags(Version version, byte flags)

abstract Tag2ExtendedHeader(int paddingLength, byte[] crc)
Tag2ExtendedHeaderPoint3(int paddingLength, byte[] crc)

abstract FrameHeader(String id, int frameLength)
FrameHeaderPoint2(String id, int frameLength)
FrameHeaderPoint3(String id, int frameLength)

Language(String code, String description)
Version(byte code, String description)

abstract Frame(Version version, String id)
TextInformationFrame(Version version, String id, EncodedString information)
CommentsFrame(Version version, String id, TextEncoding encoding, Language language, String description, String text)
UniqueFileIdentifierFrame(Version version, String id, String ownerIdentifier, byte[] identifier)
UserDefinedLinkFrame(Version version, String id, EncodedString description, String url)

UnsupportedFrame(Version version, String id, byte[] payload)


abstract Id3v2Segment()

TextEncoding(byte code, String desc)
EncodedString(TextEncoding encoding, String text)
EncodedString(TextEncoding encoding, byte[] buffer)


Colour legend

public class
package visible class
private constructor

Keep in mind that the green (public) classes are the only ones accessible to users of the library. This is a update of the older explanation of these classes, which you might want to read first.

I made the two "constant" classes I talked about last time. These types of classes are used because Java doesn't have enumerations -- yet. They limit the range of input on a parameter and simplify testing. Those two classes are Version and Language.

There are only three Version instances: 2.2, 2.3 and 2.4. The byte used in the private constructor are just the minor version numbers 2, 3 and 4. The Language class has contants for all 500 or so languages in ISO-639-2. Yes, it took quite a bit of tedious/brainless typing to make this class! To improve readability I didn't capitalize the Language constants, so English is Language.English. Language is only used in CommentsFrame so far.

Before I fixed my flag reading code I thought I was going to have to read id3v2 tag extended headers. Turns out the test files I have don't have extended headers but I'm going to save this work anyway.

To simplify the Tag2 class I took all of the frame-related methods out of it and made a new class: FramesGroup. FramesGroup uses the Hierarchy class I blogged about yesterday to organize frames. Tag2 only has one FramesGroup.

posted at August 10, 2004 at 06:58 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (3)

# The id3v2 Frames Hierarchy

There are many id3v2 frame types that allow instances of the same type with the same id to exist in a tag. One of the most frequently used is the Comments frame type.

The Comments frame type has the following pieces of information:

- language
- short description
- actual text

The id3v2 spec says that you can have multiple Comments frames as long as they don't have the same language and short description. So you might have a few comments with a different short description, or with the same short description but translated into several languages.

As an aside, WinAmp and iTunes both use a zero length string for the short description of user-entered comments (iTunes likely just followed WinAmp's de facto standard). iTunes also has a few custom comments it puts into files you rip from CD with it. One appears to be a string of combined CDDB lookup numbers, one of which should actually be stored in the Unique File Identifier frame. The other comment appears to be normalization data.

Other id3v2 frame types only let you have one frame of that type with the same id. This is true for the Text Information frame type, which holds values like artist name, album name, track number, etc. You can only have one Text Information frame in a tag with a given frame id.

A good data structure for holding frames is a Java Map. Since each different frame type has frames with a unique id, you map the id to the frame. If the tag doesn't have a frame with that id, the map returns null when you try to get it. But wait, there's a problem ...

Oh yeah, those Comments frames -- they all have the same frame id. If I only used the frame id with the Map, I'd only be able to store one comment at a time. The solution I'm using has maps within maps recursively. Except I wanted it to be a little more arbitrary, so I made a new class I called Hierarchy to abstract this away.

A Hierarchy is a wrapped Map but because it has multiple levels of hierarchy it uses paths instead of keys. The paths are just a List of any types of Object you want, except Hierarchy itself. There are three main methods to this madness:

public Object put(List path, Object o)
public Object get(List path)
public Object remove(List path)

The put method lets you create the hierarchy by stipulating a path. The Object o will be placed at the end of the path. The get method gets an object based on a path. If the path is invalid or no object is found, the method returns null. The remove method will remove the Object at the end of the given path from the Hierarchy and also remove any empty paths this creates.

With my Comments frame example, the path will be:

1. frame id
2. short description
3. language

Text Information frames can be found in the Hierarchy by just using the frame id as the path, since there can only be one with each id per tag.

posted at August 09, 2004 at 04:29 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (2)

# Java Signed Bytes and Decoding Flags

The id3v2 spec has bit flags in it, which are either 1/0 (true/false). These flags are jammed into bytes and each byte can contain up to eight flags.

For example, here's an excerpt from the 2.3.0 spec for the tag header flags:

ID3v2 flags %abc00000

The % notation is shorthand for bits. a,b and c are bit flags that can be turned off/on and the rest are expected to be zeros. Altogether these 8 bits make up a single unsigned byte.

Unsigned byte values range from 0 to 255. The most significant bit is on the left (a, worth 128 if on) and the least significant bit is on the right. If you don't know how to decode bits from an unsigned byte, you can read up on it.

The problem is that Java represents bytes as signed bytes, which range from -128 to 127. I'm reading signed bytes from the files, so I have to deal with them. You might say "OK, just take the signed byte and add 128 to it to get an unsigned byte". At first glance that might seem correct, but it's not. To understand why you have to get under the hood of the Java signed byte.

Java stores its signed bytes in two's complement notation. If only the a flag is on from the example above, the signed byte value would be -128. Adding 128 to that gives you %0000 0000 in bits: no flags on. This is obviously incorrect, we know the first flag should be on.

The trick is to use a bitwise operator in Java that doesn't care about sign: the unsigned right shift operator >>>. Here's the method I wrote to determine if a flag in a byte is on:

protected static boolean parseFlag(byte flags, int position)
   int shifted = flags >>> position;
   return (0 != (shifted & 0x01));

The >>> operator shifts the bits of flags to the right so that the bit in the desired position ends up being the least significant (right-most) bit. Zeroes are shifted into the left side. Then I bitwise AND %0000 0001 to shifted so that all of the other bits are zeroed except the least significant bit that I'm concerned about. If the resulting value isn't 0 then the bit in question was set and the method returns true.

How did I verify my code works? Unit testing, of course. Here are the tests I do:

bits       byte description
%0000 0000    0 none set
%1000 0000 -128 a set
%1100 0000  -64 a and b set
%1110 0000  -32 a, b and c set
%1010 0000  -96 a and c set
%0100 0000   64 b set
%0110 0000   96 b and c set
%0010 0000   32 c set
%0001 0000   16 bit 4 set
%0000 1000    8 bit 3 set
%0000 0100    4 bit 2 set
%0000 0010    2 bit 1 set
%0000 0001    1 bit 0 set

I could be much more comprehensive, especially on the negative side. But this seems good enough for now.

posted at August 09, 2004 at 08:30 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (4)

# An Intro to jid3rL

OK I'm ready to talk about my id3v2 library now, which I'm going to name jid3rL. I don't want to jinx it, but everything is working out well so far. The trick, as I guessed in my last post, was to approach it from the write point of view instead of the read. Then things became clearer and cleaner.

If you're unfamiliar with the id3v2 spec, you can read about all of the versions here. It looks intimidating at first but you only need to know about 25% of it. The rest isn't used very much and I'm not sure about supporting all of it in this library.

Here's a simple rundown of the spec. An id3v2 tag is a bunch of bytes at the start of a music file, usually MP3 files. It consists of, in order:

- tag header
- bunch of frames
- footer (only v2.4)
- padding

Frames contain data about the music in the file: metadata. Things like the song name, artist, album name, etc. Each frame is:

- frame header
- frame data

The id3v2 format has gone through 3 iterations of spec: 2, 3 and 4. These are commonly referred to as id3 versions 2.2, 2.3 and 2.4. This is why there is no 2.0 and 2.1.

The only things I've discovered that are different about the three specs are:

- frame headers
- flags in the tag header
- tag extended header (only version 2.3 and 2.4, and they are different)
- tag footer (only version 2.4)

This has interesting implications for a library, since I can easily abstract these details out and use the same objects for:

- tag
- tag header
- frame
- the different types of frame data

... and that's what I did. Here are the objects I have in the library, all in the same package. Constructor signatures listed.

Tag2(byte version)
Tag2Header(byte version, Tag2HeaderFlags flags, int tagLength)
Tag2HeaderFlags(byte version, byte flags)

abstract FrameHeader(String id, int frameLength)
FrameHeaderPoint2(String id, int frameLength)
FrameHeaderPoint3(String id, int frameLength)

abstract Frame(byte version, String id)
TextInformationFrame(byte version, String id, EncodedString information)
CommentsFrame(byte version, String id, TextEncoding encoding, String language, String description, String text)
UniqueFileIdentifierFrame(byte version, String id, String ownerIdentifier, byte[] identifier)
UserDefinedLinkFrame(byte version, String id, EncodedString description, String url)
UnsupportedFrame(byte version, String id, byte[] payload)

abstract Id3v2Segment()

TextEncoding(byte code, String desc)
EncodedString(TextEncoding encoding, String text)
EncodedString(TextEncoding encoding, byte[] buffer)


Colour legend

public class
package visible class
private constructor

All of the classes generate immutable objects. This means they have final member variables initialized in the constructor which cannot be changed and that makes the objects automatically thread-safe with no additional work. Most classes only have only one constructor. Sun Java guru Joshua Bloch explains immutable objects and why you should favour them in his book Effective Java.

Why is there no FrameHeaderPoint4? The frame header spec didn't change from 2.3 to 2.4, so I use FrameHeaderPoint3 for 2.4. Update Actually it did change slightly: 2.4 uses sync safe frame lengths, so I made a FrameHeaderPoint4.

Id3v2Segment is the superclass for every byte-based class in this library. It has an abstract method byte[] bytes(), which every concrete subclass must implement, which is the segment represented in bytes. It also has a method size() which is just the length of the byte[] returned by bytes().

Notice the two constructors for EncodedString. The encoded string is stored as an encoding and a string, but I accept encoded bytes in a constructor as well. In that constructor the string is decoded and also stored as a string. EncodedString then lets you access the string as a decoded string or encoded bytes, based on the encoding. TextEncoding has a private constructor because it is only used to make constants that are available statically.

ByteArrayTools is a static class (which explains the private constructor, see java.lang.Math) with some handy static methods for manipulating arrays of bytes; byte[].

There are some other candidates for constant classes, like how I did TextEncoding. In another language these would be known as enumerations but Java doesn't support them yet so you have to make your own. Java will support enumerations in version 5 (1.5). It's a good way to limit the input range on a variable, and simplifies testing. Those other candidates are the version number and language. There are only three versions numbers (2,3 and 4) and language is a String from a defined standard (ISO-639-2, for example English is "eng").

One of my primary goals is to have a good API. It shouldn't break in later versions -- I can add to it (for example, new supported frame types), just not change method signatures or remove from it. I want to be in that state before I release 1.0.

OK peanut gallery, what do you think of my first run? There are enough software engineers reading this blog that I should have some good peer review. I'll post the full code soon, under the LGPL.

posted at August 08, 2004 at 08:36 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (2)

# Empathize with your Blog Audience ... if you want

There's no specific spirit to blogging, and that's probably the point. It's the freedom to be able to publish whatever you want to say on the Internet for anyone in the world to read. Now that's freedom.

Then some bloggers gain an audience, either because the content of the blog is interesting or their friends and relatives are curious and want to read it. Sometimes knowing you have an audience affects the content of the blog and sometimes it doesn't. Some people can blog like no one is watching, as it were.

A really interesting use of blogging is by celebrities, often misquoted or misrepresented by the press, who want to set the record straight in their own voice much to the horror of their PR reps. With a blog they can get straight to their audience. Examples are Mark Cuban and Billy Corgan.

If your intent is to get down your own thoughts despite your audience, that's cool. Your blog then becomes a searchable and highly accessible notepad. But if you're trying to communicate an idea to your audience, you have to empathize with them. You need to explain the back story, expose acronyms, technical details and confusing parts. Try to make your posts explain themselves, even if it's just linking to old posts or other places where people can read up on specific background information.

This kind of thing takes a lot of work -- I'm not that great at it either but I'm working on it. It takes more work to prepare a post this way, sure. But doing it will make your blog better for your audience.

posted at August 08, 2004 at 07:55 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (1)

# Library Licenses

Back into uncharted IANAL territory again with software licenses. Before I start posting code on this blog I wanted to get licensing out of the way for the Java library I'm writing.

Do any of you have opinions on a good open source license to use? Here is my intent:

  1. Any changes or additions made to the library should be contributed back to the project.
  2. Open and closed source projects are free to use the library as long as the project gets credit somewhere.

Anything else I should be worried about?

So far I'm considering:

Common Public License (CPL) Version 1.0
GNU Lesser General Public License (LGPL)

The GNU General Public License (GPL) is too viral to be used in this case since any program that uses GPL code must also be GPL. That violates term #2 above.

posted at August 06, 2004 at 02:13 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (1)

# Exposition

Writing this id3v2 library has turned out to be just as hard as I thought it would be. I'm still in the experimentation stage at the moment, trying to figure out what the code can do for me. There's no unit testing outside of using real mp3 files to verify everything reads properly. You could say that I'm prototyping while learning the domain.

One big problem is starting the library as a read library. This skews everything over to one side, when I want to be able to write as well. A useful exercise may be to start it as a write library instead, and see how much is the same.

The other issue is encapsulation; hiding details not only to make the API simpler, but also to provide fewer entry points for library users. This lets me change more of the guts of the API without breaking the contract. Library users like that.

Java packages are useful for encapsulation. Java classes can be either public or package visible. Package visible classes can't be seen outside of the package they are in (or the library for that matter), so they're good for encapsulating functionality. I only want to expose certain classes to the user of the library by making them public. The others will be hidden from the user.

When I get a good enough first run at it done I'll post some details.

posted at August 05, 2004 at 04:01 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (10)

# Early Thoughts on Library/API Writing

While writing this little id3v2 library, the first I've written that I plan to release and support, I've been thinking about a few of the issues people have when writing libraries and APIs. Here's a few I like:

  1. Writing an API is hard.
  2. Unit testing is critical, which means
  3. Test code coverage analysis is also critical.
  4. Don't release the official 1.0 early. Wait until the API entropy slows but
  5. Get people to test the early versions and give feedback.
  6. Give people what they need, not what they want. That means you'll have to
  7. Listen to your users, but
  8. You can't please everyone, so you have to
  9. Be able to make intelligent compromises.
  10. Empathize with your users by using the API yourself in an application.
  11. Don't bite off more than you can chew; start small.

There are some other issues for dealing with data "in the wild". You can't just throw an exception when you get bad data, you should be expecting it. So you have to have a way to represent it that tells the API user "hey, this is bad".

It reminds me of the discussions around malformed syndication feeds and what feed readers should do with them. When developers started thinking more about the users and less about themselves, they realised that reading malformed feeds was a good idea. But the API should still indicate it's malformed.

The good news about id3v2? The last version is four years old. In high tech terms that's ancient, the specification entropy has slowed and it doesn't look like it's going anywhere. This reduces risk and the need for agility.

Despite its age, id3v2 is actually gaining in popularity. It's the de facto tagging format for mp3, so companies will trust it. It has hardware support. It's used in popular music players and online music stores. It's everywhere. It's amazing that there isn't a free Java implementation out there that's still supported.

posted at August 04, 2004 at 10:03 AM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (0)

# Tiger Woods I Am Not

I golfed at the Renfrew Golf Club on Saturday with some friends. The course hasn't changed much in the years since I was in high school and golfed it regularly.

We got out there at about 8am, and it was raining hard. I pressured my friends to just get out on the course, hoping that eventually it would die down. After the first hole (the 10th, we golfed the back nine) two of the guys dropped out and went back to get a raincheque. But my other friend and I pressed on.

It didn't stop raining until about the 15th hole. On the 14th, a 175-yard downhill par 3 with a forest down the right hand side, I picked my 5 iron. I planned on hitting it with a 75% swing. The rain was still coming down. My grips were wet and my glove was soaked.

I didn't think too much of it until the club slipped out of my hand on the follow through of my tee shot and -- because I'm left handed -- went flying towards the woods on the right hand side, end over end. The tee is elevated so naturally the club had no problem hitting the top of the trees first.

I went down to the landing spot to search for my club. I was looking around on the ground when my friend told me my club was actually 30 feet in the air with the end of the club stuck in a Y branch. Luckily the tree was only about 6 inches wide at the base and I could shake it. Unluckily I didn't have the inertia to jar it loose.

So my friend came down and gave the tree a good shake and the club fell down. The worst part? I also lost the ball, even though I thought it went straight off the clubface and dribbled down the hill off the tee. Both of us were too busy watching the club fly into the woods with our mouths open. Ha.

I ended up scoring a 52 in the rain. One par. Lots of fun. Not too shabby.

posted at August 02, 2004 at 05:04 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (1)

# id3v2 throws UnsupportedOperationException

After some searching for a Java id3v2 library to use with AudioMan, I'm about ready to give up. Surprisingly enough, even though the last version of the id3v2 spec has been out for almost four years no one has made a complete free Java implementation of it.

If that wasn't enough, the current incomplete implementations aren't being maintained. So if I have a problem (if my users have a problem), I have to fix it myself. Welcome to the world of free software. :)

So it's looking more and more like I'll have no choice but to make yet another incomplete implementation. I just can't bring myself to accept the risk of using any of these other ones. None of the ones I've seen even have a single unit test. Yikes.

The good news? A few of the implementations have good ideas. So I can take those good ideas and make them better. I can also test the crap out of my library .... why? Because I'll insist on it. Besides, making things bulletproof is fun -- and Roy loves shooting bullets at my software.

The bad news? This will take me at least a week. Time to put my head down and hack ... I mean uh ... TDD. :)

PS> I updated the resume. I'll be looking for a job soon.

posted at August 02, 2004 at 03:05 PM EST
last updated December 5-, 2005 at 02: 2 PM EST

»» permalink | comments (1)

Search scope: Web ryanlowe.ca