~ryanlowe/blog: Use XML for Metadata Type Description?

Use XML for Metadata Type Description?

Besides the build machine, I'm working on the Durham properties code at the moment. Properties are the name-value pairs that make up metadata, except that Durham has to handle names and values that are more complex than just one Object.

To summarize:

Each Metadata object contains a pointer to the file it's describing and a list of metadata Cluster objects
Each metadata Cluster object has a Map of properties for a specific metadata type, like ID3v2 or MPEG
Each Property contains a unique Key and an Object representing the value returned by that Key from the Cluster's Map.
Each Key contains either a Key made of a single String or a more complex Key made of a List of Objects

Durham metadata properties class diagram

We also need to describe each of these value objects because AudioMan needs to know how to create them. For example, ID3v2 tags have a Track Number property. We need to describe this property so that AudioMan can prompt the user for it and verify the value. Here are some sample restrictions:

The Key is the String "Track Number"
The MetadataType name is "ID3v2"
The Track Number value must be an integer
The Track Number value must be in the range of 1-100

Originally I was thinking of making metadata type plugin writers describe all of their properties by creating the proper objects. What a drag that would be! Not only for them, but I'd also have to verify everything they gave me. It also could be very buggy doing it this way because the definition for a metadata type plugin could be pages and pages of code.

A better format might be XML. Something like:

<MetadataType name="ID3v2"> <PropertyDescription id="TRCK" name="Track Number"> <IntegerValue minimum="1" maximum="100"/> </PropertyDescription> <PropertyDescription id="COMM" name="Comment"> <Key> <LanguageEnumeration name="Language"/> <StringValue name="Description" lengthLimit="100"> </Key> <StringValue lengthLimit="100,000"/> </PropertyDescription> </MetadataType>

The Key for a simple property like Track Number would just be the PropertyDescription id: "TRCK". The for an ID3v2 Comment the Key is more complex so we have to specify the legal values that the user can input.

If someone is making a new Comment in AudioMan2, with that description we'll know how to prompt them for the key and the value and if what they fill out is within the proper bounds. Note that in the UI we won't say "please enter the key and value" we'll probably say something like "please choose the language, enter a short description and the text of the Comment".

For the Language enumeration the user might be presented with a dropdown box of all of the available languages. OR we could just have English as the default initially and not present them with a choice until Unicode support is in. That's a detail we'll figure out later -- I'm getting off track. But supplying a default for a key value is also an option.

---

To go back to the architecture diagram I drew last week, the communication between Durham and the metadata type plugins could be XML like this.

Then Durham parses the description XML and gives AudioMan description objects so that AudioMan can validate the values that the user enters, and Durham can also validate them before it stores the values in the cache (ie. a persistent database).

This is a pretty rough idea right now. What do you think?

Posted at March 08, 2005 at 05:50 AM EST
Last updated March 08, 2005 at 05:50 AM EST

Comments

Validation of a value could be more than a simple String.length or int range test.

For text it maybe a good idea to be able to have a white / black list for accepted chars. Do you do single chars or ranges? Both? Would the value have to be in a specific format (i.e. for Dates: "2005-03-08" or "March 8, 2005", ...)

Will static xml be enough, or will you have to go with something like Jelly?
http://jakarta.apache.org/commons/jelly/

Not to be a shit disturber, but validation can (and will) bite you (and everyone) in the ass at some point, and I don't want you to choose a restrictive medium.

I have not used them, but in struts they have "dynamic" form classes (data classes that hold data and control validation) called DynaForm's. You may want to take a peek at those just for a heads up and maybe use that as a format to save yourself time.

Hope that helps! ;-)

» Posted by: Jim at March 8, 2005 10:02 AM

Thanks for the comments Jim, though it sounds almost like too much validation at this point.

I'm not really interested in validating data that much -- I just want to make sure values don't get too long to fit into fields.

Something we should keep in mind though -- and you make this point indirectly -- is that whatever way we do choose to get metadata plugin writers to specify properties that's what we'll be stuck with until the next major version.

Ya it's good to look ahead ... but not too far. I don't want to be an architecture astronaut (see Joel Spolsky).

Besides, YAGNI.

» Posted by: Ryan at March 8, 2005 10:17 AM

You asked me to provide some examples, so here's a simple hypothetical one.

2 types of file formats (type1, type2) that have a "comments" field.
Type1 is "old school" and only supports ASCII comments. Type2 supports an extended char set (Unicode) that lets you put in chars in with accents (French). You'll have to be able to alert the user (validation) if they have entered illegal chars.

It may be enough to just validate for a char set like ISO-8859-1, but other times you may have to put in a specific range (only chars a-g).

I wanted to give an actual concrete example. I'll try and come back and give a better example.

» Posted by: Jim at March 9, 2005 07:52 PM

That sounds a lot like the situation with ID3v1 and ID3v2 to me. ID3v1 only supports ASCII characters and ID3v2 supports Unicode.

I haven't talked much about this yet, but here's the approach I think I'll go for with ID3: there will be one plugin for ID3 and it will write ID3v2 primarily. If the user wants ID3v1 for backwards compatibility then it will write the same data that's in the ID3v2 fields to ID3v1 fields.

Obviously there are limitations to that -- character sets being the most obvious. There are also many more types of fields in ID3v2 that cannot be represented in ID3v1, so there will be missing information.

If the ID3v2 data is in a character set that cannot be represented by ID3v1 then the field isn't written to ID3v1. If it's only one non-ASCII character (like a © symbol) then a space or underscore could be written in its place.

» Posted by: Ryan at March 9, 2005 08:09 PM

If you knew what encoding to use, I think that you may be able to just get java to make a string with that encoding:
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/String.html#String(byte[],%20java.lang.String)

That way you could handle "illegal" chars in a "standard" way and not have to make up your own convention.

btw, in my experience when something doesn't know how to represent a char, it changes it into a question mark. That's something that I have fought many times. Damn windows character sets!
http://old.no/charmap/winadd.html

If you want to see what I mean, encode something with UTF-8 and decode it with ISO-859-1 and the fun ensues! ;-)

» Posted by: Jim at March 10, 2005 07:57 AM

that should be ISO-8859-1...

» Posted by: Jim at March 10, 2005 07:59 AM

Yep, I've got encoding Strings down pat. I'm talking about handling the conversion properly. You can convert from Unicode to ASCII quite easily if all of the characters are in the ASCII portion of Unicode.

If one of them isn't ASCII then just put a space in its place. If the whole value is non-ASCII don't fill out that field in the ID3v1 tag. Pretty simple.

I still don't understand how validation can bite me in the ass though.

» Posted by: Ryan at March 10, 2005 08:21 AM