I like the idea of creating a project around spec validation and user acceptance. It seems like a reasonable thing to do for a third party, or even a second party implementing the spec. But the open source world has a few advantages that I can leverage, plus there are the inherent advantages of having an old and unchanging spec like id3v2 as well.

What I need to do then is figure out a way to get considerable volume behind this testing effort. Like code, tests need to be moderated in a way that creates a bit of a human bottleneck...

Let's say I started a web server that took mp3 files as input, read their tags and spit out the data in the id3v2 tag as a result. A user could look at this output and make a decision about its correctness. There's one level of validation already, and the user didn't have to download any code at all, just upload a file to my server.

If the user thinks the output is wrong, the user enters a comment saying what the correct tag should be. Or in later versions the web interface could support tag construction to pointpoint specific problem areas.

Then someone on the project has to go through these submissions and check them over. The ones that the users perceive are incorrect might be useful (true negatives) but every entry is a possible error, especially false positives that the users miss. This can start out as a manual validation process done by people (me), automating bottlenecks where possible/practical. A good server-side interface would also help.

The compiled tests also need to be categorized/sorted by people capable of moderating that.

The advantage with this approach is that the community is in a sense building the user acceptance test suite -- THEY are saying what they expect from a library that conforms to the test suite, not the writer of the library. It still has to follow the spec, but like most specs written by humans there's often room for interpretation.

With enough files you can get a really good idea of de facto standards that have popped up in the most popular tag editors and music players and be able to recognize them and handle them. The pool of sample files could come from all over the place -- in many different languages and variations even -- and that would stress test libraries.

A possible testing hierarchy could involve the following dimensions:

- id3v2 version (2.2, 2.3 or 2.4)
- frame type
- frame popularity
- frame complexity

The popularity and complexity dimensions are particularly interesting. It could allow libraries to conform to a "basic", "intermediate", "advantanced" or "complete" stages of implementation, for example.

People wouldn't have to implement the whole spec to see the benefit of the acceptance testing, they could roll their implementation out in layers and know exactly where they stand against the specs (versions 2.2, 2.3 and 2.4). They might stard with the most popular and basic tags first and then move on to the more complicated ones later.

There are a few things I could do to get more publicity for this project more test file submissions. It would all about creating a community around it, hellbent on exposing errors in libraries. Some people really enjoy breaking software *cough*Roy*cough*...

- post it on freshmeat
- make a sourceforge project for it
- test some of the existing popular id3v2 libraries myself, and suck their developers and users into the testing program to improve their quality
- test my own libraries

I still don't see much of a way to completely automate the process but giving users a server to test files with plus a bugzilla-like system attached to it could definitely get me a wider variety of tests, leaving me with the job of sorting them out. (I could actually use a custom front-end to bugzilla for it -- great idea)

The main point, I think, is that the community needs solid open source id3v2 libraries. It needs solid libraries period. The specs have been around for a long time and the libraries still aren't out there. Why not? If we knew where these libraries stood in terms of completion, there might be a better chance of maintaining them and getting them really solid.

Having a third party validate them could also increase user trust in the good libraries that they actually work ... and that would help eliminate the feeling of uncertainty that I get with open source libraries. Something like this could also be extended to other libraries as well. Wouldn't that be weird -- a SourceForge for spec/library validation through user-submitted acceptance tests? Sounds farfetched enough to be a good idea...