«« Bye Bye Hotmail Week 11 Status Report »»
blog header image
Looking Forward to Metadata

I've been concerned with file metadata for some time -- not just with the Durham and AudioMan projects but also when it comes to my own personal file management.

As far as my music collection goes, its at the size right now where I believe that accurate and complete metadata is the only practical way to manage it. Since I have no tool to do that with right now -- outside of manual tag editing in iTunes -- and no massive storage device to hold my entire collection on, I am currently not managing my collection properly.

This worries me a bit. I invest a lot of time and money in my music collection. But I digress...

Metadata is starting to get more attention lately. In the present day we have Apple's release of Mac OS 10.4 Tiger and its Spotlight search technology. After reading Ars Technica's review of Tiger I'm convinced that Mac OS X and Spotlight are still far from solving my metadata problems but they are one step closer.

Versions of Microsoft Windows that may natively support file metadata on the other hand are so far off they are still vapourware. This doesn't help Windows users much in the present. Metadata support in Windows probably won't be mature or exploited effectively by applications until years after it is released.

Third party tools like Google Desktop Search are interesting in that they put pressure on the operating system vendors to get their asses moving on metadata and search. These third party tools are showing users that metadata is useful and practical. At the same time, users don't even realise they are dealing with metadata, they just think that searching is better.

But these desktop tools are in the unfortunate position of having the rug suddenly pulled out from under them. The operating system vendors will put most of this functionality right in the operating system (Apple, to a large extent, already has). In terms of building applications on top of these desktop tools, that makes them poor candidates. In a few years these third party projects will be redundant. So developers will wait for the operating system vendors like Microsoft and Apple to make decent APIs to access file metadata.

The open source project I'm working on, the Durham Metadata Framework for Eclipse aims to give an operating system agnostic abstraction to file metadata. Right now plugins are written for certain file/content types but later metadata could be read straight from the operating system-level APIs instead (or as well).

This kind of cross-platform metadata support will be needed for Eclipse plugins and Eclipse Rich Client Platform (RCP) applications. It's not needed now and not even in two years. It will probably be needed in five years and definitely in ten years.

It doesn't hurt to work on it in advance to get the kinks out. I expect the pace to be slow (unless I get development help or funding to work on it full time) but I'll probably have metadata on my mind for my whole career. It seems like a good project to work on in my spare time.

Posted at April 30, 2005 at 10:49 AM EST
Last updated April 30, 2005 at 10:49 AM EST
Comments

"I'm convinced that Mac OS X and Spotlight are still far from solving my metadata problems but they are one step closer."

Agreed. The problem is that we're looking at the metadata problem in the middle. All the content is out there like an orphan, and only now are we trying to find that content's real identity. The problem is that there are too many orphans already for us to make a large dent.

"Versions of Microsoft Windows that may natively support file metadata on the other hand are so far off they are still vapourware."

That and some of the file metadata (right-click the column header (e.g. Name) to see the rest) that are there already are not relevant, and the fact that most programs don't write out that metadata anyways. There is a windows file metadata attribute called "Camera Model." Sounds great! Only problem is how many pictures do you have on your hard drive actually contain that metadata?!? 10%? 20%? All of them if you're lucky!! ?

Another problem I see is if MS does incorporate these wonderful file metadata, are developers going to actually write apps that fully populate the metadata of files that their app writes to? Right now, most of us don't. It's my opinion, not a fact.

» Posted by: roy at May 2, 2005 12:50 AM

Are you a little disappointed that MS had to drop WinFS from it's yet to be released Longhorn? Even though I'm a Linux user, i'm pretty disappointed. This was supposed to be a major feature of the next windows, and they had to drop it. I think that was probably the biggest mistake they could have made with this release. They could release an update once it's ready, but I don't see them doing that. Secondly, I don't see people wanting to change their file system after everything is installed and working.

I think that file metadata is something that could have some serious power. However, I also find there is a lack of metadata where it should be. Most mp3s don't have proper data, and I don't think there's metadata in any of my pictures. At least not real useable stuff. Who cares about camera model. I want to know what's in the picture. People really don't want to spend the time entering metadata. For mp3s, there are tools that will put the metadata in when you rip it. However, for pictures, videos, and any other file that you've created by yourself, entering that data most likely will never get done. Or it will get done in some instances, but not in all. And lets face it, bad metadata is almost as bad as no metadata.

» Posted by: Kibbee at May 2, 2005 08:44 AM

I'd go further -- bad metadata is WORSE than no metadata. One example is read-only media like burned backup CDs ... once that metadata is burned, it cannot be changed. If it is bad it will be bad forever.

I'm a little disappointed that Microsoft doesn't seem to be *shipping* metadata solutions in the short term. They sure are talking about it though. Maybe Microsoft just sees metadata support as a Really Hard Problem that will take a long time, a lot of work and most importantly a lot of (scary) change to solve.

But just look at the usability impact that Google Desktop Search has had and it had very little impact on the operating system internals. Google has sold metadata as useful, just in time for Apple to exploit it.

As usual Microsoft was too worried about (or comfortable in) their position of dominance to take a "risk" on being the first to release a Desktop Search product. Now that WinFS has been delayed we might see some stop-gap solution under the MSN brand to compete with Google in this "search space".

But metadata goes far beyond search, I think. Microsoft knows this and they're trying to do too much at once -- and in the process getting nothing done at all. Apple's approach of slowly improving metadata support in their operating system is, in my opinion, a better strategy.

Apple's strategy promotes a healthy and incrementally improving discussion on the merits of metadata, instead of one day having it plunked in our laps and having no idea what to do with it (and only emphasizing the chicken and egg problem metadata suffers from).

The strategy also leaves the door open and encourages third-party developers to solve the tough problems, leaving Apple with the relatively easy job of integrating those mostly mature (and now popular) solutions into their OS, something they are notorious for doing.

I think we should consider Microsoft's position of dominance from a business perspective to understand their lack of motivation. Microsoft has everything to lose by changing the way that Windows and Office works, and proper metadata support would change both products dramatically. Like Mac OS X, metadata support would probably involve changing the kernel of Windows itself.

Windows and Office are Microsoft's bread and butter; their meal tickets to pay for expensive (and money losing) experiments like the XBox. They don't want to rock the boat or take big risks to upset this reliable revenue stream. But in the process their business and marketing people are stifling innovation into an *obvious* technical area like metadata.

How far ahead will Mac OS X have to get before Microsoft pushes the panic button? I'm betting very far. Far enough for Windows users to see the greener grass and bitch and moan about it. That point might be so far down the road that it would be too late for Redmond to fix Windows...

...but we should consider this: maybe they won't care about the WinNT family of operating systems by then. Maybe they will kill it off, just like the 9x/Me family and start something completely new. That might be a more likely option than people think.

» Posted by: Ryan at May 2, 2005 09:29 AM

You know what, as scared as I am of the concept of just killing off a line of OS...I think they just might do it! *shudders* They've done it before. And once it's killed, it's gone for good.

How many people out there run Windows 3.1? Nobody. MS did a perfect job of killing that OS and moving onto Windows 95. So, looking back at History, they've done this sort of thing before.

MS just knows how to do that. They get a lot of people upset, but they do it, and then people forget about it. The only problem is, the general population is now more educated in computers than what it was 10 years ago, so this time, I don't think it'll go under the radar that easily.

Oh man. I can sit here all day and hypothesize theories. But I won't. That's enough out of me.

» Posted by: roy at May 2, 2005 10:47 AM

Killing off an old OS is something that needs to be done. It's important to realize when your old model just isn't working anymore, and it would be cheaper to completely replace it then to keep on fixing it. With Windows 3.1, this was easy, because windows wasn't an OS, and was terrible, and they knew it needed to be replaced. Same with Win 9X. They knew they couldn't go on supporting 2 separate operating systems forever, so they just picked the better OS. Eventually Microsoft will have to replace the Windows NT line. Although I don't think that will be any time in the near future. As far as I'm concerned , Microsoft needed to think more modular when they did WinFS. It seems like because it wasn't ready, none of it was ready, and none of its features could be used. Instead, had they used a modular approach, at least some of it would have been working, and they would have had something to ship when the next version of windows came out.

» Posted by: Kibbee at May 2, 2005 11:45 AM

Going off-topic now, but I like the way that Mac OS X did backwards-compatibility. Forget the fact that I'm a Mac fanboy and look at the technical advantages.

Instead of making compatibility a main part of the OS like Windows 9x APIs are in NT, Mac OS X had a seperate OS 9 compatibility "runtime". Everyone using OS 9 gradually switched over to OS X (Quark Xpress customers were a big group of late switchers because that product was delayed). Eventually there will be no reason to go into OS 9 compatibility and it can die off safely, leaving only OS X.

Instead Microsoft has code and workarounds from the Windows 3.1 era still in Windows today! It's there just to keep old software running. And it's not just one level of compatibility hacks, it's multiple. What's wrong with that picture?

Wouldn't that make anyone afraid to make major changes? The code is full of workarounds for years-old defects. Sure, it's nice and stable now after years of bugfixes and thousands of man-hours of manual testing but it's FRAGILE, not agile.

The approach Microsoft is taking with .NET is interesting. They are making a consolidated higher level API, an abstraction, to write programs against. Once that API has settled down, people will use it instead of the older bare-metal C++ APIs.

Microsoft could keep replacing legacy APIs with .NET code, going deeper and deeper into the internals. Not only are there security benefits to that because of the relative safety of the .NET runtime, but they are also move into a better position to quickly *completely replace* the lower level code (ie. the old Windows kernel) without breaking .NET apps.

Microsoft is years away from this but it seems like a reasonable NT escape plan. They may even force people to move to .NET when Longhorn arrives.

» Posted by: Ryan at May 2, 2005 12:51 PM
Google
 
Search scope: Web ryanlowe.ca