~ryanlowe/blog: Why not inherently facilitate feed deltas?

Why not inherently facilitate feed deltas?

Bob Wyman has a good summary about the RSS bandwidth situation and polling. It's interesting that he mentioned deltas and how this could save bandwidth.

What I'm really curious about is why feeds aren't organized to make sending deltas easier, rather than creating a delta system around existing feeds.

For example, a feed could be broken into a main index and feed text pages. The feed pages would contain the text of the blog posts in separate XML files. The main index would be a stripped down version of what an RSS feed looks like today: just post ids, titles, dates and pointers to the corresponding feed text pages.

A feed reader would use the main index to figure out which posts have been modified since the last time the feed was read. The reader could even be configured to not download the feed pages at all, like newsgroups, and just get them on demand when the user want to read the text. Obviously, that doesn't work well offline and doesn't even suit everyone's preferences. It's just an option.

No delta magic would have to be done to the RSS/Atom feed file on the server. If the reader already has a page in its cache, it would not need to read it again. You wouldn't even need to do HTTP conditional-get on the feed text, only the main index which is much much smaller. The main index would tell you if any of the feed pages have been updated, and the feed reader could fetch them.

I have a feeling though, that someone out there has a good reason why an RSS/Atom feed cannot be broken up like this. My burning question is: what is it? :)

Posted at September 11, 2004 at 07:34 PM EST
Last updated September 11, 2004 at 07:34 PM EST

Comments

By breaking the feed into an index-part and individual article parts, you're really only deferring the problem for a bit. You'll still need to decide on a window size for your index. Should it contain dozens of entries? Hundreds? Or, thousands? Should you have to download the entire index every time you check it? How frequently should you check the index file? Or, can you get only the updated index entries? At that point, you're probably talking about RFC3229 being used on the index file...

bob wyman

» Posted by: Bob Wyman at September 11, 2004 08:10 PM

True, you would need all of that delta technology on the part of the feed that changes the most: the index, which is the status of the feed. The rest of the feed does not change much, or only changes when the index says it did (ie. updating a post after initially publishing it).

Doing the delta operations on an index with no post body text *might be much less* server computation than doing it on a full feed. But maybe modern XML parsers are fast enough these to close this performance gap I'm perceiving, I don't know enough about XML parsing performance to make that call.

» Posted by: Ryan at September 11, 2004 08:22 PM

I completely agree with Ryan on this. Not only would having individual article parts (what in atom speak we call Entries) each available at their own url, reduce the size of the feed, make the entries themselves cacheable, and so in general be a good idea, but it has already be done in RSS2.0 [1], and so should be completely feasable in the still to be developed Atom.

A combination of both solutions would clearly be optimal, and would also give credence to Bob's saying "It's the Entries, Stupid!"

[1] http://admin.support.journurl.com/?template=rss-ext

» Posted by: Henry Story at September 12, 2004 07:22 AM