[mythtv-users] Mooting architecture for a DataDirect replacement

[Peter Schachte]

Rod Smith wrote:
> On Saturday 23 June 2007 09:23, Peter Schachte wrote:
>> If there's an update that says that station X is
>> airing program Y at time Z, and another update sent an hour later that says
>> no, it'll be at time Q instead, then you don't want to handle them in the
>> wrong order.
> 
> This would only be a problem if the code were poorly written (you snipped my 
> original comment to that effect) or if the original data never arrived. Each 
> message could have a list of the original files and patches upon which it 
> relies, and if messages arrive out of order, the client-side software will 
> know to delay processing of the later data until the earlier data is 
> available

OK, that certainly helps.

> and/or to not apply earlier changes that would overwrite more 
> recent ones.

That only works if your updates are always complete replacements, never just
corrections.  Otherwise if one correction updates the end time of a program and
 the next one changes the subtitle, but they arrive out of order and so you
don't apply the first change, then you wind up with the wrong end time.

> Alternatively, the software could cache all the original update 
> files, detect the out-of-order arrivals, and re-run the whole sequence from 
> the start. Obviously a lot of details would depend on the exact file formats 
> used, but I doubt if this problem would be all that difficult to solve.

It certainly requires some care to get it all right.  It would probably be
worth looking at the "theory of patches" that the darcs revision control system
is based on.

And there's still nothing you can do about a posting that never arrives.

>> A [requirement] is that if something goes wrong and somehow myth doesn't have the
>> correct schedule, it should be able to tell, and be able to correct it. 
>> It's hard for me to see how to satisfy that with NNTP.
> 
> NNTP isn't the issue here; it's error detection and correction (or lack 
> thereof) in the encapsulated files.

Agreed.  A lot can be achieved with redundancy.  But this seems to be just
working around the problem created by having a decentralized distribution
system.  Has anyone really evaluated whether the number of downloads times the
size of the downloads (assuming that differences can be downloaded, rather than
a full 2-week schedules every day; that should save more than an order of
magnitude) is high enough to be a problem?

>> Someone else suggested bittorrent for this, and it actually has some
>> properties that make it quite appropriate.
> 
> One problem with BitTorrent is that it's strongly associated with piracy.

Sadly, people often have knee-jerk reactions, without thinking much.  OK, so we
make a new implementation of the bittorrent protocol using different ports, and
call it "CoastGuard" (because it thwarts pirates) or "ApplePie" or
"Motherhood".  Surely they can't complain about motherhood?

But seriously, leaving aside non-technical issues and clueless FUD-slingers, a
swarming distribution technology distributing signed, compressed differences
from a trusted central site would seem to me to satisfy all the requirements
better and more efficiently than something like usenet.  But I agree that
usenet, augmented with some redundancy and a fall-back web or ftp site that
could be used when an article gets dropped, could also fit the bill.  I really
don't see its advantages, though.

The most interesting, and I expect the most difficult, part of the puzzle has
nothing to do with distribution, though.  The trick is to get the data in the
first place.  Good luck with it.  Here in Australia, where we don't have the
Feist decision, it's effectively impossible.  One TV network claims to *own*
the guide data for (almost?) all the stations in the country.  Yeah, I can't
see how they get away with it, either.  They're also trying to sue the only
commercial provider of guide data into oblivion, and they use IP tracking to
keep people from scraping their web site.  They *really* want to kill the DVR
industry before it takes hold.  And they don't even seem to realize they're
trying to hold back the tide.

-- 
Peter Schachte              I worry that 10 or 15 years from now, [my child]
schachte at cs.mu.OZ.AU        will come to me and say 'Daddy, where were you
www.cs.mu.oz.au/~schachte/  when they took freedom of the press away from
Phone: +61 3 8344 1338      the Internet?' -- Mike Godwin