[mythtv-users] Mooting architecture for a DataDirect replacement

[Rod Smith]

On Sunday 24 June 2007 02:45, Peter Schachte wrote:
> Rod Smith wrote:
>
> > and/or to not apply earlier changes that would overwrite more
> > recent ones.
>
> That only works if your updates are always complete replacements, never
> just corrections.  Otherwise if one correction updates the end time of a
> program and the next one changes the subtitle, but they arrive out of order
> and so you don't apply the first change, then you wind up with the wrong
> end time.

That does bring up a question that needs answering before such a system could 
be formally designed: Just what form do updates take? They could be similar 
to diff files (or be ACTUAL diff files, for that matter), changes to 
individual fields in the file (as you imply in the above), changes to 
multiple fields in the file, or complete replacement files. Each possibility 
brings a different set of challenges.

Before answering this question, we'd need to know how often changes are likely 
to be propagated through the system. If there's an average of one update per 
lineup per week, then the overhead of providing updates as complete 
replacement files would be minor (particularly if each file is small, such as 
an individual show's description), and the simplification of everything that 
cascades from it would make it worthwhile to do it this way. If there are an 
average of ten updates per station per day, fixing minor things like typos in 
program descriptions, then some other update method makes sense and the logic 
to handle it would have to be correspondingly more complex.

> And there's still nothing you can do about a posting that never arrives.

*ONCE AGAIN*: There's par2, which is designed for precisely this purpose. It 
provides one or more data-recovery postings that enable the recreation of 
missing data, similar to the way certain RAID levels work. Actually using 
this technique for this particular application would take some careful 
planning (you'd need to create reasonable groupings of postings to which a 
given set of par2 files apply), but it ought to work. Of course, even this 
will break down if the user's NNTP server is sufficiently unreliable, but 
when it gets to that point the solution is for the user to find a better NNTP 
server.

> >> Someone else suggested bittorrent for this, and it actually has some
> >> properties that make it quite appropriate.
> >
> > One problem with BitTorrent is that it's strongly associated with piracy.
>
> Sadly, people often have knee-jerk reactions, without thinking much.

Yes, but we're still stuck with that. Once a technology, such as BitTorrent, 
has been tarred with a negative label, it's hard to disassociate it. Since 
we'll probably need the cooperation of those who don't see the issue in the 
way we do, using BitTorrent will put the project at a disadvantage from a 
data-acquisition point of view. This isn't fair, but it's life.

> But seriously, leaving aside non-technical issues and clueless
> FUD-slingers, a swarming distribution technology distributing signed,
> compressed differences from a trusted central site would seem to me to
> satisfy all the requirements better and more efficiently than something
> like usenet.  But I agree that usenet, augmented with some redundancy and a
> fall-back web or ftp site that could be used when an article gets dropped,
> could also fit the bill.  I really don't see its advantages, though.

One advantage of Usenet is that, in the modern world, files get distributed 
quickly, and it's not unreasonable for clients to check their local servers 
for updates very frequently. This would enable updates to be disseminated 
quickly, so our boxes could become aware of last-minute schedule changes -- 
possibly even things like sporting events going into overtime, if somebody 
were to keep an eye on it and issue predictive updates. (That would open 
another can of worms, but at least the infrastructure could handle it.) I 
honestly don't know as much about BitTorrent or other P2P protocols, so I 
don't know how quickly such changes could be propagated via that method.

-- 
Rod Smith
http://www.rodsbooks.com