[mythtv-users] Mooting architecture for a DataDirect replacement
Rod Smith
mythtv at rodsbooks.com
Sun Jun 24 17:11:17 UTC 2007
On Sunday 24 June 2007 02:45, Peter Schachte wrote:
> Rod Smith wrote:
>
> > and/or to not apply earlier changes that would overwrite more
> > recent ones.
>
> That only works if your updates are always complete replacements, never
> just corrections. Otherwise if one correction updates the end time of a
> program and the next one changes the subtitle, but they arrive out of order
> and so you don't apply the first change, then you wind up with the wrong
> end time.
That does bring up a question that needs answering before such a system could
be formally designed: Just what form do updates take? They could be similar
to diff files (or be ACTUAL diff files, for that matter), changes to
individual fields in the file (as you imply in the above), changes to
multiple fields in the file, or complete replacement files. Each possibility
brings a different set of challenges.
Before answering this question, we'd need to know how often changes are likely
to be propagated through the system. If there's an average of one update per
lineup per week, then the overhead of providing updates as complete
replacement files would be minor (particularly if each file is small, such as
an individual show's description), and the simplification of everything that
cascades from it would make it worthwhile to do it this way. If there are an
average of ten updates per station per day, fixing minor things like typos in
program descriptions, then some other update method makes sense and the logic
to handle it would have to be correspondingly more complex.
> And there's still nothing you can do about a posting that never arrives.
*ONCE AGAIN*: There's par2, which is designed for precisely this purpose. It
provides one or more data-recovery postings that enable the recreation of
missing data, similar to the way certain RAID levels work. Actually using
this technique for this particular application would take some careful
planning (you'd need to create reasonable groupings of postings to which a
given set of par2 files apply), but it ought to work. Of course, even this
will break down if the user's NNTP server is sufficiently unreliable, but
when it gets to that point the solution is for the user to find a better NNTP
server.
> >> Someone else suggested bittorrent for this, and it actually has some
> >> properties that make it quite appropriate.
> >
> > One problem with BitTorrent is that it's strongly associated with piracy.
>
> Sadly, people often have knee-jerk reactions, without thinking much.
Yes, but we're still stuck with that. Once a technology, such as BitTorrent,
has been tarred with a negative label, it's hard to disassociate it. Since
we'll probably need the cooperation of those who don't see the issue in the
way we do, using BitTorrent will put the project at a disadvantage from a
data-acquisition point of view. This isn't fair, but it's life.
> But seriously, leaving aside non-technical issues and clueless
> FUD-slingers, a swarming distribution technology distributing signed,
> compressed differences from a trusted central site would seem to me to
> satisfy all the requirements better and more efficiently than something
> like usenet. But I agree that usenet, augmented with some redundancy and a
> fall-back web or ftp site that could be used when an article gets dropped,
> could also fit the bill. I really don't see its advantages, though.
One advantage of Usenet is that, in the modern world, files get distributed
quickly, and it's not unreasonable for clients to check their local servers
for updates very frequently. This would enable updates to be disseminated
quickly, so our boxes could become aware of last-minute schedule changes --
possibly even things like sporting events going into overtime, if somebody
were to keep an eye on it and issue predictive updates. (That would open
another can of worms, but at least the infrastructure could handle it.) I
honestly don't know as much about BitTorrent or other P2P protocols, so I
don't know how quickly such changes could be propagated via that method.
--
Rod Smith
http://www.rodsbooks.com
More information about the mythtv-users
mailing list