[mythtv-users] Re: Bleb.org down??

Tue Apr 20 09:40:30 EDT 2004

Simon,

I'm not a Bittorrent expert, I do have a faint recollection of opening up my
firewall tough. I wouldn't say this was a major problem though. The single point
of failure of the tracker is! If we have a SPF we're back to what we had before
<sigh>.

I think we have two very distinct problems to solve:

1. Getting the EPG data.

2. Distributing the information such that it's:
   a) easy (no difficult to install software)
   b) low cost. Free (!), low usage of bandwidth, http connections
   c) quick
   d) reliable

So let's look at 1. As I understand it we already have a program to grab The
Radio Times listings. I don't know the output of the grabbers are as I'm at
work. Are they XML? If they are then I think we have a solution to 1. if they
aren't then we have to parse the data and write XML suitable for MythTV to
handle. Whether it's a single file or multiple files in multiple directories or
even a database is neither here nor there for the moment. 

2. The second stage is how we distribute the information. None of us want to be
the master source for all users even if there are only 20 of us. Single points
of failure etc come to mind. We need the system to be auto-load balance, be
resiliant and easy to use and administer. I think we need some sort of P2P
program. Never having used Kazaa or Napster I have no idea how they work but I'm
fairly sure that we could use their ideas. Perhaps Gnutella (or some other P2P
program) may be of some use to us. What we really want is peer to peer sharing
with a number of nodes acting as the master source to 'seed' the rest of the
network. 

Comments welcomed.

Rob.

Quoting Simon Kenyon <simon at koala.ie>:

> On Tuesday 20 April 2004 11:58, Rob Willett wrote:
> > Simon,
> >
> > Most of the software needed is readily available.
> >
> > 1. My understanding is that bleb.org doesn't simply use one of the
> existing
> > trawl programs. I *thought* I had read somewhere that Andrew stated this
> > wasn;t the case. I could be wrong though...
> 
> it would be good if he made his software available
> 
> >
> > 2. If the XML file is generated automatically each day then all we are
> > really doing is spreading the download across many servers. The
> > syncronisation of these servers is easy using something like wget. If the
> > XML file is generated on each request for the XML file then we have to
> sync
> > the databases. A little difficult but not that bad, simply sqldump the
> data
> > into a file, wget it and then reinsert into the mirror. (assuming we're
> > running mySQL)
> 
> that would work
> 
> >
> > 3. My first thought is that something like bittorrent
> > http://bitconjurer.org/BitTorrent/ might be the quickest and easiest way
> to
> > make this available. Admittedly bittorrent is somewhat overkill for a
> small
> > file but it's the thought that counts. <grin>. This addresses most of your
> > last few points.
> 
> i've being doing some small research and so far come up wih nothing 100% 
> suitable. bittorrent has a couple of characteristics which would lead me to 
> believe that it is not suitable:
> 
> - it uses its own port number. this makes it firewall unfriendly
> - it has a single point of failure; the tracker
> 
> >
> > If we use bittorrent then the problem becomes a little easier. I've never
> > set bittorrent up but will have a look at it later this week. (Just after
> > I've solved world hunger...)
> >
> > Rob.
> 
> it seems like there are a number of discrete problems to solve:
> 
> - define a database schema which can store xmltv data
> - define a mechanism for the exchange of (improper subsets of) the data
> - define a mechanism for the administration (addition and deletion) of 
> participating nodes with no central registry
> - define a mechanism for the discovery of servers
> - define a set of grabbers
> 
> the network (serious overkill) that i have in mind is contained in the 
> attached diagram. comments welcome (including - you're mad!)
> --
> simon
> 
> >
> > Quoting Simon Kenyon <simon at koala.ie>:
> > > On Tuesday 20 April 2004 08:01, Rob Willett wrote:
> > > > It may be that bleb.org actually constructs the XML files out of a DB
> > > > to
> > >
> > > be
> > >
> > > > returned each time, I thought that this would be inefficient since we
> > >
> > > could
> > >
> > > > generate the entire XML ech day, but I don't know how it's setup
> > > > either.
> > > >
> > > > Who needs to approach Andrew about this?
> > >
> > > a while ago i asked andrew about the software he runs
> > > i was concerned about the single point of failure issue
> > > i also wanted to spread the load and increase the range of channels
> > > supported
> > > [i have ntl cable and sky digital]
> > >
> > > he ignored the question (for whatever reason)
> > >
> > > i would like to build a network of servers to provide this facility
> > > that way it would be fault tolerant
> > >
> > > required:
> > >
> > > a set of grabbers from the various tv stations (tvlive, bbc, etc)
> > > a database to store them
> > > a web system for providing the listsings in xmltv format
> > > a mechanism for providing the list of servers
> > > a mechanism for keeping the server list current
> > > a mechanism for sharing the load
> > >
> > > i am willing and capable of helping with the design, code and hosting of
> > > such
> > >
> > > a system. i need partners to help in the design.
> > >
> > > i would suggest the following:
> > >
> > > a unix like os (i use linux but any should be acceptable - bsd, solaris,
> > > aix,
> > >
> > > etc)
> > > mysql at least, but possibly postgres for the database
> > > perl or python for the grabbers
> > > php or mod_perl for the servers
> > >
> > > ? for the server sync - there must be software in this space already!
> > >
> > > --
> > > simon
> > > _______________________________________________
> > > mythtv-users mailing list
> > > mythtv-users at mythtv.org
> > > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
> >
> > -------------------------------------------------
> > This mail sent through IMP: http://horde.org/imp/
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users at mythtv.org
> > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
> 

-------------------------------------------------
This mail sent through IMP: http://horde.org/imp/