[mythtv-users] Re: Bleb.org down??

Tue Apr 20 10:04:20 EDT 2004

Just as a note, I wanted to chime in on this thread as I'm reading it with
some interest.

I'm currently in the Caribbean and am working on something similar to
generate listings for my local cableco in the Turks and Caicos.

All of our signals come from North America, but I need to hit 5-6 cities to
get them all.  What I have done is write some scripts that call the
tv_grab_na grabber for each of the cities, and then I have only the channels
I want in each city being collected.  From there I strip off the header and
footer of each file and combine them together with a master head and footer.

Then I can do a mythfilldatabase --file 1 0 file.xml

That works great for the full pass, but when I try and do my dailies it
dies.  I'm investigating why and will report back, probably a weird XML
parsing issue somewhere, a step I missed.

The reason I mention it in this thread is once I get it sorted I intend to
place the daily files and a master 14 day file on a server in country so
others can just grab it locally, the same idea as you are thinking, so the
distribution method is of interest as well as the tribulations of getting
the file formatted.:)

Thanks

-Jeff

-----Original Message-----
From: mythtv-users-bounces at mythtv.org
[mailto:mythtv-users-bounces at mythtv.org] On Behalf Of Rob Willett
Sent: Tuesday, April 20, 2004 9:41 AM
To: Discussion about mythtv
Subject: Re: [mythtv-users] Re: Bleb.org down??

Simon,

I'm not a Bittorrent expert, I do have a faint recollection of opening up my
firewall tough. I wouldn't say this was a major problem though. The single
point
of failure of the tracker is! If we have a SPF we're back to what we had
before
<sigh>.

I think we have two very distinct problems to solve:

1. Getting the EPG data.

2. Distributing the information such that it's:
   a) easy (no difficult to install software)
   b) low cost. Free (!), low usage of bandwidth, http connections
   c) quick
   d) reliable

So let's look at 1. As I understand it we already have a program to grab The
Radio Times listings. I don't know the output of the grabbers are as I'm at
work. Are they XML? If they are then I think we have a solution to 1. if
they
aren't then we have to parse the data and write XML suitable for MythTV to
handle. Whether it's a single file or multiple files in multiple directories
or
even a database is neither here nor there for the moment. 

2. The second stage is how we distribute the information. None of us want to
be
the master source for all users even if there are only 20 of us. Single
points
of failure etc come to mind. We need the system to be auto-load balance, be
resiliant and easy to use and administer. I think we need some sort of P2P
program. Never having used Kazaa or Napster I have no idea how they work but
I'm
fairly sure that we could use their ideas. Perhaps Gnutella (or some other
P2P
program) may be of some use to us. What we really want is peer to peer
sharing
with a number of nodes acting as the master source to 'seed' the rest of the
network. 

Comments welcomed.

Rob.

Quoting Simon Kenyon <simon at koala.ie>:

> On Tuesday 20 April 2004 11:58, Rob Willett wrote:
> > Simon,
> >
> > Most of the software needed is readily available.
> >
> > 1. My understanding is that bleb.org doesn't simply use one of the
> existing
> > trawl programs. I *thought* I had read somewhere that Andrew stated this
> > wasn;t the case. I could be wrong though...
> 
> it would be good if he made his software available
> 
> >
> > 2. If the XML file is generated automatically each day then all we are
> > really doing is spreading the download across many servers. The
> > syncronisation of these servers is easy using something like wget. If
the
> > XML file is generated on each request for the XML file then we have to
> sync
> > the databases. A little difficult but not that bad, simply sqldump the
> data
> > into a file, wget it and then reinsert into the mirror. (assuming we're
> > running mySQL)
> 
> that would work
> 
> >
> > 3. My first thought is that something like bittorrent
> > http://bitconjurer.org/BitTorrent/ might be the quickest and easiest way
> to
> > make this available. Admittedly bittorrent is somewhat overkill for a
> small
> > file but it's the thought that counts. <grin>. This addresses most of
your
> > last few points.
> 
> i've being doing some small research and so far come up wih nothing 100% 
> suitable. bittorrent has a couple of characteristics which would lead me
to 
> believe that it is not suitable:
> 
> - it uses its own port number. this makes it firewall unfriendly
> - it has a single point of failure; the tracker
> 
> >
> > If we use bittorrent then the problem becomes a little easier. I've
never
> > set bittorrent up but will have a look at it later this week. (Just
after
> > I've solved world hunger...)
> >
> > Rob.
> 
> it seems like there are a number of discrete problems to solve:
> 
> - define a database schema which can store xmltv data
> - define a mechanism for the exchange of (improper subsets of) the data
> - define a mechanism for the administration (addition and deletion) of 
> participating nodes with no central registry
> - define a mechanism for the discovery of servers
> - define a set of grabbers
> 
> the network (serious overkill) that i have in mind is contained in the 
> attached diagram. comments welcome (including - you're mad!)
> --
> simon
> 
> >
> > Quoting Simon Kenyon <simon at koala.ie>:
> > > On Tuesday 20 April 2004 08:01, Rob Willett wrote:
> > > > It may be that bleb.org actually constructs the XML files out of a
DB
> > > > to
> > >
> > > be
> > >
> > > > returned each time, I thought that this would be inefficient since
we
> > >
> > > could
> > >
> > > > generate the entire XML ech day, but I don't know how it's setup
> > > > either.
> > > >
> > > > Who needs to approach Andrew about this?
> > >
> > > a while ago i asked andrew about the software he runs
> > > i was concerned about the single point of failure issue
> > > i also wanted to spread the load and increase the range of channels
> > > supported
> > > [i have ntl cable and sky digital]
> > >
> > > he ignored the question (for whatever reason)
> > >
> > > i would like to build a network of servers to provide this facility
> > > that way it would be fault tolerant
> > >
> > > required:
> > >
> > > a set of grabbers from the various tv stations (tvlive, bbc, etc)
> > > a database to store them
> > > a web system for providing the listsings in xmltv format
> > > a mechanism for providing the list of servers
> > > a mechanism for keeping the server list current
> > > a mechanism for sharing the load
> > >
> > > i am willing and capable of helping with the design, code and hosting
of
> > > such
> > >
> > > a system. i need partners to help in the design.
> > >
> > > i would suggest the following:
> > >
> > > a unix like os (i use linux but any should be acceptable - bsd,
solaris,
> > > aix,
> > >
> > > etc)
> > > mysql at least, but possibly postgres for the database
> > > perl or python for the grabbers
> > > php or mod_perl for the servers
> > >
> > > ? for the server sync - there must be software in this space already!
> > >
> > > --
> > > simon
> > > _______________________________________________
> > > mythtv-users mailing list
> > > mythtv-users at mythtv.org
> > > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
> >
> > -------------------------------------------------
> > This mail sent through IMP: http://horde.org/imp/
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users at mythtv.org
> > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
> 

-------------------------------------------------
This mail sent through IMP: http://horde.org/imp/
_______________________________________________
mythtv-users mailing list
mythtv-users at mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users