[mythtv-users] Data Direct "service"

[Bruce Markey]

Rich West wrote:
> I just have to ask.  I certainly don't mean to add fuel to the fire, and
> I will admit that, other than this thread (and similar ones to it), as
> well as some of the gossamer threads I followed, I am pretty ignorant to
> the entire agreement with DataDirect/Zap2It.  I searched around and have
> been following this entire thread, but I am still kind of unsure as to
> how the MythTV and DataDirect/Zap2it Labs relationship came about and
> what the expectations are.
> 
> Maybe the history lesson on 'how it came about' isn't really important,
> but I think there is some confusion as to what the expectations are
> (from both sides).
> 
> I'm still new to MythTV (only started with it in November).  As a
> relative newbie to MythTV, the impression that I had was that
> DataDirect/Zap2It was doing MythTV a favor.  I have no idea as to why,
> but we get guide data from them.  Now that I have read some of the
> threads (and this one), as well as some indirect comments in previous
> threads over the past few months, I realize that the download feed was
> created out of necessity on their part (to handle the ever growing load
> imposed by screen scraping), but I still don't quite understand why they
> are providing the service to MythTV users.
> 
> Please excuse me if this question is out of line or off topic..

It is part and parcel to the topic. These are my opinion
based on my understanding.

TV station broadcasts entertainment programming inter-cut with
advertisements for products from sponsors. The stations' revenue
comes from how much they can charge those sponsors. The larger
the audience, the more the sponsors are willing to pay. Therefore,
it is in the stations best interest to attract the largest audience
possible. In order for potential viewers to tune into a show,
the viewer needs to know when a show is on and what channel.
Stations can promote their shows by including promos in other shows,
having the stars do interviews and so on. However, an important
way to make viewers aware of shows they might like to watch is
to distribute their schedules as widely as possible.

GemStar, for one, whet into the business of gathering all the
schedules of all the stations and putting them together in a
magazine called "TV Guide". This is sold for a small price and
included advertisements. I do not know the business arrangements
between GemStar and the stations but all stations report in
which is in their best interest and this has been a very
profitable business for GemStar over several decades.

The Chicago Tribune (which is not located in Podunk, NY for
those who thought it was ;-) has an interest in gathering TV
listings to include as a feature of their newspapers which
are sold in many markets. Tribune Media Services is now the
business entity that gathers and distributes this information.
Their success is determined by how effectively they distribute
this information to the widest possible audience.

If GemStar and TMS where not effective in distributing TV listings,
the TV stations would need to find or create another entity to
do the job because they need their schedules to reach potential
viewers.

TMS created a web site to broaden their "reach". I don't know
the details but I have to assume that the number of people
who use the TMS web site affects their revenue and the statistics
for their site are important.

Before MythTV, there already was an open source project XMLTV
that automatically requests web pages and extracts TV listings
to be used in other application. There may have been others
that did the same thing but XMLTV seemed to have become the
most popular. They include grabbers for many web sites in many
countries around the world. MythTV used this to gather listing.
[The channel table includes a field named "xmltvid". This should
have been named "stationid" or "listingsid" but the premise was
so strong that xmltv was the only choice to ever be considered.]

Each time a user would use this program to grab the data (available
to individuals and TMS wants to distribute to as many individuals
as possible), the program would hit a hundred or more web pages
to grab all the data for all the days and channels. As the number
of XMLTV users increased, this created problems for TMS. The
increasing web traffic would cause them to spend millions of
dollars in hardware and IT costs to keep up with demand. Perhaps
more importantly, they had no good way to prove to stations or
advertisers how many individual potential viewers where getting
their information from TMS.

TMS came up with a practical solution. They decided that they
could run an application that would bundle the data that XMLTV
was crawling for and deliver it to a specific, known end user
without a single hit to their web servers. Now I can get a day
of listings in 200k rather than 200 http hits. TMS traffic from
me goes down by magnitudes and they know who I am, where I live,
what stations I do and don't grab, what applications I use and
lots of information that they can cross reference from their
surveys. They can now show exactly who their users are and,
most likely, can resell the marketing information they gather.

[BTW No one involved in the MythTV projects has the foggiest
idea how many people are using myth. The last estimate I recall
from ijr was 6. TMS knows the precise number of active registered
users who have downloaded listings in the past 24 hours to be
used with MythTV.]

As far as I know, no users had asked for this service but this
is clearly a big win for the end users too. My impression is
that TMS worked closely with XMLTV to meet deadlines to switch
over to this new method. As soon as the DataDirect version of
XMLTV was ready, the zap2it site was changed in order to break
the scraper versions of XMLTV.

The DD format is well documented and easily parsed. XMLTV simply
reformatted this data to their own but in the process, left out
a lot if good info that isn't supported in the XMLTV format
(like programid, rated sub genre, the organization of people
named in the credits, etc.).

David Shay, who has posted in this thread, did the work (not whined
that someone else has to do this for him because he pretended to
represent other angry user but ...did  the  work!) for myth to
grab and parse the data from DataDirect instead of accepting the
crippled data from the XMLTV grabber.

http://cvs.mythtv.org/trac/changeset/3529

So to answer part of your question, MythTV wasn't directly involved
when this went down. Isaac was given an access code, documentation
links, etc. and asked if developers would get involved in the beta.
However, it was XMLTV that was in the hot seat to convert by a
deadline.

Having this detailed, authoritative data for free is a great
benefit to all of us. But here's where it gets weird. People
saw this as a good thing and jumped to conclusions that they
would feel so strongly about that they would present as fact:

1) this is a free trial and they will start charging through
the nose once we're hooked

2) this is provided as an act of charity out of the goodness of
their hearts. We dare not speak of any imperfection else they
will take this generous offer away from us.

Number one eventually died off over time. The other is still
'number two' as recently as yesterday.

They want this service to run smoothly and reliably. It's what
they wanted and in their best interest. If they tried to charge
for DD, people would simply go back to scraping web pages for
free. There, I said it again but it's the truth. As much as I
don't want to go back to the bad old days, there is nothing I
can say or do that would prevent everyone from using a scraper
if that was their best alternative (picture Gene Barry on the
back of a truck in "War of the Worlds" "My instruments!").

http://www.imdb.com/title/tt0046534/quotes

I have to believe that the folks working on DataDirect want
to provide a good service and are paid well for a job well
done. They've always in the past wanted to correct problems
even if they didn't have a instant, perfect solution. They
have forums to ask for reports of problems so they can correct
them.

However they've had a problem for the past week or two and in
this case either they weren't paying attention, ignored it or
knew they had a problem and are trying to keep quiet about it.
I don't want to see their service deteriorate for any of these
reasons and I have to believe that they don't either.

--  bjm