[mythtv-users] How to configure XMLTV

[Andrew Sayers]

Hi list,

This is a guide for setting up XMLTV on a Myth box.  I've only used it
on my own machine, so there are probably bugs and eccentricities in this
procedure.  However, if you find this useful, please consider taking it
(with appropriate fixes) somewhere that I lack the
time/skill/energy/passwords to do, like one of the install guides out there.

_Who should read this_

This guide is intended for people who want to download TV listings from
the Internet on an already-configured MythTV box.  The scripts presented
in the guide are designed to work with any listings grabber supported by
MythTV, except SchedulesDirect and EIT.  However, they've only been
tested with the Radio Times grabber in the UK, so if it breaks in
another country, please fix it and reply with the changes needed.

This guide is very much in the spirit of Garry Parker's guide at
http://parker1.co.uk/mythtv_id.php, although it doesn't rely on a static
list of "known" TV channels, so can more easily cope when new channels
are born.

_Overview_

I'll present a 5 stage process for setting up XMLTV:

1. Tell MythTV how to grab your listings
2. Run a script that sets an XMLTV ID and icon for each channel
3. Check that the script set the correct IDs
4. Run a second script that generates the XMLTV configuration file
5. Download TV listings

_Stage 1_

In this stage, you'll tell MythTV which listings grabber to use.

1. Run the MythTV setup program.
   You can do this from a command line by typing `mythtv-setup`
   In Ubuntu, you can go to System -> Administration -> MythTV backend
setup.

2. If the setup program warns you that the backend is running, then
   1) press exit
   2) stop the backend server
      You can do this from the command line by typing
`/etc/init.d/mythtv-backend stop`
      In Ubuntu, you should be given the option to do this when running
the setup program
   3) Run the setup program again.  If you still get the same warning,
you have bigger problems than XMLTV to worry about :)

3. Go to option '3. Video sources'.

4. Select the video source that you want (you should already have one
configured, and if you have more than one, then you should already know
which one you want).

5. Change the 'Listings grabber' setting to the country you live in.
The other settings don't matter that much, but personally I prefer to
tick 'Perform EIT Scan'.  I'll discuss why later.

6. Finish this screen, then exit MythTV setup.  Whatever you're told,
you don't want to run mythfilldatabase right now.

You have now told MythTV which program to use to grab TV listings over
the Internet.

_Stage 2_

In this stage, you'll run a script that sets your initial TV listings
information.  At the time of writing, this script has only been tested
for a single user in the UK, but is designed to work for any user
anywhere.  Please reply to this message and tell the world about its
success/failure if you run it somewhere else on the planet.

1. Paste everything between (but not including) the lines "START OF
SCRIPT" and "END OF SCRIPT" into a file called 'set-listings.sh':

START OF SCRIPT
#!/bin/bash

# Bash script to update MythTV's channel list

# Set environment variables
. /etc/mythtv/mysql.txt
export mythtv_dir=/home/mythtv/.mythtv/channels

mkdir -p "$mythtv_dir"

# Lists normally have the same display name printed twice.
# This is a cheap way of removing the duplicate
exec 3> >(uniq | tee listings.$(date +%s).txt)

# Number of display names so far
n=0

# For each video source...
mysql -N "-h$DBHostName" "-u$DBUserName" "-p$DBPassword" "$DBName" \
    --exec "SELECT sourceid, xmltvgrabber FROM videosource;" | \
while read SOURCEID GRABBER
do
    # Grab the relevant list.  A typical entry looks like this:
    #  <channel id="1.setanta.com">
    #    <display-name>Channel Name</display-name>
    #    <display-name>Channel Name</display-name>
    #    <icon src="http://www.lyngsat-logo.com/.../channel.jpg" />
    #  </channel>
    "$GRABBER" --list | \
    while read LINE
    do
	# Get the channel ID
	if [[ "$LINE" =~ \<channel\ id=\"(.*)\" ]]
	then
	    xmltvid=${BASH_REMATCH[1]}
	# Get a display name
	# (there are usually two, but allow for any number)
	else if [[ "$LINE" =~ \<display-name\>(.*)\</display-name\> ]]
	then
	    names[$((n++))]=${BASH_REMATCH[1]}
	# Get (and download) the icon
	else if [[ "$LINE" =~ \<icon\ src=\"(.*/(.*))\" ]]
	then
	    icon="$mythtv_dir/${BASH_REMATCH[2]}"
	    [ -r "$icon" ] || wget -q ${BASH_REMATCH[1]} -O "$icon" >&2
	# Process this channel
	else if [[ "$LINE" =~ \</channel\> ]]
	then
	    # For each display name...
	    for ((m=0; m!=n; ++m))
	    do
		# Update every matching channel in MythTV's database
		cat <<END
UPDATE channel
   SET xmltvid="$xmltvid",
       icon="$icon"
 WHERE name="${names[$m]}"
   AND sourceid=$SOURCEID;
END
		# Tell the user
		echo $xmltvid"	"${names[$m]} >&3
	    done
	    n=0
	fi ; fi ; fi ; fi
    done | \
    mysql -u "$DBUserName" -h "$DBHostName" -"p$DBPassword" "$DBName"
done
END OF SCRIPT

2. Make this script executable.
   You can do this from the command line by typing `chmod 755
set-listings.sh`
   You can do this from your favourite file browser by right-clicking on
it and turning on the "execute" permission.

3. Run the script
   Preferably, run this from the command line by typing `./set-listings.sh`.
   Alternatively, run it from your favourite file browser by
double-clicking on it.

4. If you ran the program from the command line (or a good file
browser), you should see a long list of channel names flash up in front
of you.  This is the complete set of channels known to your grabber.
The list is also saved in a file called 'listings.<a-long-number>.txt'.
 If you've run the script more than once, the most recently created file
is the one with the highest number.

5. If the file wasn't created, or is empty, or is otherwise obviously
broken, then the script didn't run properly.  Reply to the list if you
figure out what the problem is.

You have now set the XMLTV ID for all of the channels that can be
automatically determined.

_Stage 3_

In this stage, you will check that the XMLTV IDs you set in the previous
stage are correct.

1. Find the list of channel information
   If you have set MythWeb up, you can point a web browser at
http://<your-computer>/mythweb/settings/channels, or you can go to the
MythWeb listings page, click on the keyring/spanner icon at the top of
the page, then click on "MythTV channel info".
   Alternatively, you can run the MythTV setup program as described in
stage 1.  Be warned that this stage will take you a long time this way.

2. For each channel, check that it has an appropriate XMLTV ID.
Inappropriate/missing XMLTV IDs are usually fairly obvious.  For
example, the BBC broadcasts slightly different regional variations of
its main TV channels, so if you're in the UK, you want BBC1 and BBC2 to
use your local XMLTV ID rather than the national ID.

3. For each channel without an appropriate XMLTV ID, consult the
'listings.<a-long-number>.txt' file that you created during stage 2.
Each line has an XMLTV ID, a tab, then an associated channel name.  If
you can find an appropriate channel name in the list, tell MythTV the
associated XMLTV ID.

4. For each channel, decide whether you want to grab EIT listings over
the airwaves.  This is up to you, but some things to consider are:
   a) If your channel has no XMLTV ID, this is the only way to get listings
   b) If the listings ever change at the last minute (such as an
overrunning football match), EIT tends to respond more quickly
   c) Grabbing EIT listings is time-consuming
   d) Depending on your country, EIT listings might give less (or more!)
information than XMLTV.  In the UK, it's much less.

5. If you are using MythWeb, click 'Save' at the bottom of the page.

6. If you set any channels to grab EIT listings, make sure you ticked
"Perform EIT Scan" during stage 1.  If you are using the MythTV Setup
program, you can navigate to that menu within the program.  Otherwise,
follow the steps in stage 1 to check.

You have now told MythTV how to get listings information for all of your
channels.

_Stage 4_

In this stage, you will export the settings you have just created into a
file used by XMLTV.

1. Paste everything between (but not including) the lines "START OF
SCRIPT" and "END OF SCRIPT" into a file called 'set-xmltv.sh':

START OF SCRIPT
#!/bin/sh

# Shell script to update the channel lists

. /etc/mythtv/mysql.txt
export mythtv_dir=/home/mythtv/.mythtv

mysql -N "-h$DBHostName" "-u$DBUserName" "-p$DBPassword" "$DBName" \
    --exec "SELECT DISTINCT name FROM videosource;" | \
while read FILE
do
    if ! echo -n > "$mythtv_dir/$FILE.xmltv"
    then
	echo Error: cannot write to "$mythtv_dir/$FILE.xmltv"
	echo Maybe you need to do \`sudo $0\`?
	exit 1
    fi
done && \
\
mysql -N "-h$DBHostName" "-u$DBUserName" "-p$DBPassword" "$DBName" \
	--exec "
SELECT videosource.name, concat('channel ', xmltvid)
  FROM videosource, channel
 WHERE videosource.sourceid=channel.sourceid
   AND xmltvid!=''
 GROUP BY xmltvid" | \
while read FILE LINE
do
	echo "$LINE" >> "$mythtv_dir/$FILE.xmltv"
done
END OF SCRIPT

2. Make this script executable.
   You can do this from the command line by typing `chmod 755 set-xmltv.sh`
   You can do this from your favourite file browser by right-clicking on
it and turning on the "execute" permission.

3. Run the script as root
   You can do this from the command line by typing `sudo ./set-xmltv.sh`.

4. Unlike the previous script, this program doesn't print any output
when it runs successfully.  If you really want to check whether it ran
correctly, open the file '/home/mythtv/.mythtv/<source-name>.xmltv'.
The file should contain a series of lines that look like "channel <XMLTV
ID>", one for each XMLTV ID you defined earlier.

You have now exported the XMLTV settings.

_Stage 5_

1. Run mythfilldatabase
   You can do this on the command line by running `mythfilldatabase`
   You can do this from a 'run' dialogue by typing 'mythfilldatabase'.
This is usually available in the menu system, or by pressing alt-F2.
You should run this in a terminal, so you can watch the program running.

2. Wait.  Downloading listings can take a long time, depending on how
many channels you configured XMLTV IDs for.

3. When the listings finish downloading, check that you have complete
listings for every channel you created an XMLTV ID for.  You can do this
by checking the listings in MythWeb, or by going to Manage Recordings ->
Schedule Recordings -> Programme Guide in the MythTV frontend.

Congratulations, you have now set up your MythTV listings!

_Epilogue_

If this guide didn't work for you, please reply to the list and explain
why.  Preferably include how you fixed the problem :)

If you found this guide useful, here are some ways you could pass the
favour forward:

* Write the guide into one of the MythTV installation guides online

* Extend this guide for dealing with channels being created/destroyed.
Hint: start at stage 2, and everything should be pretty much the same.

* Send a patch in to the MythTV developers that creates a button in the
'video source' menu to set XMLTV IDs in a manner similar to stage 2.

* Send a patch in to the MythTV developers that gets mythfilldatabase to
automatically generate the XMLTV file each time it runs, in a manner
similar to stage 4.

Good luck, and happy surfing!

	- Andrew Sayers