[mythtv-users] I could use some help with Schedules Direct and/or XMLTV
Stephen Worthington
stephen_agent at jsw.gen.nz
Mon Jun 27 05:43:23 UTC 2016
On Mon, 27 Jun 2016 03:23:01 +0100, you wrote:
>On 25/06/16 08:11, John Veness wrote:
>>
>> The file format that radiotimes.com provided (the raw 2047.dat files) was something particular to that site, so you won't be able to get the exact same format from Schedules Direct.
>>
>> As I understand it, the "rawest" data you can get out of Schedules Direct is in JSON format, but it will take a lot of interpretation and programming.
>>
>> For you, I think your best bet is to use xmltv tv_grab_sd_json to get the data you want, and work with that. I suggest asking on the xmltv-users mailing list.
>>
>> John
>
>
>There are many large holes in my knowledge and understanding of Linux, MythTV, MYSQL, XML, XMLTV and Schedules Direct.
>
>I stumbled on an actual JSON file on an unrelated web site and started some experimental coding. The radiotimes data is essentially just a variation of CSV and the JSON format is significantly more complex, but I estimate between 4 and 10 hours work would be sufficient to write the necessary ObjectPal code. That part is easy in comparison to the problem of obtaining the JSON format schedule data to process. So far the story goes like this --
>
>tv_grab_sd_json is part of XMLTV and XMLTV uses data from Schedules Direct. XMLTV is a Schedules Direct approved application. The terms and conditions of Schedules Direct expressly forbid the use of unapproved applications and the extraction of schedule data from approved applications and the archiving of data from Schedules Direct which are all things I might want to do. So it looks as if tv_grab_sd_json is a non-starter. Reading between the lines, I do get the impression that there are discrepancies between the usage of schedule data and the Schedules Direct terms and conditions. For the moment I'm looking at either -
>
>extracting EIT data from applications that use it
>
>or
>
>obtaining EIT data directly from the digital tuner mpeg stream.
>
>This option still presents many challenges.
>
>I have posted enquiries on the Schedules Direct forum and the xmltv-users mailing list but I'm not expecting any useful results.
Getting the EIT EPG data from a DVB-T or DVB-S multiplex is done using
epgsnoop:
https://launchpad.net/epgsnoop
If you have DVB-T, sometimes (as in New Zealand), the EPG data is
available in MPEG5 format. You get that using mhegepgsnoop:
https://sourceforge.net/projects/mhegepgsnooppy/
I have both working - epgsnoop on DVB-T and DVB-S and mhegepgsnoop on
DVB-T. They are both not too difficult to set up.
Here is what I use to run mhegepgsnoop (do_mhegepgsnoop.sh):
#!/bin/bash
# DVB-T multiplex frequency (kHz)
DVB_T_FREQ=594000
# Adapter number of DVB-T card.
ADAPTER=2
# Output file to store the xmltv EPG data.
TEMP_FILE=/tmp/xmltv.xml
OUTPUT_FILE=/var/www/epg/listings-mheg5.xml
PWD=`pwd`
if [ -e "$TEMP_FILE" ] && [ -f "$TEMP_FILE" ] ; then
rm "$TEMP_FILE"
fi
# Get the MHEG5 EPG data from the DVT-T multiplex.
dvbtune -f $DVB_T_FREQ -qam 64 -gi 16 -cr 3_4 -bw 8 -tm 8 -m -c
$ADAPTER 2>&1 &
sleep 1
echo Running mhegepgsnoop.py now
mhegepgsnoop.py -v -zp -d /dev/dvb/adapter${ADAPTER}/demux0 -o
$TEMP_FILE
if [ $? -ne 0 ]; then
rm $TEMP_FILE
echo Error: mhegepgsnoop failed!
else
#cp -a $TEMP_FILE /tmp/mheg5.xml
# Temporary fix for bad <channel id> values in the MHEG5 EPG data.
sed -f /usr/local/bin/mhegepgsnoop_sed_replacements --in-place
$TEMP_FILE
mv $TEMP_FILE $OUTPUT_FILE
chmod a+rw $OUTPUT_FILE
fi
kill $DVBTUNE_PID
(Watch out for lines that got wrapped by the email processing)
This creates a listings-mheg5.xml file in my /var/www/epg directory
from where the MythTV boxes on my network can download it and use it
with mythfilldatabase.
Similarly, this is what I use to get the EIT data from my SkyTV NZ
DVB-S service (sky-eit-epg.sh):
#!/bin/bash
# Get the SkyNZ EIT EPG data from a SkyNZ satellite multiplex.
# DVB-S/S2 adapter number to use.
ADAPTER=4
cd /tmp
if [ -e epgsnoop.pid ]; then
rm epgsnoop.pid
fi
epgsnoop --tune 12671 --lnb 10750 --adapter=$ADAPTER --config-dir
/usr/local/bin/.epgsnoop >listings-sky.xml
mv listings-sky.xml /var/www/epg/listings-sky.xml
chmod a+rwx /var/www/epg/listings-sky.xml
(Again, watch out for wrapped lines).
This creates listings-sky.xml available for download.
Running epgsnoop requires a .epgsnoop config directory in the
directory where epgsnoop is, or pointed to by a command line setting.
I have epgsnoop and mhegepgsnoop in /usr/local/bin, so my .epgsnoop
directory is below that. There are two config files, epgsnoop.conf
and channels.conf. Here is my epgsnoop.conf for SkyTV NZ:
[general]
processors:
StripHtml,CategoryList,MovieTitle,MovieDesc,Subtitle,SkyRatings,Widescreen,Year,Credits,CategoryDb,Imdb,SearchReplaceTitle,HD
[CategoryDb]
database: /usr/local/bin/.epgsnoop/epgsnoop.sqlite
#[Imdb]
#access_system: sql
#database_uri: mysql://mythtv:mythtv@localhost/imdb
[XMLTV]
show_icons: true
icon_url_base: http://epg.org.nz.org/icons/
[SearchReplaceTitle]
url: http://mypvr.jsw.gen.nz/json/title-replacements/+json
I have some automatic title replacement fixes done on the EPG using
the SearchReplaceTitle option. The /json/title-replacements/+json
file contains:
[
{"search": "^All New ", "replace": "", "description_match": ""},
{"search": "^New: ", "replace": "", "description_match": ""}
]
which removes the stupid "All New" and "New" that the local TV
stations like to put on the front of programme titles. Without that,
existing recording rules often fail to match the title at the start of
a new season.
Here are the first few lines of my .epgsnoop channels.conf file for
SkyTV NZ, to give you the idea how that works:
###
#
# channels.conf.sky
#
# This file is part of epgsnoop
#
# Entries are in the format show below;
#
# ***********************************
# *** The line below is important ***
# *** it is used by epgsnoop. ***
# ***********************************
# CHANNEL_ID|XMLTVID|NAME|ICON|WEBSITE|CHANNEL_NUMBER
#
# Note that the channel number currently serves no purpose.
#
###
1000|preview1000.sky.co.nz|preview.jpg||0
1001|thebox.sky.co.nz|thebox.jpg||5
1041|vibe.sky.co.nz|vibe.jpg||6
1003|uktv.sky.co.nz|uktv.jpg||7
1190|jones.sky.co.nz|jones.jpg||8
1253|thezone.sky.co.nz|thezone.jpg||9
1179|soho.sky.co.nz|soho.jpg||10
1042|comedy-central.sky.co.nz|comedy-central.jpg||11
1021|e.sky.co.nz|e.jpg||14
1039|mtv.sky.co.nz|mtv.jpg||15
1243|tlc.sky.co.nz|TLC.jpg||16
1022|living-channel.sky.co.nz|living_channel.jpg||17
1052|food-tv.sky.co.nz|food_tv.jpg||18
1029|arts-channel.sky.co.nz|arts_channel.jpg||20
1046|travel-channel.sky.co.nz|travelchannel.jpg||22
This file is the original source of the xmltvid values and icon file
names for all the channels getting EPG via epgsnoop. It has to be
manually created, or if you already have the channels from a prior EPG
source, you can create it from your database using SQL. The actual
xmltvid values you use can be anything you like, but it is best to
copy what other people are doing, if possible, or to create meaningful
values. This is the create-epgsnoop-channels-conf.sh script that I
use for my Freeview DVB-T/DVB-S and SkyTV NZ DVB-S/DVB-S2 channels
(two different EPG sources):
#!/bin/bash
# Create the epgsnoop channels.conf files from the current channel
table.
#source /etc/mythtv/mysql.txt
which xmlstarlet >/dev/null
if [ $? != 0 ] ; then
echo 'Error: xmlstarlet is not installed.'
echo 'On Ubuntu, you can use the command "sudo apt-get install
xmlstarlet" to install it.'
exit 1
fi
configxml=/etc/mythtv/config.xml
DBUserName=`xmlstarlet sel -t -v
Configuration/UPnP/MythFrontend/DefaultBackend/DBUserName $configxml`
if [ "$DBUserName" == "" ]; then
# Either config.xml does not exist, or it is in MythTV 0.26 or
greater
# format. Try 0.26+ format.
DBUserName=`xmlstarlet sel -t -v Configuration/Database/UserName
$configxml`
if [ "$DBPassword" == "" ]; then
echo "Error: Either $configxml does not exist or it is in an
unknown format."
exit 1
fi
DBPassword=`xmlstarlet sel -t -v Configuration/Database/Password
$configxml`
DBHostName=`xmlstarlet sel -t -v Configuration/Database/Host
$configxml`
DBName=`xmlstarlet sel -t -v Configuration/Database/DatabaseName
$configxml`
else
# The config.xml file is in 0.25 and before format.
DBPassword=`xmlstarlet sel -t -v
Configuration/UPnP/MythFrontend/DefaultBackend/DBPassword $configxml`
DBHostName=`xmlstarlet sel -t -v
Configuration/UPnP/MythFrontend/DefaultBackend/DBHostName $configxml`
DBName=`xmlstarlet sel -t -v
Configuration/UPnP/MythFrontend/DefaultBackend/DBName $configxml`
fi
if [ -f "/tmp/channels.conf.freeview" ]; then
rm /tmp/channels.conf.freeview
fi
if [ -f "/tmp/channels.conf.sky" ]; then
rm /tmp/channels.conf.sky
fi
mysql -vvv --show-warnings --local-infile -u $DBUserName -p$DBPassword
-h $DBHostName $DBName << EOF
use mythconverg;
select
serviceid as CHANNEL_ID,
xmltvid as XMLTVID,
icon as ICON,
replace(default_authority, 'crid://', 'http://') as WEBSITE,
channum as CHANNEL_NUMBER
into outfile '/tmp/channels.conf.freeview'
fields terminated by '|'
lines terminated by '\n'
from channel where sourceid=3 order by channum+0;
select
serviceid as CHANNEL_ID,
xmltvid as XMLTVID,
icon as ICON,
replace(default_authority, 'crid://', 'http://') as WEBSITE,
freqid as CHANNEL_NUMBER
into outfile '/tmp/channels.conf.sky'
fields terminated by '|'
lines terminated by '\n'
from channel c where sourceid=4 and freqid is not null and freqid
< 900 order by freqid+0;
EOF
cat - /tmp/channels.conf.freeview >./channels.conf.freeview << EOF
###
#
# channels.conf.freeview
#
# This file is part of epgsnoop
#
# Entries are in the format show below;
#
# ***********************************
# *** The line below is important ***
# *** it is used by epgsnoop. ***
# ***********************************
# CHANNEL_ID|XMLTVID|NAME|ICON|WEBSITE|CHANNEL_NUMBER
#
# Note that the channel number currently serves no purpose.
#
###
EOF
cat - /tmp/channels.conf.sky >./channels.conf.sky << EOF
###
#
# channels.conf.sky
#
# This file is part of epgsnoop
#
# Entries are in the format show below;
#
# ***********************************
# *** The line below is important ***
# *** it is used by epgsnoop. ***
# ***********************************
# CHANNEL_ID|XMLTVID|NAME|ICON|WEBSITE|CHANNEL_NUMBER
#
# Note that the channel number currently serves no purpose.
#
###
EOF
rm /tmp/channels.conf.freeview
rm /tmp/channels.conf.sky
(As usual, watch out for the wrapped lines).
This script puts the two resulting channels.conf files in the current
directory where it was run from. They need to be moved to the
appropriate .epgsnoop directory.
I hope that is enough to get you going with getting your EPG directly
from the channels. Various discussions of using epgsnoop and
mhegepgsnoop turn up if you google those names. Let me know if you
need more help.
More information about the mythtv-users
mailing list