[mythtv-users] Fwd: Fwd: Re: channel4 hd has no eit - cross-eit

Stephen Worthington stephen_agent at jsw.gen.nz
Mon Sep 13 09:58:32 UTC 2021


On Sun, 12 Sep 2021 22:16:46 +0100, you wrote:

>On Thu, 9 Sept 2021 at 18:29, John <jksjdevelop at gmail.com> wrote:
>
>>
>> Ok I have done some testing.
>>
>> The EPG is certainly being stored in the database as UTC so the News
>> which is shown at 19:00 BST is stored as starting at 18:00.
>>
>> The old version of cross-eit with date = dateobj +
>> datetime.timedelta(seconds=time.altzone) writes the correct times to the
>> database during BST.
>>
>> JSW version with start="20210915190000 +0000" writes incorrect times to
>> the database.
>>
>> Nemo version with start="20210915180000 +0000" writes correct times to
>> the database during BST.
>>
>
>
>I am still getting off-by-one-hour failures, though only for SOME channel 4
>HD recordings, not all, and this is all very confusing. I have also been
>doing some debugging but I can't figure out where this extra hour
>difference comes from. Here is what I've got to so far.
>
>Ground truth: Karate Kid II was broadcast on Channel 4 HD today
>(2021-09-12) starting at 1350 UTC+1.
>Indeed the channel 4 website says, at
>https://www.channel4.com/programmes/the-karate-kid-part-ii , "Sun 12 Sep,
>1.50pm"
>
>It appeared in my Myth schedule (using the old cross-eit at
>https://pastebin.com/d51VUASq ) as starting at 1450, hence still with the
>error. The recording scheduled from the myth GUI would have missed the
>first hour.
>
>The above "old" cross-eit produced
>
><programme channel="hd.channel4.com" start="20210912135000" stop=
>"20210912160000"><title>The Karate Kid Part II</title>
>
>(start time correct IFF you assume UTC+1; incorrect in UTC+0)
>
>
>The JSW 2.0 cross-eit produced
>
>  <programme channel="hd.channel4.com" start="20210912135000" stop=
>"20210912160000">
>
>    <title>The Karate Kid Part II</title>
>
>(start time correct IFF you assume UTC+1; incorrect in UTC+0)
>
>
>The JSW 2.1 cross-eit produced
>
>
>  <programme channel="hd.channel4.com" start="20210912135000 +0000" stop=
>"20210912160000 +0000">
>
>    <title>The Karate Kid Part II</title>
>
>(start time in UTC+0 that's definitely incorrect: if anything it ought to
>be 1250)
>
>
>
>Last night, a cron job ran these two commands:
>
>
>/home/tv/bin/cross-eit.py -o /tmp/export.xml 19211,"hd.channel4.com"
>
>/usr/bin/mythfilldatabase --refresh-all --verbose xmltv --file --sourceid 1
>--xmlfile /tmp/export.xml
>
>
>and the first of them produced this file (still untouched)...
>
>
>-rw-rw-r-- 1 tv tv 216039 Sep 12 04:30 /tmp/export.xml
>
>and the following database content:
>
>
>mysql> select p.chanid, c.name, p.starttime, p.endtime, p.title from
>program as p, channel as c where p.chanid=c.chanid and p.title
>like("%karate kid part ii");
>
>+--------+--------------+---------------------+---------------------+------------------------+
>
>| chanid | name         | starttime           | endtime             | title
>                |
>
>+--------+--------------+---------------------+---------------------+------------------------+
>
>|  18320 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  19211 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  19212 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  19213 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  19214 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  19216 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  18311 | Channel 4+1  | 2021-09-12 13:50:00 | 2021-09-12 16:00:00 | The
>Karate Kid Part II |
>
>|  18312 | Channel 4+1  | 2021-09-12 13:50:00 | 2021-09-12 16:00:00 | The
>Karate Kid Part II |
>
>|  18314 | Channel 4+1  | 2021-09-12 13:50:00 | 2021-09-12 16:00:00 | The
>Karate Kid Part II |
>
>|  18316 | Channel 4+1  | 2021-09-12 13:50:00 | 2021-09-12 16:00:00 | The
>Karate Kid Part II |
>
>|  31200 | Channel 4 HD | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>+--------+--------------+---------------------+---------------------+------------------------+
>
>11 rows in set (0.29 sec)
>
>
>Here the start time on channels 19211 and 31200 is 1250 rather than 1350,
>but if it's true that the database always contains UTC+0 then that's fine,
>that would be equivalent to 1350 UTC+1, hence still the correct time. Yet
>the GUI showed a start time of 1450. Why??
>
>
>
>Further experimentation: I then imported the xml produced by JSW 2.1 (the
>one with a start time of 1350 UTC+0, which is INCORRECT) with the following
>command:
>
>
> /usr/bin/mythfilldatabase --refresh-all --verbose xmltv --file --sourceid
>1 --xmlfile export-jsw21.xml
>
>
>and, after that, the same query resulted in
>
>
>mysql> select p.chanid, c.name, p.starttime, p.endtime, p.title from
>program as p, channel as c where p.chanid=c.chanid and p.title
>like("%karate kid part ii");
>
>+--------+--------------+---------------------+---------------------+------------------------+
>
>| chanid | name         | starttime           | endtime             | title
>                |
>
>+--------+--------------+---------------------+---------------------+------------------------+
>
>|  31200 | Channel 4 HD | 2021-09-12 13:50:00 | 2021-09-12 16:00:00 | The
>Karate Kid Part II |
>
>|  18320 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  19211 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  19212 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  19213 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  19214 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  19216 | Channel 4    | 2021-09-12 12:50:00 | 2021-09-12 15:00:00 | The
>Karate Kid Part II |
>
>|  18311 | Channel 4+1  | 2021-09-12 13:50:00 | 2021-09-12 16:00:00 | The
>Karate Kid Part II |
>
>|  18312 | Channel 4+1  | 2021-09-12 13:50:00 | 2021-09-12 16:00:00 | The
>Karate Kid Part II |
>
>|  18314 | Channel 4+1  | 2021-09-12 13:50:00 | 2021-09-12 16:00:00 | The
>Karate Kid Part II |
>
>|  18316 | Channel 4+1  | 2021-09-12 13:50:00 | 2021-09-12 16:00:00 | The
>Karate Kid Part II |
>
>+--------+--------------+---------------------+---------------------+------------------------+
>
>11 rows in set (0.29 sec)
>
>
>You will note that now 19211 stays at 1250 (correct in UTC+0) but 31200 has
>gone to 1350 (incorrect in UTC+0, although unsurprising as that's what the
>xml said). Now what does the GUI say? The program guide still says the
>start time is 1450 local time (incorrect).
>
>
>My unanswered questions:
>
>
>1) why, if the imported XML says 1350 (not qualified with a time zone),
>mythfilldatabase translates that to 1250 UTC+0?
>
>(Educated guess: because myth assumes that an unqualified date is in local
>time, which would make a lot of sense. But then, if myth is smart enough to
>translate from local time to UTC, why do we ever have off-by-one errors? By
>this logic, these should never occur.)
>
>
>2) why does cross-eit JSW 2.1 output 1350 UTC+0 into the XML if the EIT
>stored in the database (already in UTC+0) says 1250? Shouldn't cross-eit
>just repeat 1250 and just add UTC+0?

Yes, that is what it does.  It just takes the UTC timestamp from the
database and adds the string +0000 to say that the timestamp is in
UTC.  If there is a problem with the data, the problem must be already
present in the data in the program table.

>3) what's the proper way to fix all this? I can add and subtract 1 hour as
>well as anyone else, but by now it's totally unclear under which
>circumstances it should be done.

You need to check the data in the database before using cross-eit.py
to export it.  It looks to me that the data in the database is the
problem, not cross-eit.py.

>4) is there a way to see (and possibly even log) what the on-air EIT
>originally said, before being imported in the database as UTC+0? Did the
>EIT mention a time zone or was it in local time?

If you have a Windows machine with access to a tuner (eg network
tuner), you can fetch the EIT data outside MythTV using EPG Collector:

https://sourceforge.net/projects/epgcollector/

You will need to ensure that MythTV is not trying to use the tuner
when EPG Collector is using it.  You need to tell MythTV to release
the tuner between recordings and to not use it for EIT.

To get EIT data from tuners in Linux, dvbsnoop can do that.  I use
epgsnoop to run dvbsnoop for doing that:

https://github.com/hadleyrich/epgsnoop

I have never used epgsnoop with my DVB-T(2) tuners, as in New Zealand
our EPG on DVB-T is in MHEG5 format.  But it should work.  You do have
to set up a channels.conf file with all the channel data.  You can
create that from the MythTV channel table with some SQL:

sudo mysql mythconverg
select chanid,xmltvid,icon,'',channum from channel where sourceid=1
order by chanid into outfile '/tmp/channels.conf' fields terminated by
'|' lines terminated by '\n';
quit

From my database, that creates this in channels.conf:

1001|tv1.freeviewnz.tv|tvnz_tv_1.png||1
1002|tv2.freeviewnz.tv|tvnz_tv_2.png||2
1003|tv3.freeviewnz.tv|THREE_1280x1280.png||3
1004|bravo.freeviewnz.tv|bravo.jpg||4
1005|maori-tv.freeviewnz.tv|maori_tv.jpg||5
1006|tv1-plus1.freeviewnz.tv|tvnz1p1.jpg||6
1007|tv2-plus1.freeviewnz.tv|tvnz2p1.jpg||7
1008|tv3-plus1.freeviewnz.tv|threep1.jpg||8
1009|bravo-plus1.freeviewnz.tv|bravop1.jpg||9
1010|prime.freeviewnz.tv|prime.jpg||10
1011|theedgetv.freeviewnz.tv|theedgetv.png||11
1012|choice.freeviewnz.tv|ChoiceTV_logo.png||12
1013|duke.freeviewnz.tv|tvnz_tv_duke.png||13
1014|breezetv.freeviewnz.tv|breezetv.png||14
1015|te-reo.freeviewnz.tv|tereo.jpg||15
1016|aljazeera.freeviewnz.tv|al_jazeera.jpg||16
1017|hgtv.freeviewnz.tv|hgtv.jpg||17
1018|duke-plus1.freeviewnz.tv|tvnz_tv_duke_plus_1.png||18
1020|tvsn-shopping-channel.freeviewnz.tv|TVSN.jpg||20
1021|shine.freeviewnz.tv|shinetv.jpg||21
1022|parliament.freeviewnz.tv|parliament.jpg||25
1024|te-reo.freeviewnz.tv|tereo.jpg||24
1026|firstlight.freeviewnz.tv|Firstlight.jpg||26
1027|hope-channel.freeviewnz.tv|hope_channel.jpg||27
1028|chinese-tv8.freeviewnz.tv|ctv8.gif||28
1029|tv29.freeviewnz.tv|tv29.jpg||29
1033|c33.freeviewnz.tv|C33.jpg||33
1036|apna.freeviewnz.tv|Apna.jpg||36
1050|rnz-national.freeviewnz.tv|radio_nz_national.jpg||50
1051|rnz-concert.freeviewnz.tv|radio_nz_concert.jpg||51
1071|basefm.freeviewnz.tv|basefm.jpg||71
1107|tvnz7.freeviewnz.tv|tvnz7.jpg||107
1114|tvnz-games-extra.freeviewnz.tv|tvnz_tv_games_extra.png||114
1200|kordiatv.freeviewnz.tv|KordiaTV.png||200

You need to then copy the headers from the channel.conf examples that
come with epgsnoop - add them above the data created by the SQL query.

[Update]

I have now tried epgsnoop with DVB-T and it does not work - it is set
up for DVB-S(2) only.  Manual use of dvbtune and dvbsnoop would work,
but it would be a pain and the output format would not be XMLTV.  And
epgsnoop is Python 2.  So I am working on a Python 3 version of
epgsnoop that will support DVB-T as well.  I may not be able to test
it properly here in New Zealand as I am not sure if we now have EIT
EPG as well as MHEG5.


More information about the mythtv-users mailing list