[mythtv-commits] Ticket #11739: EIT over the air program guide title description mismatch FIXED/SOLVED

Fri Nov 20 18:54:07 UTC 2015

#11739: EIT over the air program guide title description mismatch FIXED/SOLVED
-----------------------------------+-----------------------------
 Reporter:  deadletterfile@…       |          Owner:  dekarl
     Type:  Bug Report - General   |         Status:  assigned
 Priority:  minor                  |      Milestone:  0.28
Component:  MythTV - ATSC          |        Version:  Master Head
 Severity:  medium                 |     Resolution:
 Keywords:  EIT title description  |  Ticket locked:  0
-----------------------------------+-----------------------------

Comment (by carlpny <carlpny@…>):

 I've had this problem for a while, so I spent some time looking through
 the code. I believe that there is a single bug that is responsible for
 most (probably all) title/description mismatches. Looking back through the
 comments, deadletterfile roughly identified the problem as being stale eit
 data. I believe deadletterfile is correct, and I will describe below how
 there is a particular aspect of how eit data is matched that exacerbates
 the problem with stale data.

 Background:

 The atsc standard separates the Event information (EIT) (title, start
 time, etc) from the ETT (detailed description). They are separate tables
 ("table" here not in the sense of mysql, but in the sense of broadcast
 data) and they come at different times while a channel is tuned.

 These two tables are joined via an integer "event_id" key.

 To perform this join, the EITHelper holds two mappings in memory:
  - incomplete_events: source to event_id/EIT - (title, start time, etc)
  - unmatched_etts : source to event_id/ETT - (description)

 As an EIT or ETT table is processed, these two maps are referenced and
 updated. When processing an EIT event, if there is already an ETT present
 in unmatched_etts, then the two values are joined and processed. If not,
 then the event added to incomplete_events for subsequent processing. A
 similar process is followed when handing ETT data.

 Observed bug behavior:

 In the early lifetime of a fresh event database, the events appear healthy
 and show strictly matching titles and descriptions.

 Over an unknown time period, as the EIT is updated, gradually, the EIT
 table appears to be corrupted. Although all the events appear to have the
 correct time and title, the descriptions are typically copied from some
 other show *in the same channel*.

 Others have reported that this error happens when they have dual tuners,
 although I speculate that to be a red herring. Either way, I have not
 investigated that aspect, and I have not tried parsing through the
 complicated locking behavior of EIT.

 Root Cause:

 The EITHelper's lifespan is for the duration of the mythbackend process.
 The unmatched_etts and incomplete_events maps are member variables of the
 EITHelper. However, these maps are never cleared. Their values are only
 deleted when a successful match is found.

 Here is a description of a common scenario to leading to mismatched
 title/description.

 * mythbackend runs and starts scanning for EIT at regular intervals.
 * The Foo channel is tuned.
 * The Foo channel gets an EIT event, and adds it to "incomplete_events".
 * The Foo channel gets an ETT event with description A, finds the
 corresponding EIT event, and completes the event.
 * The Foo channel gets a second ETT event with description A, and adds it
 to "unmatched_etts".
 * The channel is changed as the scan proceeds. "unmatched_etts" still
 contains A.
 * Some time later the scan again reaches the Foo channel. By this time,
 the broadcaster has recycled some of their event_ids to correspond to
 different shows.
 * The foo channel gets an ETT event with *updated* description B. However,
 there is a check inside the EITHelper that says, if an unmatched ett
 already exists for a given event ID, don't overwrite it, so the A value is
 retained. Here is the check:
 https://github.com/MythTV/mythtv/blob/fixes/0.27/mythtv/libs/libmythtv/eithelper.cpp#L209
 * The Foo channel sends an EIT event. There is a matching event_id already
 present in "unmatched_etts", so the Foo channel completes the event with
 erroneous description A, even though description A actually corresponds to
 some event reported a long time in the past.

 Proposed FIx:

 Barring some larger refactor, I am attaching a patch which does two
 things:
 * Allows unmatched_etts to be overwritten by fresh data.
 * Ignored ETTs and EITs that are unreasonably old.

 The patch includes a debug statement which is a smoking gun for the
 problem. The output below demonstrates that an old text description shared
 the same event id with a newer event. Previously, the old text description
 would have been retained and reused.

 2015-11-20 01:13:01.360345 I [29224/29509] HDHRStreamHandler
 eithelper.cpp:239 (AddETT) - EITHelper: Overwriting previously unmatched
 ett. stale: 1 major: 44 minor: 1 old ett: Paid programming.  new ett: News
 and features with focus on local news, traffic and weather.

 Testing:

 I have tested the patch on the latest 0.27 branch, and I have verified
 that the patch will at least apply cleanly in the master branch. I have
 **not** tried running this for an extended period of time, and I cannot
 say that there are no other mismatch bugs. However, the change is
 isolated, and I don't expect it to cause any regressions.

 Applying the Patch:

 Assuming you are using the 0.27 branch and git, do the following:

 * Checkout mythtv with git: https://code.mythtv.org/trac/wiki/UsingGit
 * git checkout fixes/0.27
 * git apply ~/Downloads/eit_ett_staleness_fix_20151120.patch

--
Ticket URL: <https://code.mythtv.org/trac/ticket/11739#comment:42>
MythTV <http://www.mythtv.org>
MythTV Media Center