[mythtv-users] DB Schema update 1261: broken duplicate detection with historic non-DD data

[Michael T. Dean]

On 04/20/2015 04:42 PM, Nick Morrott wrote:
> Background:
>
> UK-based setup. XMLTV/tv_grab_uk_rt listings since 2004.
>
> Over the last couple of weeks I've been responsible for upgrading a
> super-stable 'embedded' MythTV system, originally installed with 0.16
> and most recently running 0.21-fixes, to current 0.27-fixes. The
> two-step upgrade process went perfectly, despite changing Linux
> distributions at the same time.
>
> To make sure everything was working normally, I disabled listings
> updates immediately pre-upgrade and used the 14 days of in-database
> listings for recordings during the transition period.
>
> Once things were looking good, I started pulling in the regular 14
> days of new listings data after dropping the program and eitcache
> tables.
>
>
> Observation:
>
> After mythfilldatabase had successfully imported the new "clean" data
> for all sources, the Upcoming Recordings list grew quite spectacularly
> in size, with several previously-recorded programmes due to re-record
> almost immediately, and dozens more over the coming days. Wondering
> what the underlying issue(s) was I started to investigate...
>
>
> Investigation:
>
> Being aware of the changes to the season number encoding (1-9a-z)
> which would affect previous recordings of shows having 11+ seasons I
> still checked this possibility. One or two of the "new" upcoming
> recordings were as a result of this update. Changing the historic data
> in oldrecorded to match the newly-generated data in program cleared
> these few cases.
>
> I then checked some newly-selected previous recordings (with<10
> seasons) and noticed that the programids in the current program data
> and historic oldrecorded data no longer matched.
>
> There are only a couple of ways they could differ:
> - the ELFHash has changed - not the case
> - the season/episode numbering was different - not the case
> - duplicate detection had changed - not the case
> - ???
>
> Looking at the ELFHash code and noticing seriesids are not generated
> with leading zeroes (which the offending oldrecorded ids now
> contained), I started to look in the codebase and commit history and
> noticed this update:
>
> https://github.com/MythTV/mythtv/commit/257abc2b
>
> This update takes a DataDirect-specific programid formatting routine,
> and applies it to *all* data in the oldrecorded, recorded and program
> tables, regardless of source.

All that change did was take code that would check (using a setting) 
every single time mythfilldatabase was run and every single time a 
database update was run and then run those exact same updates if it had 
not already been run once and move it so that it would run once as a DB 
update--so that it was properly "documented" that it had been run--and 
never need to be checked again (in mythfilldatabase or in DB updates).

The code that changed the program IDs was put into MythTV in 0.20-fixes 
and there is no possible way for it not to have already run on your 
system if you had ever run a 0.20-fixes-or-later mythfilldatabase (and, 
since you said you were on 0.24-fixes, I'm assuming you had--unless you 
were manually poking your listings data into the database).  There was 
never any check whether the data was put in place by DataDirect or other 
listings sources since it was originally included in 0.20-fixes:

https://github.com/MythTV/mythtv/commit/30123f07

Specifically:
https://github.com/MythTV/mythtv/commit/30123f07#diff-3dff050bcb9a9eab495ef1d85162fc2c

if (usingDataDirectLabs ||
     !gContext->GetNumSetting("MythFillFixProgramIDsHasRunOnce", 0))

where if usingDataDirectLabs (the old, pre-Schedules-Direct source) is 
true, it would run
or if MythFillFixProgramIDsHasRunOnce is 0 or a non-existent setting, it 
would run.

As well as:
https://github.com/MythTV/mythtv/commit/30123f07#diff-8b944e034f25420ca3f0a3c174c5ac0e

if (!gContext->GetNumSetting("MythFillFixProgramIDsHasRunOnce", 0))
     DataDirectProcessor::FixProgramIDs();

where the code would run any time a DB upgrade was run if the 
MythFillFixProgramIDsHasRunOnce setting is 0 or a non-existent setting.

As shown at:
https://github.com/MythTV/mythtv/commit/257abc2b#diff-8b944e034f25420ca3f0a3c174c5ac0e
if the setting exists and is non-zero, the DB version is incremented and 
the DB update continues with change 1261.

So is there some way you corrupted your database and removed that 
setting?  I.e. if you don't properly restore the database--and do 
something broken, like restoring only certain tables (where doing so is 
not only broken, but out-right dangerous if the database version isn't 
correct/up-to-date) to "clean up" the database--you could end up running 
that code again, even though it was already run in 2007.  Similarly, if 
you "cleaned up" the database by removing old settings values that you 
thought you didn't need, you could end up running that code again.

Mike