[mythtv-users] Duplicate matching methods - make program ID an explicit choice
Michael T. Dean
mtdean at thirdcontact.com
Thu May 19 16:13:01 UTC 2016
On 05/19/2016 10:24 AM, John Veness wrote:
> On 19/05/2016 15:04, Michael T. Dean wrote:
>> On 05/19/2016 09:50 AM, Michael T. Dean wrote:
>>> It should not be an option as the program ID is the definitive
>>> identifier of a program. If it's available, it should be used.
>>>
>>> What you really want is to expand the definition of program ID to
>>> recognize its source dependence. So, the most straightforward
>>> approach would be to include some source identifier in the program
>>> ID, something like <sourceid>-<programid>. However, the source
>>> identifier could not be anything like a MythTV Video Source sourceid
>>> because that can change even when the actual source is the same
>>> (i.e. after a Delete all Video Sources). It could be the Video
>>> Source name, but that would require that the user never change that
>>> name (and know its additional importance). It almost makes sense for
>>> it to be the Video Source xmltvgrabber (for XMLTV or Schedules
>>> Direct), except in some areas the grabber name is changed
>>> frequently, though it ultimately gets its data from the same
>>> upstream source. Ideally, it would be some value taken from the data
>>> provider itself, though I don't know my XMLTV/SD/EIT data well
>>> enough to suggest any particular value to use for a listings source
>>> ID that would be specific to a given program ID provider.
>>>
>>> Any patch making this change would need to appropriately update
>>> program, record, recorded, oldrecorded, and recordedprogram data.
>>
>> Oh, and then the matching should be done using program ID where both
>> episode have a program ID and the program ID source is the same for
>> both, else it falls back to the user-specified duplicate matching
>> algorithm.
>>
>> And, really, the duplicate matching itself would be easier with the
>> listings provider ID (program ID source's ID) in a separate column,
>> though the patch would be much more difficult (as it would require
>> modifying program data in the protocol and such).
> Thanks Mike. I'm glad you added the paragraph "the matching should be
> done using program ID where both episode have a program ID and the
> program ID source is the same for both, else it falls back to the
> user-specified duplicate matching algorithm" as otherwise, with your
> first response, I didn't see how that would fix the problem when one
> changes listings provider, which at the moment will cause all previous
> episodes to re-record due to all new listings having different format IDs.
>
> It would be great if there was a "universal" program ID that would
> give a particular episode of a particular series the same ID among all
> listings providers, but I don't think this will ever happen.
>
> My problem is particularly with the user interface, where it looks
> like we can control how duplicate matching works (and a few years ago,
> indeed we could)
MythTV has always used program ID if it was there and ignore the
specified duplicate matching method. What you're likely thinking of is
the fact you used to think you had control because users of XMLTV
grabbers used to not get program IDs. Now they do (they have for a
long, long time, now, but you probably remember the before). For the
XMLTV parser, the title of a show is used to generate a series ID that's
the ELF hash of the title. If a program's XMLTV <episode-num> element's
system attribute is "dd_progid", the episode_num is used as the
programid (and this should remain this way). This provides a means for
the provider to say, "this value should be used as the definitive
episode identifier". Otherwise, if the <episode-num> system attribute
is "xmltv_ns" and if the XMLTV listings contain both a season and
episode number and if the season is 35 or less, a program ID is
generated by concatenating a category type identifier (MV, EP, SP, or
SH), the series ID, the episode number, a single-digit season number (in
base 36--1-Z, where Z has a decimal value of 35), and, if applicable,
the partnumber and parttotal. This seems like it would be a good
"universal" program ID, but it's not--because the season and episode
numbers aren't universally consistent (nor defined--reference Firefly
and it's out-of-order airing, for one).
In fact, IMHO, autogenerating program IDs is just plain wrong. If you
want a quick fix, I'd rather see a patch that removes the program ID
generation from the xmltvparser (
https://github.com/MythTV/mythtv/blob/master/mythtv/programs/mythfilldatabase/xmltvparser.cpp#L551
). Then, the motivated user could even add in a duplicate matching
mechanism that compares season and episode (and part/parttotal) number
(after adding season and episode columns to program data and updating
the code to insert them appropriately at listings retrieval) since
that's really all the autogenerated program IDs were comparing.*** Now,
if you're switching to Schedules Direct (or some other provider that
provides a program ID encoded using the dd_progid system, this won't
help to fix duplicate matching for previous episodes--but if nothing
else, it should convince you that there's no harm removing the broken
program ID data from your database.
> with the subtitle, description, subtitle+description stuff, but in
> most cases we cannot. I don't like UIs that lie to me! At the very
> least, some explanatory text could be added to say that these options
> only take effect when program IDs are missing. Even better if there
> was an advanced tickbox to ignore program IDs when doing duplicate
> matching if we wish to do so.
>
> It has been suggested that one possible fix of the re-record problem
> when changing listing providers is to blank the program ID from all
> oldrecorded entries, which would force duplicate matching to match on
> subtitle etc. I'm happy to do that, even though I dislike throwing
> away information, but others might not be happy with direct database
> manipulation.
>
> Which is why I suggested that, rather than blanking old program IDs,
> it would be better to add an option to ignore old program IDs when
> duplicate matching, which seems a relatively easy fix. If this option
> was added, I'm not sure if that would be best added to individual
> recording rules, which would mean a lot of editing recording rules for
> people but which would give maximum flexibility, or as a global
> option. Personally, I'm slightly preferring the option available in
> individual recording rules because at the moment, even disregarding
> changing listings providers, I have a problem with some series not
> correctly matching duplicates more than others.
The problem is that requires the user to know too much about the
internal workings of MythTV. It should just do the right thing--meaning
it should always use program IDs (since they're defined to be unique
identifiers of episodes) if they exist and--ideally, after someone who
actually cares/switches listings providers or has multiple listings
providers with differing program ID writes a patch to make program ID
listings-source-specific--come from the same listings provider. A quick
and messy fix isn't the right solution here--it will just prevent anyone
from ever doing things right and we already have plenty of old "good
enough for now" garbage in MythTV that still needs to be fixed. If it's
going to be messy, it's better to keep the mess out of MythTV and let
the users mess with the data in their database and clear out their old
program IDs.
So, basically, your concerns over program ID result solely from your
experience with a) multiple providers with different program ID values
(so we need listings-source awareness in the program ID usage) and b)
having program IDs where you shouldn't--all of those XMLTV-parser
autogenerated garbage values.
Mike
*** And, FWIW, there are good reasons to add season and episode to
program, so doing this would have other benefits, too.
More information about the mythtv-users
mailing list