[mythtv-users] Issue with accents in titles and descriptions: Myth or Schedules Direct?

[Nick Morrott]

On 6 April 2015 at 19:28, Karl Newman <newmank1 at asme.org> wrote:
> On Mon, Apr 6, 2015 at 11:14 AM, Ian Evans <dheianevans at gmail.com> wrote:
>>
>> Noticed an issue with accents in some shows and descriptions in the
>> program grid. As I was scrolling through the listings, both on the TV and
>> through Mythweb, I saw this show title:
>>
>> Justice for All With Judge Cristina Pérez
>>
>> on Zap2it it shows as Justice for All With Judge Cristina Pérez.
>>
>> I've noticed the same é issue on French Canadian shows. Just wondering if
>> it's a Myth issue or possibly how it's being received from Schedules Direct.
>>
> I think it's a known unicode conversion issue from Schedules Direct that
> they haven't been able to track down. If I recall it happens only part of
> the time, so sometimes you'll get good characters and sometimes not. Kinda
> sucks for duplicate matching, but fortunately I think mythtv uses the
> programid/seriesid from SchedulesDirect, which is still consistent.

I've seen similar issues in the past in the upstream listings data for
the XMLTV/tv_grab_uk_rt grabber. I did a lot of work to both find and
then correct such errors whenever they were spotted in the source data
so that end-users were not affected.

In my uk_rt experience, bad upstream listings data itself was the culprit.

Instead of having the relevant 'extended ASCII' character (or higher
Unicode codepoint) correctly encoded as 2 (or more) bytes in the UTF-8
output, each of the bytes of the UTF-8 representation of the character
had been encoded into UTF-8 again. Reversing this 'double UTF-8
encoding' of the bad characters produced the intended output.

The end result of this double-encoding issue was that the UTF-8 data
contained some quite 'regular' characters (in the UK data's case,
mostly French, German and Scandinavian characters) encoded with 4 (or
sometimes 6) bytes.

When the consuming application read in the allegedly-correct 'UTF-8'
data, instead of decoding to the intended character, you got another 2
(or more) bytes which are themselves then displayed.

I think a lot of these issues arose due to the assumption that the
original source data (i.e. the data received _by_ the upstream
listings provider) was in Latin-1 before converting to UTF-8, when it
was really Windows-1252.

Cheers,
Nick