[mythtv-users] Browsing UPnP "By title" + international characters = bug [PATCH]

[Michael T. Dean]

  On 11/06/2010 02:45 PM, David Kubicek wrote:
> On 6.11.2010 18:24, David Kubicek wrote:
>> On 19.6.2010 01:29, Svend Høst wrote:
>>> A recording has been made and the title from the epg contains danish
>>> letters "æ/ae,ø/oe and å/aa".
>>> If i try to locate and play the recording by upnp and browsing the
>>> upnp tree By Date, the recording is shown properly.
>>> If i choose By Title no content is streamed to my player, so the
>>> recording is there and it is properly recorded since i can see it
>>> when choosing by date.
>>>
>>>  From the log :
>>> 2010-06-19 00:30:16.119 HTTPRequest::ProcessSOAPPayload :
>>> "urn:schemas-upnp-org:service:ContentDirectory:1#Browse" :
>>> 2010-06-19 00:30:16.413 UPnpCDS::HandleBrowse ObjectID=RecTv/1,
>>> ContainerId=
>>> 2010-06-19 00:30:16.431 UPNP Browse : Searching for : RecTv  /
>>> ObjectID : RecTv/1
>>> 2010-06-19 00:30:16.449 UPnpCDSTv::IsBrowseRequestForUs - Not sure...
>>> Calling base class.
>>> 2010-06-19 00:30:16.548 HTTPRequest::SendResponse(xml/html) () :200
>>> OK ->  172.17.14.200<http://172.17.14.200>: 1
>>> 2010-06-19 00:30:31.478 HTTPRequest::ProcessSOAPPayload :
>>> "urn:schemas-upnp-org:service:ContentDirectory:1#Browse" :
>>> 2010-06-19 00:30:31.566 UPnpCDS::HandleBrowse
>>> ObjectID=RecTv/1/key=Bonder�ven retro, ContainerId=
>>> 2010-06-19 00:30:31.586 UPNP Browse : Searching for : RecTv  /
>>> ObjectID : RecTv/1/key=Bonder�ven retro
>>> 2010-06-19 00:30:31.605 UPnpCDSTv::IsBrowseRequestForUs - Not sure...
>>> Calling base class.
>>> 2010-06-19 00:30:31.699 HTTPRequest::SendResponse(xml/html) () :200
>>> OK ->  172.17.14.200<http://172.17.14.200>: 1
>> I've had the same issue - living in the Czech Republic, many shows and
>> movies have the Czech diacritics (CP: iso-8859-2 / win1250) in them.
>> Until now, I had to use "By Date", which was a bit of a pain in the
>> ass to be honest.
>>
>> I never looked into it before, just a bit of googling, and I haven't
>> found anybody with the same issue, so I thought it was a local problem
>> in my setup. Probably nobody ever reported it - except you, that is.
>> :) About an hour ago, I noticed your email by pure chance and it
>> helped me to see that my situation was the same: the bug with empty
>> folders via UPnP appeared only for shows with Czech characters -- I
>> never noticed that, thought the issue was "random".
>>
>> For example: "*By Title*" browsing displayed a folder "*C(erná zmie
>> (18)*", but when I clicked it open on PS3 or in Totem on the desktop,
>> it was *empty*. Locating the show via "*By Date*" played it without fail.
>>
>> So, I checked the source in my local MythTV 0.23-fixes repo and
>> indeed, there was a bug in handling UTF8 requests. US coders didn't
>> expect the search filters to contain UTF8 characters, so they used
>> .toLatin1() conversions all around the place.

Actually, they wrote code assuming that QUrl worked as described in the 
Qt documentation.  In Qt's API, URIs are UTF-8 encoded characters 
percent-encoded to ASCII (as they should be, per RFCs 3986 and 3987).

>>   I'm attaching a simple
>> fix, switching from .toLatin1() to .toUtf8() fixes the whole issue.
>> Applying to upstream is as simple as "search and replace".
>>
>> Some of the 11 conversions don't actually need .toUtf8(), but it
>> doesn't hurt either. Developers can apply just those parts that are
>> required.
> New ticket for this issue: http://svn.mythtv.org/trac/ticket/9188

Out of curiosity, are you using a properly-configured environment:

http://www.gossamer-threads.com/lists/mythtv/dev/439348#439348

Svend, please try the configuration in that post, too.  It should allow 
everything to work without code changes/requiring a recompile and redeploy.

Please post the output of locale /in the environment that's running 
mythbackend/.  And please try again, without your patch, with a 
properly-configured UTF-8 locale (which is the only way QUrl works as 
documented--the bug mentioned in the above)--i.e. starting mythbackend 
from a shell where you can verify the environment.  Most distros have 
start scripts that don't properly configure the environment.

I'm pretty sure the changes in your patch will have bad effects on some 
configurations.

If existing code works with a UTF-8 locale (and I'm almost positive it 
will), please say so on your ticket.  Your changes will definitely need 
to be tested in multiple different environments running with multiple 
different Latin-compatible and non-Latin-compatible encodings specified.

If you're really interested in tracking down the issue I think may exist 
in Qt (which I haven't made time to do since I'm way behind on my list 
of deliverables for MythTV), I'd be very appreciative.  I can give you 
some information that should be a good start (and will require a bit of 
code-sleuthing in QUrl, QTextCodec, QString, and some related classes).

Thanks,
Mike