[mythtv] wanted: developers that use mythfilldatabase

[Hans Dingemans]

Op zo 9 feb. 2020 15:05 schreef Peter Bennett <pb.mythtv at gmail.com>:

>
>
> On 2/9/20 10:59 AM, Hans Dingemans wrote:
>
> On Sun, Feb 9, 2020 at 10:20 AM Peter Bennett <pb.mythtv at gmail.com> wrote:
>
>>
>>
>> On 2/8/20 6:34 PM, Hans Dingemans wrote:
>>
>> Hi all,
>>
>> First of all big thanks to all of you for this great software; I have
>> been using Mythtv for years, very happy with it!!! However, as an
>> enthusiastic open-source developer I always seek possibilities to improve
>> and to contribute.
>>
>> Apart from my production server I now am experimenting with frontend +
>> backend on a Raspberry Pi, trying to get it so far that it can play HLS
>> streams from websites like https://watchnewslive.tv .
>> A problem I encountered is that when trying to load the EPG for all those
>> stations, mythfilldatabase uses too much memory and freezes up the
>> (4Gb-version of the) Pi.
>> Turns out that the entire xml-file is kept in memory multiple times. I
>> made a patch which reduces memory usage 6 times.
>>
>> The nice thing about these offline software is you can test it easily; I
>> just backup my sql database, load a test xml file with "production"
>> mythfilldatabase, dump the database, restore it to its old version, reload
>> the same xml file with my patched mythfilldatabase, dump it again, and do a
>> compare against the dumps.
>>
>> To make sure all the different xmltv-dialects out there are working, I am
>> looking for test xml-files from allover the world; the more diverse, the
>> better.
>> Please send them to my gmail address, if it is too large, send me a pm so
>> we can see how to transfer.
>>
>> Thank you!
>>
>> Hans.
>>
>> _______________________________________________
>> mythtv-dev mailing listmythtv-dev at mythtv.orghttp://lists.mythtv.org/mailman/listinfo/mythtv-devhttp://wiki.mythtv.org/Mailing_List_etiquette
>> MythTV Forums: https://forum.mythtv.org
>>
>> Did you see the solution that I use? I run 3 days at a time in a script.
>> If needed you can reduce it to 1 day at a time. See link below
>> https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage
>>
>> Peter
>>
> Hi Peter,
>
> Yes I saw the solutions that you use, that post in fact showed me the
> --no-allatonce parameter. But then I found out that parameter is not
> working when you use the --xmlfile parameter (this seems undocumented).
> So then I looked into the code, because I didn't understand why
> Mythfilldatabase was using substantially more memory than the xml-file it
> was processing; turned out one of the reasons is because Mythfilldatabase
> uses QDomDocument to parse and store the XML data that is read; according
> to QDomDocument documentation this object is not meant to handle large XML
> files; it recommends to use QXmlStreamReader in these situations. So I made
> a patch that actually does this: https://code.mythtv.org/trac/ticket/13517.
> For some reason it reduces memory usage by a factor 6 (in my situation),
> where I was expecting "only" a factor 2. Also it reduces run-time somewhat.
>
> I hope I don't offend you if I say that I look at "chopping up" the
> epg-reading process in several days (either 1 day or 3 days chunks) as a
> workaround, and that reducing the excessive memory usage of
> Mythfilldatabase is solving the root cause. Everybody would profit of it,
> whether you are chopping the reading process by your script or by use
> -no-allatonce, or when you process your xml-files as a whole.
>
> That is why I am requesting xml files from all over the world (already
> received some good stuff!), to do excessive testing, and then ask one of
> the developers to commit my patch. I saw you committed the -no-allataonce
> parameter, would you be the designated developer for this?
>
> Best regards, Hans.
>
> _______________________________________________
> mythtv-dev mailing listmythtv-dev at mythtv.orghttp://lists.mythtv.org/mailman/listinfo/mythtv-devhttp://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
>
>
>
> Yes sure I can commit it for you. However I would prefer a patch, if you
> attach a diff file to the ticket when you are happy with it I will take a
> look and commit if it looks good. I will also get you an xml file for
> testing.
> Peter
>

Great!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-dev/attachments/20200209/d2b3cd72/attachment.htm>