[mythtv] wanted: developers that use mythfilldatabase

[Peter Bennett]

On 2/9/20 10:59 AM, Hans Dingemans wrote:
> On Sun, Feb 9, 2020 at 10:20 AM Peter Bennett <pb.mythtv at gmail.com 
> <mailto:pb.mythtv at gmail.com>> wrote:
>
>
>
>     On 2/8/20 6:34 PM, Hans Dingemans wrote:
>>     Hi all,
>>
>>     First of all big thanks to all of you for this great software; I
>>     have been using Mythtv for years, very happy with it!!! However,
>>     as an enthusiastic open-source developer I always seek
>>     possibilities to improve and to contribute.
>>
>>     Apart from my production server I now am experimenting with
>>     frontend + backend on a Raspberry Pi, trying to get it so far
>>     that it can play HLS streams from websites like
>>     https://watchnewslive.tv .
>>     A problem I encountered is that when trying to load the EPG for
>>     all those stations, mythfilldatabase uses too much memory and
>>     freezes up the (4Gb-version of the) Pi.
>>     Turns out that the entire xml-file is kept in memory multiple
>>     times. I made a patch which reduces memory usage 6 times.
>>
>>     The nice thing about these offline software is you can test it
>>     easily; I just backup my sql database, load a test xml file with
>>     "production" mythfilldatabase, dump the database, restore it to
>>     its old version, reload the same xml file with my patched
>>     mythfilldatabase, dump it again, and do a compare against the dumps.
>>
>>     To make sure all the different xmltv-dialects out there are
>>     working, I am looking for test xml-files from allover the world;
>>     the more diverse, the better.
>>     Please send them to my gmail address, if it is too large, send me
>>     a pm so we can see how to transfer.
>>
>>     Thank you!
>>
>>     Hans.
>>
>>     _______________________________________________
>>     mythtv-dev mailing list
>>     mythtv-dev at mythtv.org  <mailto:mythtv-dev at mythtv.org>
>>     http://lists.mythtv.org/mailman/listinfo/mythtv-dev
>>     http://wiki.mythtv.org/Mailing_List_etiquette
>>     MythTV Forums:https://forum.mythtv.org
>     Did you see the solution that I use? I run 3 days at a time in a
>     script. If needed you can reduce it to 1 day at a time. See link below
>     https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage
>
>     Peter
>
> Hi Peter,
>
> Yes I saw the solutions that you use, that post in fact showed me the 
> --no-allatonce parameter. But then I found out that parameter is not 
> working when you use the --xmlfile parameter (this seems undocumented).
> So then I looked into the code, because I didn't understand why 
> Mythfilldatabase was using substantially more memory than the xml-file 
> it was processing; turned out one of the reasons is because 
> Mythfilldatabase uses QDomDocument to parse and store the XML data 
> that is read; according to QDomDocument documentation this object is 
> not meant to handle large XML files; it recommends to use 
> QXmlStreamReader in these situations. So I made a patch that actually 
> does this: https://code.mythtv.org/trac/ticket/13517. For some reason 
> it reduces memory usage by a factor 6 (in my situation), where I was 
> expecting "only" a factor 2. Also it reduces run-time somewhat.
>
> I hope I don't offend you if I say that I look at "chopping up" the 
> epg-reading process in several days (either 1 day or 3 days chunks) as 
> a workaround, and that reducing the excessive memory usage of 
> Mythfilldatabase is solving the root cause. Everybody would profit of 
> it, whether you are chopping the reading process by your script or by 
> use -no-allatonce, or when you process your xml-files as a whole.
>
> That is why I am requesting xml files from all over the world (already 
> received some good stuff!), to do excessive testing, and then ask one 
> of the developers to commit my patch. I saw you committed the 
> -no-allataonce parameter, would you be the designated developer for this?
>
> Best regards, Hans.
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev at mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org


Yes sure I can commit it for you. However I would prefer a patch, if you 
attach a diff file to the ticket when you are happy with it I will take 
a look and commit if it looks good. I will also get you an xml file for 
testing.

Peter

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-dev/attachments/20200209/aedb14cc/attachment.htm>