[mythtv] wanted: developers that use mythfilldatabase
Peter Bennett
pb.mythtv at gmail.com
Sun Feb 9 20:04:59 UTC 2020
On 2/9/20 10:59 AM, Hans Dingemans wrote:
> On Sun, Feb 9, 2020 at 10:20 AM Peter Bennett <pb.mythtv at gmail.com
> <mailto:pb.mythtv at gmail.com>> wrote:
>
>
>
> On 2/8/20 6:34 PM, Hans Dingemans wrote:
>> Hi all,
>>
>> First of all big thanks to all of you for this great software; I
>> have been using Mythtv for years, very happy with it!!! However,
>> as an enthusiastic open-source developer I always seek
>> possibilities to improve and to contribute.
>>
>> Apart from my production server I now am experimenting with
>> frontend + backend on a Raspberry Pi, trying to get it so far
>> that it can play HLS streams from websites like
>> https://watchnewslive.tv .
>> A problem I encountered is that when trying to load the EPG for
>> all those stations, mythfilldatabase uses too much memory and
>> freezes up the (4Gb-version of the) Pi.
>> Turns out that the entire xml-file is kept in memory multiple
>> times. I made a patch which reduces memory usage 6 times.
>>
>> The nice thing about these offline software is you can test it
>> easily; I just backup my sql database, load a test xml file with
>> "production" mythfilldatabase, dump the database, restore it to
>> its old version, reload the same xml file with my patched
>> mythfilldatabase, dump it again, and do a compare against the dumps.
>>
>> To make sure all the different xmltv-dialects out there are
>> working, I am looking for test xml-files from allover the world;
>> the more diverse, the better.
>> Please send them to my gmail address, if it is too large, send me
>> a pm so we can see how to transfer.
>>
>> Thank you!
>>
>> Hans.
>>
>> _______________________________________________
>> mythtv-dev mailing list
>> mythtv-dev at mythtv.org <mailto:mythtv-dev at mythtv.org>
>> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
>> http://wiki.mythtv.org/Mailing_List_etiquette
>> MythTV Forums:https://forum.mythtv.org
> Did you see the solution that I use? I run 3 days at a time in a
> script. If needed you can reduce it to 1 day at a time. See link below
> https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage
>
> Peter
>
> Hi Peter,
>
> Yes I saw the solutions that you use, that post in fact showed me the
> --no-allatonce parameter. But then I found out that parameter is not
> working when you use the --xmlfile parameter (this seems undocumented).
> So then I looked into the code, because I didn't understand why
> Mythfilldatabase was using substantially more memory than the xml-file
> it was processing; turned out one of the reasons is because
> Mythfilldatabase uses QDomDocument to parse and store the XML data
> that is read; according to QDomDocument documentation this object is
> not meant to handle large XML files; it recommends to use
> QXmlStreamReader in these situations. So I made a patch that actually
> does this: https://code.mythtv.org/trac/ticket/13517. For some reason
> it reduces memory usage by a factor 6 (in my situation), where I was
> expecting "only" a factor 2. Also it reduces run-time somewhat.
>
> I hope I don't offend you if I say that I look at "chopping up" the
> epg-reading process in several days (either 1 day or 3 days chunks) as
> a workaround, and that reducing the excessive memory usage of
> Mythfilldatabase is solving the root cause. Everybody would profit of
> it, whether you are chopping the reading process by your script or by
> use -no-allatonce, or when you process your xml-files as a whole.
>
> That is why I am requesting xml files from all over the world (already
> received some good stuff!), to do excessive testing, and then ask one
> of the developers to commit my patch. I saw you committed the
> -no-allataonce parameter, would you be the designated developer for this?
>
> Best regards, Hans.
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev at mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
Yes sure I can commit it for you. However I would prefer a patch, if you
attach a diff file to the ticket when you are happy with it I will take
a look and commit if it looks good. I will also get you an xml file for
testing.
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-dev/attachments/20200209/aedb14cc/attachment.htm>
More information about the mythtv-dev
mailing list