[mythtv] wanted: developers that use mythfilldatabase

[Hans Dingemans]

On Sun, Feb 9, 2020 at 10:20 AM Peter Bennett <pb.mythtv at gmail.com> wrote:

>
>
> On 2/8/20 6:34 PM, Hans Dingemans wrote:
>
> Hi all,
>
> First of all big thanks to all of you for this great software; I have been
> using Mythtv for years, very happy with it!!! However, as an enthusiastic
> open-source developer I always seek possibilities to improve and to
> contribute.
>
> Apart from my production server I now am experimenting with frontend +
> backend on a Raspberry Pi, trying to get it so far that it can play HLS
> streams from websites like https://watchnewslive.tv .
> A problem I encountered is that when trying to load the EPG for all those
> stations, mythfilldatabase uses too much memory and freezes up the
> (4Gb-version of the) Pi.
> Turns out that the entire xml-file is kept in memory multiple times. I
> made a patch which reduces memory usage 6 times.
>
> The nice thing about these offline software is you can test it easily; I
> just backup my sql database, load a test xml file with "production"
> mythfilldatabase, dump the database, restore it to its old version, reload
> the same xml file with my patched mythfilldatabase, dump it again, and do a
> compare against the dumps.
>
> To make sure all the different xmltv-dialects out there are working, I am
> looking for test xml-files from allover the world; the more diverse, the
> better.
> Please send them to my gmail address, if it is too large, send me a pm so
> we can see how to transfer.
>
> Thank you!
>
> Hans.
>
> _______________________________________________
> mythtv-dev mailing listmythtv-dev at mythtv.orghttp://lists.mythtv.org/mailman/listinfo/mythtv-devhttp://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
>
> Did you see the solution that I use? I run 3 days at a time in a script.
> If needed you can reduce it to 1 day at a time. See link below
> https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage
>
> Peter
>
Hi Peter,

Yes I saw the solutions that you use, that post in fact showed me the
--no-allatonce parameter. But then I found out that parameter is not
working when you use the --xmlfile parameter (this seems undocumented).
So then I looked into the code, because I didn't understand why
Mythfilldatabase was using substantially more memory than the xml-file it
was processing; turned out one of the reasons is because Mythfilldatabase
uses QDomDocument to parse and store the XML data that is read; according
to QDomDocument documentation this object is not meant to handle large XML
files; it recommends to use QXmlStreamReader in these situations. So I made
a patch that actually does this: https://code.mythtv.org/trac/ticket/13517.
For some reason it reduces memory usage by a factor 6 (in my situation),
where I was expecting "only" a factor 2. Also it reduces run-time somewhat.

I hope I don't offend you if I say that I look at "chopping up" the
epg-reading process in several days (either 1 day or 3 days chunks) as a
workaround, and that reducing the excessive memory usage of
Mythfilldatabase is solving the root cause. Everybody would profit of it,
whether you are chopping the reading process by your script or by use
-no-allatonce, or when you process your xml-files as a whole.

That is why I am requesting xml files from all over the world (already
received some good stuff!), to do excessive testing, and then ask one of
the developers to commit my patch. I saw you committed the -no-allataonce
parameter, would you be the designated developer for this?

Best regards, Hans.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-dev/attachments/20200209/7677bfd4/attachment.htm>