<div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Feb 9, 2020 at 10:20 AM Peter Bennett <<a href="mailto:pb.mythtv@gmail.com">pb.mythtv@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<br>
<br>
<div>On 2/8/20 6:34 PM, Hans Dingemans
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi all,</div>
<div><br>
</div>
<div>First of all big thanks to all of you for this great
software; I have been using Mythtv for years, very happy with
it!!! However, as an enthusiastic open-source developer I
always seek possibilities to improve and to contribute.</div>
<div><br>
</div>
<div>Apart from my production server I now am experimenting with
frontend + backend on a Raspberry Pi, trying to get it so far
that it can play HLS streams from websites like <a href="https://watchnewslive.tv" target="_blank">https://watchnewslive.tv</a>
.</div>
<div>A problem I encountered is that when trying to load the EPG
for all those stations, mythfilldatabase uses too much memory
and freezes up the (4Gb-version of the) Pi.</div>
<div>Turns out that the entire xml-file is kept in memory
multiple times. I made a patch which reduces memory usage 6
times.</div>
<div><br>
</div>
<div>The nice thing about these offline software is you can test
it easily; I just backup my sql database, load a test xml file
with "production" mythfilldatabase, dump the database, restore
it to its old version, reload the same xml file with my
patched mythfilldatabase, dump it again, and do a compare
against the dumps.</div>
<div><br>
</div>
<div>To make sure all the different xmltv-dialects out there are
working, I am looking for test xml-files from allover the
world; the more diverse, the better.</div>
<div>Please send them to my gmail address, if it is too large,
send me a pm so we can see how to transfer.</div>
<div><br>
</div>
<div>Thank you!</div>
<div><br>
</div>
<div>Hans.<br>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
mythtv-dev mailing list
<a href="mailto:mythtv-dev@mythtv.org" target="_blank">mythtv-dev@mythtv.org</a>
<a href="http://lists.mythtv.org/mailman/listinfo/mythtv-dev" target="_blank">http://lists.mythtv.org/mailman/listinfo/mythtv-dev</a>
<a href="http://wiki.mythtv.org/Mailing_List_etiquette" target="_blank">http://wiki.mythtv.org/Mailing_List_etiquette</a>
MythTV Forums: <a href="https://forum.mythtv.org" target="_blank">https://forum.mythtv.org</a>
</pre>
</blockquote>
Did you see the solution that I use? I run 3 days at a time in a
script. If needed you can reduce it to 1 day at a time. See link
below<br>
<a href="https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage" target="_blank">https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage</a><br>
<br>
Peter<br></div></blockquote><div>Hi Peter,</div><div><br></div><div>Yes I saw the solutions that you use, that post in fact showed me the --no-allatonce parameter. But then I found out that parameter is not working when you use the --xmlfile parameter (this seems undocumented).</div><div>So then I looked into the code, because I didn't understand why Mythfilldatabase was using substantially more memory than the xml-file it was processing; turned out one of the reasons is because <span class="gmail-searchword0">Mythfilldatabase</span> uses QDomDocument to
parse and store the XML data that is read; according to QDomDocument
documentation this object is not meant to handle large XML files;
it recommends to use QXmlStreamReader in these situations. So I made a patch that actually does this: <a href="https://code.mythtv.org/trac/ticket/13517">https://code.mythtv.org/trac/ticket/13517</a>. For some reason it reduces memory usage by a factor 6 (in my situation), where I was expecting "only" a factor 2. Also it reduces run-time somewhat.<br></div><div><br></div><div>I hope I don't offend you if I say that I look at "chopping up" the epg-reading process in several days (either 1 day or 3 days chunks) as a workaround, and that reducing the excessive memory usage of Mythfilldatabase is solving the root cause. Everybody would profit of it, whether you are chopping the reading process by your script or by use -no-allatonce, or when you process your xml-files as a whole.<br></div><div><br></div><div>That is why I am requesting xml files from all over the world (already received some good stuff!), to do excessive testing, and then ask one of the developers to commit my patch. I saw you committed the -no-allataonce parameter, would you be the designated developer for this?</div><div><br></div><div>Best regards, Hans.<br></div></div></div>