[mythtv] wanted: developers that use mythfilldatabase

[Mike Bibbings]

On 09/02/2020 10:47, Hans Dingemans wrote:
>
>     On 08/02/2020 23:34, Hans Dingemans wrote:
>     > Hi all,
>     >
>     > First of all big thanks to all of you for this great software; I
>     have
>     > been using Mythtv for years, very happy with it!!! However, as an
>     > enthusiastic open-source developer I always seek possibilities to
>     > improve and to contribute.
>     >
>     > Apart from my production server I now am experimenting with
>     frontend +
>     > backend on a Raspberry Pi, trying to get it so far that it can play
>     > HLS streams from websites like https://watchnewslive.tv .
>     > A problem I encountered is that when trying to load the EPG for all
>     > those stations, mythfilldatabase uses too much memory and
>     freezes up
>     > the (4Gb-version of the) Pi.
>     > Turns out that the entire xml-file is kept in memory multiple
>     times. I
>     > made a patch which reduces memory usage 6 times.
>     >
>     > The nice thing about these offline software is you can test it
>     easily;
>     > I just backup my sql database, load a test xml file with
>     "production"
>     > mythfilldatabase, dump the database, restore it to its old version,
>     > reload the same xml file with my patched mythfilldatabase, dump it
>     > again, and do a compare against the dumps.
>     >
>     > To make sure all the different xmltv-dialects out there are
>     working, I
>     > am looking for test xml-files from allover the world; the more
>     > diverse, the better.
>     > Please send them to my gmail address, if it is too large, send
>     me a pm
>     > so we can see how to transfer.
>     >
>     > Thank you!
>     >
>     > Hans.
>
>
>     I did some testing on a Pi4 (4GB) using external xmltv data from
>     Schedules Direct tvgrab_zz_sdjson a while ago see
>     https://forum.mythtv.org/viewtopic.php?f=46&t=3232&sid=8706e6ce203ad5b3a6d95c291add157c#p15702
>
>
>     As with any other mythbackend with 4 GB or even 8 GB of RAM, the
>     mythfilldatabase option of --no-allatonce  dramatically reduced
>     memory
>     usage when there are lots of channels, e.g. Astra 28E2 satellite.
>
>
>     Mike
>
>
> What commandline are you using tot start mythfilldatabase?
>
> I use the --xmlfile parameter, and then the --no-allatonce parameter 
> is totally ignored (without reporting anything).
>
> But even when you are using that parameter, all EPG info is kept in 
> RAM multiple times until mythfilldatabase exits, which is totally 
> unnecessary...


Not using any command line to run mythfilldatabase.

I am using the standard built-in feature of mythbackend which runs 
mythfilldatabase command automatically at the appropriate time, using 
the specified grabber from VideoSource configuration, the --no-allatonce 
tells mythfilldatabase to request data 1 day at a time from the grabber 
(in my case tv_grab_zz_sdjson) instead of all data, which could be up to 
3 weeks. Using --no-allatonce the grabber is run multiple times until no 
data is returned.

To illustrate the difference, the data file size for Sky Satellite 
(Astra 28E2), using grabber tv_grab_zz_sdjson for all possible channels, 
results in file size of 211 MB without --no-allatonce and a file size of 
14 MB with --no-allatonce.


In mythtv-setup:
1. General>Programme Schedule Downloading Options
     right arrow to Automatically update programme listings then right 
arrow to Listing data arguments, this where --no-allatonce goes.


Out of interest which xmltv grabber are you using ?


Mike

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-dev/attachments/20200209/a5a3b2c2/attachment.htm>