[mythtv] wanted: developers that use mythfilldatabase

[Hans Dingemans]

On Sun, Feb 9, 2020 at 7:51 AM Mike Bibbings <mike.bibbings at gmail.com>
wrote:

> On 09/02/2020 10:47, Hans Dingemans wrote:
>
>
> On 08/02/2020 23:34, Hans Dingemans wrote:
>> > Hi all,
>> >
>> > First of all big thanks to all of you for this great software; I have
>> > been using Mythtv for years, very happy with it!!! However, as an
>> > enthusiastic open-source developer I always seek possibilities to
>> > improve and to contribute.
>> >
>> > Apart from my production server I now am experimenting with frontend +
>> > backend on a Raspberry Pi, trying to get it so far that it can play
>> > HLS streams from websites like https://watchnewslive.tv .
>> > A problem I encountered is that when trying to load the EPG for all
>> > those stations, mythfilldatabase uses too much memory and freezes up
>> > the (4Gb-version of the) Pi.
>> > Turns out that the entire xml-file is kept in memory multiple times. I
>> > made a patch which reduces memory usage 6 times.
>> >
>> > The nice thing about these offline software is you can test it easily;
>> > I just backup my sql database, load a test xml file with "production"
>> > mythfilldatabase, dump the database, restore it to its old version,
>> > reload the same xml file with my patched mythfilldatabase, dump it
>> > again, and do a compare against the dumps.
>> >
>> > To make sure all the different xmltv-dialects out there are working, I
>> > am looking for test xml-files from allover the world; the more
>> > diverse, the better.
>> > Please send them to my gmail address, if it is too large, send me a pm
>> > so we can see how to transfer.
>> >
>> > Thank you!
>> >
>> > Hans.
>>
>>
>> I did some testing on a Pi4 (4GB) using external xmltv data from
>> Schedules Direct tvgrab_zz_sdjson a while ago see
>>
>> https://forum.mythtv.org/viewtopic.php?f=46&t=3232&sid=8706e6ce203ad5b3a6d95c291add157c#p15702
>>
>>
>> As with any other mythbackend with 4 GB or even 8 GB of RAM, the
>> mythfilldatabase option of --no-allatonce  dramatically reduced memory
>> usage when there are lots of channels, e.g. Astra 28E2 satellite.
>>
>>
>> Mike
>
>
> What commandline are you using tot start mythfilldatabase?
>
> I use the --xmlfile parameter, and then the --no-allatonce parameter is
> totally ignored (without reporting anything).
>
> But even when you are using that parameter, all EPG info is kept in RAM
> multiple times until mythfilldatabase exits, which is totally unnecessary...
>
>
> Not using any command line to run mythfilldatabase.
>
> I am using the standard built-in feature of mythbackend which runs
> mythfilldatabase command automatically at the appropriate time, using the
> specified grabber from VideoSource configuration, the --no-allatonce tells
> mythfilldatabase to request data 1 day at a time from the grabber (in my
> case tv_grab_zz_sdjson) instead of all data, which could be up to 3 weeks.
> Using --no-allatonce the grabber is run multiple times until no data is
> returned.
>
> To illustrate the difference, the data file size for Sky Satellite (Astra
> 28E2), using grabber tv_grab_zz_sdjson for all possible channels, results
> in file size of 211 MB without --no-allatonce and a file size of 14 MB with
> --no-allatonce.
>
>
> In mythtv-setup:
> 1. General>Programme Schedule Downloading Options
>     right arrow to Automatically update programme listings then right
> arrow to Listing data arguments, this where --no-allatonce goes.
>
>
> Out of interest which xmltv grabber are you using ?
>
>
> Mike
>
Very interesting Mike, I hadn't discovered that built-in option yet!
Certainly are going to look at that!

I am using zap2xml for US channels and tvgrabpyAPI for Dutch channels.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-dev/attachments/20200209/aab8f8ff/attachment.htm>