[mythtv-users] MythFrontend out of memory problem

Fri May 20 08:35:18 UTC 2022

> On 20 May 2022, at 4:30 am, Kenneth Emerson <kenneth.emerson at gmail.com> wrote:
> 
> 
> 
> On Thu, May 19, 2022 at 12:19 PM Jay Foster <jayf0ster at roadrunner.com> wrote:
> On 5/19/2022 10:03 AM, Kenneth Emerson wrote:
>> 
>> 
>> On Mon, Apr 11, 2022, 2:53 PM Kenneth Emerson <kenneth.emerson at gmail.com> wrote:
>> 
>> 
>> On Mon, Apr 11, 2022, 1:04 PM Kenneth Emerson <kenneth.emerson at gmail.com> wrote:
>> 
>> 
>> On Mon, Apr 11, 2022, 12:53 AM Stephen Worthington <stephen_agent at jsw.gen.nz> wrote:
>> On Sun, 10 Apr 2022 23:00:17 -0500, you wrote:
>> 
>> 
>> >> Stephen:
>> >
>> >
>> >
>> >The system RAM is 8Gb.  There is a 2Gb swap space.  There are about 3,000
>> >recordings.  Here is the output from swap.sh:
>> >
>> >
>> >ken at mythtv:~$ ./swap.sh
>> >
>> >
>> >..............................
>> >
>> >Overall swap used: 798096 kB
>> >
>> >
>> >========================================
>> >
>> >kB      pid     name
>> >
>> >
>> >========================================
>> >
>> >404072  1408    mythfrontend.re
>> >
>> >77376   1365    xfdesktop
>> >
>> >42080   1277    xfwm4
>> >
>> >38504   1377    applet.py
>> >
>> >37120   1509    blueman-tray
>> >
>> >34504   1426    blueman-applet
>> >
>> >15624   1131    xfce4-session
>> >
>> >14224   1310    panel-1-whisker
>> >
>> >12480   1321    panel-10-pulsea
>> >
>> >12224   1317    panel-8-statusn
>> >
>> >11536   1314    panel-6-notific
>> >
>> >10552   1401    update-notifier
>> >
>> >10216   1411    nm-applet
>> >
>> >9408    1360    Thunar
>> >
>> >9336    1282    xfsettingsd
>> >
>> >8608    1313    panel-5-systray
>> >
>> >8576    1318    panel-9-power-m
>> >
>> >8528    1385    xfce4-power-man
>> >
>> >8240    1315    panel-7-indicat
>> >
>> >5840    1283    xfce4-panel
>> >
>> >3280    12076   bash
>> >
>> >3272    1382    xiccd
>> >
>> >3032    3791    bash
>> >
>> >2688    1123    systemd
>> >
>> >2640    1442    polkit-gnome-au
>> >
>> >1920    1130    pulseaudio
>> >
>> >1224    1437    agent
>> >
>> >504     1145    dbus-daemon
>> >
>> >248     3784    tmux: server
>> >
>> >240     1368    mythfrontend
>> >
>> >
>> >ken at mythtv:~$ free -m
>> >
>> >
>> >                    total        used        free      shared  buff/cache
>> >available
>> >Mem:           7934        2964         202           7        4768
>> > 4668
>> >
>> >If it's meaningful the CPU is an Intel i5-2500K 3.3GHz
>> >
>> >Perhaps I am hitting some limit on number of recordings, but I don't ever
>> >remember there being a (soft?) limit.
>> >The system is a bit old, but I would hope it's powerful enough.  I only use
>> >it as a media center with only MythTv as the main application.
>> >
>> >Regards,
>> >
>> >Ken Emerson
>> 
>> With 8 Gibytes of RAM, it should be fine.  Only 3000 recordings is not
>> a problem (I have 52880 on an 8 Gibyte frontend/backend system). There
>> is no fixed limit on recordings - the list can grow as big as your
>> system will allow, although things do slow down a bit when you have as
>> many as I do.
>> 
>> It does look like the swap space is filling up though - I would
>> suggest increasing that to 8 Gibytes.  But it is currently using
>> 798096 kiB, which is under half full so it is not obvious that it will
>> run out of swap.
>> 
>> I have my system and swap partitions on a fast M.2 NVME SSD, which
>> helps with swapping.
>> 
>> The free command gives you the total RAM and swap usage.  This is my
>> system with mythfrontend showing my recordings list:
>> 
>> root at mypvr:~# free -m
>>               total        used        free      shared  buff/cache
>> available
>> Mem:           7902        2829         120          23        4952
>> 4752
>> Swap:         10239        1556        8683
>> 
>> (Sorry about the line wrapping - my email client does that and it is
>> not able to be turned off).
>> 
>> So your frontend only system with only 3000 recordings should be just
>> fine.  So my best guess is that something is triggering a bug that is
>> causing a memory leak.  But the problem might be in another program,
>> rather than mythfrontend.  The OOM system is known for killing the
>> largest RAM user, when the problem program that is suddenly using up
>> RAM is a different program.
>> 
>> There was one old bug that caused the cache directory not to be
>> cleaned up properly, and mythfrontend would then load the cache and
>> run out of RAM.  That was fixed a while ago, but it is possible that
>> you still somehow have a huge cache.  So take a look at
>> $(HOME)/.mythtv/cache and see how big it is:
>> 
>> root at mypvr:~# du -hcs /home/stephen/.mythtv/cache/
>> 376M    /home/stephen/.mythtv/cache/
>> 376M    total
>> 
>> This mine now, but when the bug was happening, it was many gibytes. If
>> yours is too large, shut down mythfrontend and do:
>> 
>> rm -r $(HOME)/.mythtv/cache
>> 
>> Mythfrontend should then rebuild the cache as required.
>> 
>> I would suggest upgrading to v32.  That is the currently supported
>> version, so if you have a problem that needs the devs to fix it, you
>> will need to be on v32 for them to be interested.  And it is vaguely
>> possible your problem may be fixed already in v32-fixes.
>> 
>> 
>> Stephen:
>> Unfortunately, I decided to take the plunge and upgrade to 32. Looked simple; however, in stuck years ago right at the beginning running mythtv-setup with "Failed to get schema upgrade lock". Can't go back or forward at this point. I have a good DB backup just don't know what to do with it.
>> 
>> Ken Emerson
>> 
>> Update:
>> 
>> I changed the hostname in the setup dialog from localhost to mythtv (the actual hostname) and then rebooted. The master backend ran  after the reboot and asked me if I wanted to upgrade the schema; answered yes and all I good now; upgraded from 1361 to 1376. Don't fully understand the problem. There were no clues in the mythtv-setup log. Perhaps the backend takes care of all that now?
>> 
>> Anyway, sorry for all the cruft. 
>> 
>> BTW: 681Mb in .mythtv/cache
>> Will have to wait several weeks to see if the OOM kill problem is gone.
>> 
>> Thanks for your time and help.
>> 
>> Regards,
>> Ken Emerson
>> 
>> It's been a month now, and I haven't seen a re-occurance of the  OOM killer taking out the frontend; however, by keeping an eye out on resources I notice that memory is still tight.  I've updated myth to version 32 and did maintenance on the cache disk space as suggested. This morning when a OTA program was recording with no one watching any recordings, the frontend.real task (using top) was using the majority of resources, both cpu and memory. CPU was 47-80% range with memory 80-90% range. Along  with the frontend, mythcommflag was also running using minimal resources as was the backend. frontend.real was always first in listing using top looking at both cpu and  memory. 
>> 
>> Using free, there was only about 60mb memory available (system has 8 gb installed). I don't understand what the frontend is doing when there is nothing being played back: just sitting at the recordings menu. I also have a 2gb swap partition  that is also more than 90% consumed.
>> 
>> FYI: This is a dedicated frontend/backend server with no other services running other than what myth requires.
>> 
>> I've ordered and am now waiting for an additional 8gb of memory but would appreciate any enlightenment on what the frontend is busy doing.
>> 
>> Regards, 
>> 
>> Ken Emerson 
>> 
>> 
>> _______________________________________________
>> mythtv-users mailing list
>> 
>> mythtv-users at mythtv.org
>> http://lists.mythtv.org/mailman/listinfo/mythtv-users
>> http://wiki.mythtv.org/Mailing_List_etiquette
>> 
>> MythTV Forums: 
>> https://forum.mythtv.org
> My combined FE/BE also has 8GB or RAM and has no issues (running 0.28).  Something to keep in mind when looking at how much RAM is 'free' (/proc/meminfo) is that the linux kernel will use any available RAM for caching, so the reported free RAM is usually low.  This is not indicative of a problem (i.e., memory leak).
> Jay
> 
> Thanks for the input.  The problem seems to be intermittent. I'm looking at the memory now with two programs recording (just for a test).  The results from the 'free' command are:
> 
>               total        used        free      shared  buff/cache   available
> Mem:          7.7Gi       6.8Gi       130Mi       5.0Mi       884Mi       752Mi
> Swap:         2.0Gi       2.0Gi          0B
> 
> With the exception of the swap numbers, this seems reasonable.  Maybe everything is fine.  Watching the numbers it appears the kernel is doing what it's supposed to do.  I still don't understand why swap is getting used.  I'll see what happens when I add the addtional 8Gb of RAM next week.

The trouble is when us mere mortals try to understand ,,,
If you have oodles of ram and some of it is untouched (as opposed to unused) then the kernel will swap that out so that you’ve got more cache that is *touched* and there are *signicant* arguments on ram vs swap with tuning available.
So you might go from

4G used 4G free swap 0
to
2G used 6G free swap 2G

eg consider these

'echo 'vm.swappiness=1' > /etc/sysctl.d/99-sysctl.conf'
'echo 'vm.vfs_cache_pressure=50' >> /etc/sysctl.d/99-sysctl.conf’

James