[mythtv-users] System freezes

[Stephen Worthington]

On Mon, 20 Mar 2023 16:22:22 +0000, you wrote:

>Since the end of February I've been getting occasional freezes of my 
>Fedora 36 / MythTV master box, needing a complete reboot.  Their onset 
>coincided with a major update of the desktop environment (KDE) and I 
>have posted about the problem on the fedora-kde users list; there hasn't 
>been much response.
>
>My best clue about the cause seems to be a 'canary' from the 
>rtkit-daemon, where the 'action' has often included the freeze.  My 
>fixes-32 box (still with el7) has no canary events and I've added a few 
>lines relating to its daemon at the end.
>
>Any suggestions?  Thanks.  The 'tda1004' refs are to drop tuner reinit 
>events that seem to be routine.
>
>John P
>
>{{{
>
>[john at HPFed ~]$ sudo SYSTEMD_COLORS=false journalctl --since -40d  | 
>grep -v tda1004x | grep  canary
>
>[sudo] password for john:
>Feb 27 01:39:02 HPFed rtkit-daemon[888]: The canary thread is apparently 
>starving. Taking action.
>Feb 28 18:29:01 HPFed rtkit-daemon[906]: The canary thread is apparently 
>starving. Taking action.
>Mar 02 02:41:48 HPFed rtkit-daemon[903]: The canary thread is apparently 
>starving. Taking action.
>Mar 04 02:34:24 HPFed rtkit-daemon[889]: The canary thread is apparently 
>starving. Taking action.
>Mar 07 18:54:38 HPFed rtkit-daemon[903]: The canary thread is apparently 
>starving. Taking action.
>Mar 10 01:58:04 HPFed rtkit-daemon[909]: The canary thread is apparently 
>starving. Taking action.
>Mar 13 01:19:45 HPFed rtkit-daemon[887]: The canary thread is apparently 
>starving. Taking action.
>Mar 15 02:18:06 HPFed rtkit-daemon[891]: The canary thread is apparently 
>starving. Taking action.
>Mar 20 01:57:02 HPFed rtkit-daemon[887]: The canary thread is apparently 
>starving. Taking action.
>[john at HPFed ~]$
>
>Fixes-32 system:
>
>[root at HP_Box john]# journalctl -S -4d | grep  rtkit
>
>Mar 20 15:13:36 HP_Box rtkit-daemon[875]: Supervising 6 threads of 3 
>processes of 1 users.
>Mar 20 15:13:36 HP_Box rtkit-daemon[875]: Supervising 6 threads of 3 
>processes of 1 users.
>Mar 20 15:13:36 HP_Box rtkit-daemon[875]: Supervising 6 threads of 3 
>processes of 1 users.
>
>Fedora 36 most recent exammples:
>
>[john at HPFed ~]$ sudo SYSTEMD_COLORS=false journalctl --since -6d  | grep 
>-v tda1004x | grep -C 5 canary
>--
>Mar 15 02:01:03 HPFed anacron[56959]: Anacron started on 2023-03-15
>Mar 15 02:01:03 HPFed run-parts[56961]: (/etc/cron.hourly) finished 0anacron
>Mar 15 02:01:03 HPFed CROND[56947]: (root) CMDEND (run-parts 
>/etc/cron.hourly)
>Mar 15 02:01:03 HPFed anacron[56959]: Normal exit (0 jobs run)
>Mar 15 02:09:32 HPFed kioslave5[57197]: kf.coreaddons: Expected a 
>KPluginFactory, got a KIOPluginForMetaData
>Mar 15 02:18:06 HPFed rtkit-daemon[891]: The canary thread is apparently 
>starving. Taking action.
>
>
>-- Boot 9895bb71d2bb46228c4a08159c32218a --
>Mar 15 08:32:40 HPFed kernel: Linux version 6.1.15-100.fc36.x86_64 
>(mockbuild at bkernel02.iad2.fedoraproject.org) (gcc (GCC) 12.2.1 20221121 
>(Red Hat 12.2.1-4), GNU ld version 2.37-37.fc36) #1 SMP PREEMPT_DYNAMIC 
>Fri Mar  3 17:22:46 UTC 2023
>Mar 15 08:32:40 HPFed kernel: Command line: 
>BOOT_IMAGE=(hd0,msdos1)/vmlinuz-6.1.15-100.fc36.x86_64 
>root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap 
>rd.lvm.lv=fedora/root rhgb quiet
>Mar 15 08:32:40 HPFed kernel: x86/fpu: x87 FPU will use FXSAVE
>Mar 15 08:32:40 HPFed kernel: signal: max sigframe size: 1440

There are endless different scenarios for system freezes, so it would
be helpful to have a bit more information about your particular one.

What exactly happens when it freezes?  Can you still ssh into the box
and reboot it that way (ie is the display frozen but the box still
running underneath that)?  Can you still ping it and get a response?
Do you have a systemd early debug shell enabled?  See here for how to
do that:

https://freedesktop.org/wiki/Software/systemd/Debugging/

If you do have an early debug shell available, does Ctrl-Alt-F9 get
you to it?  Does it still work and allow you to do a reboot command?

What sort of video card does it use?  Nvidia driver updates are a
common cause of freeze problems.

When you reboot, how are you doing that?  Do you have SysRq support
enabled so you can use the SysRq REISUB keyboard sequence to do the
reboot as safely as possible to prevent filesystem corruption?

https://fedoraproject.org/wiki/QA/Sysrq

Do you do a full fsck of at least the system partition(s) before
rebooting the system normally, to ensure the system is not getting
more and more corrupted?  The easiest way to do that is to boot a live
USB image and do it from there, or to have another bootable partition
on the system.