[mythtv-users] backend stops working for a couple hours every morning at 2?

[Stephen Worthington]

On Wed, 12 Jan 2022 09:47:58 -0500, you wrote:

>Hey, all. Over the past week, I've noticed my backend stops working every
>morning at 2am local time (0700 UTC). The symptoms are that I get emails
>from my every-10-minutes cron job that runs /usr/sbin/mythtv-update-motd .
>The emails repeat for roughly 2-3 hours, give or take, and say "Our child
>has stopped talking to us, kill it off. Unknown error during retrieval of
>status from the MythTV backend." During that time, recordings also silently
>fail.
>
>The problem is that I can't figure out what's causing this at all. I don't
>seem to have any cron jobs or systemd timers that run daily at 2am. I can't
>find anything unusual in the mythtv-backend systemd journal or in any of
>the /var/log/mythtv/*.log files, and in fact, the evidence is that the
>backend process/service is still running. Things that appear in the logs
>every 15 minutes or so continue to appear during the time period when the
>cron emails are going out and recordings aren't working, and systemd
>reports that the service has been running since before last night's gap.
>
>Can you help me figure out what's causing this, or at least help me figure
>out where to look for information or how to increase logging appropriately?
>My usual sysadmin log-reading skills are failing me here...
>
>Thanks!
>Josh

Two hours is a very long time.  Usually, the culprit with this sort of
thing is the database backup happening, but even with my huge database
that only takes 20 minutes or so.  Mythbackend is quite multithreaded,
so bits of it can work when other bits do not.  But if the database is
running very slowly as it can during backups or database checks, as
soon as mythbackend tries to use the database, that bit of it can be
stalled for a long time waiting for it.  So it would pay to run a full
check on all database tables to see if there are any that are crashed.

A long time ago, I had a problem where mythbackend would not record
(or do much else) for a couple of weeks, and I eventually discovered
that the cause was two hard drives that were on an external dual USB
mount.  They had been accidentally turned off without being unmounted
properly and their JFS partitions needed a full fsck before they would
mount.  As they had partitions which were used by a storagegroup,
mythbackend was attempting to mount them all the time and getting a
huge timeout each time.  When both drives' timeouts had happened,
mythbackend would operate normally for a second or two before
attempting to mount them again.  The same sort of thing can happen if
mythbackend tries to access a network drive which will not connect
properly - badly configured NFS mounts are a frequent problem with
this.  SAMBA mounts using the default configuration seem to be more
forgiving, but they can cause problems too.