[mythtv-users] Mythbackend recording glitches

[Stephen Worthington]

On Mon, 15 Oct 2018 22:18:45 -0700, you wrote:

>As you know, I had a disk failure on my Mythbuntu 8 machine.  I did not
>migrate any files or databases to the new machine.  It is a straight
>Mythbunty 16.041 install with no intervention on my part.  I got whatever
>partitions it selected and nothing was carried over. We just lost a few
>shows we wanted to watch but found them online so it wasn't a big deal.

The standard install is not the recommended install.  See below.

>The antenna, cable, connectors are all professional quality.  The HDHR
>reports 100% signal strength.  I don't think that is the problem.

Signal strength is different from signal quality and each tuner
measures it differently, so those values can only be compared with the
same exact model of tuner, not with anything else.  That said, 100%
signal strength is a good indicator.  But there can also be problems
with too much signal.

>My next experiment is recording without mythcommflag running.  I strongly
>suspect that is the problem. I will post the result of whatever tests I run
>tomorrow.

Yes, that could indeed be the problem, depending on how mythcommflag
is running.  Mythcommflag uses one CPU core only.  If your system has
a modern multicore CPU like mine, then it is likely capable of running
one mythcommflag process per core and it will run them in real time -
each CPU core is capable of doing all the processing needed for
mythcommflag on one recording faster than the recording is arriving
from the tuner, with enough CPU left over for all the other normal
operations.  So for a system like that, you set up mythcommflag with a
maximum number set to either the number of CPU cores, or if you are
conservative, to one less than the number of CPU cores.  And you set
mythcommflag to run as soon as the recording has started, rather than
after it is finished.  That means that mythcommflag will process the
recording data it needs out of the disk buffers, before or at the same
time as that data is written to disk.  Run that way, mythcommflag only
increases the disk usage a little for when it writes its data to the
database.

If your system is incapable of running mythcommflag processes in real
time, then you need a completely different strategy, as mythcommflag
will lag behind the recording and will need to re-read it from disk,
so each mythcommflag process will need as much disk access as each
recording does.  Mythcommflag is run as lower priority, so the
recording threads will be preferred over mythcommflag whenever there
is conflict, but once mythcommflag does get disk access, it will
likely have to move the heads to do its read - and that is what makes
up the bulk of the disk usage, and will interfere with the recording
process.  So you have to think of a non-real time mythcommflag as
around the same disk usage as a recording or playing back a recording,
and do your disk usage calculations accordingly.  In your case, with
only one hard disk shared between the system, database and possible
multiple recordings, there is a lot of scope for mythcommflag to cause
problems if it is not running in real time.  However, almost any
system should be able to do one recording at a time, and play back a
recording, and run mythcommflag once, all at the same time as there is
background system and database activity.  With only one recording at
once, I would only expect problems if the system was very old or had a
very slow hard disk.

>New problem.  On MythWeb, the left column shows small screen shot out of
>whatever show was recorded. The oldest show is August 29 and there are 52
>recordings that show the clip, then 2 that do not, one that does and all 4
>from today are without the clip.  Not that important but what is going on?

That sounds like mythpreviewgen may not been run on those recordings.
Do they show an icon on mythfrontend?  The ones in mythweb are derived
from the mythfrontend ones, and then cached in a web server cache.  I
do not use mythweb much, so I do not know exactly how all that works.

>And don't forget that the progress bar is missing.  I think I need a new
>disk and new install.  Random things are happening.  It is near Halloween
>so maybe it is ghosts.

Yes, the progress bar is an ongoing mystery.  I believe it can be
missing if the recordedseek data is not available for it to be
calculated from, which leads to the possibility your database has a
corrupt or crashed recordedseek table.  Recordedseek is the table most
likely to be affected if there are problems with the database.  See
below for how to check and fix it if necessary.

>Question.  If I get a new HD and install Mythbuntu 16.04.5 on it to start
>over, how should I set up the partitions?  A 1T HD is more than big enough
>for what we do so it is not a big expense, just another time sink.

If you have your system and recordings on one hard disk, the
recommended way to do that is to have an ext4 system partition of
relatively small size - 50 Gibytes is sufficient unless you have a
monster database like I have.  I use 100 Gibytes now.  Then the rest
of the drive is a separate recording and video storage partition.  It
should be XFS or JFS - I recommend JFS because I have always used that
and know all about it.  But lots of other people use XFS or even some
other newer things.  In the recording partition, you create a
subdirectory where your recordings go (I call mine "recordings"), and
you may have another subdirectory to store video files such as
Internet downloads and ripped DVDs and BluRays.  You set up the
storage groups to point to those subdirectories.

But much better these days is to have an SSD for the system drive, and
one or more hard disks for the recordings and videos and anything else
that needs big files or lots of files.  I use a super fast M.2 NVMe
SSD for my system drive now, but a normal SATA SSD is still much
better than a hard drive - there are no heads to move.

>I do not understand your comment about recordedseek.  Where do you see that?

The recordedseek table stores data that is needed to skip forwards and
backwards quickly in variable bit rate recordings (which is all
recordings these days).  Also stored in the recordedseek table are
other things like bookmarks and the data from mythcommflag.  When you
are making a recording, as the data is received from the tuner, the
recording process calculates the seek table data from the incoming
data and regularly writes it to the recordedseek table.  This uses
quite a bit of disk access, and keeps on moving the heads back to the
recordedseek table file and its index file.  The recordedseek table is
usually the largest one in the database and is frequently larger than
all the other data in the database combined.  So if your old MythTV
system happened to be so ancient that the recordedseek table did not
exist, then adding all of that access to the recordedseek table could
be what is making the difference between recordings working or not.
But recordedseek has been around for a very long time - if my memory
serves me, it was there when I first used MythTV in 2007.

As you have just installed a new MythTV setup, you will likely not
have set up database backups or automatic checking and repair of the
database.  That used to be done using Mythbuntu Control Centre (MCC),
but MCC is no longer supported.  Installs done on 14.xx or before
where MCC still worked and then upgraded will likely have the backups
and check & repair tasks installed by MCC and they will keep on
working as upgrades happen.  But now from 16.xx onwards, they need to
be manually installed.

In the /usr/share/mythtv directory you will find the
mythconverg_backup.pl program which does the database backups.  It is
normally run by a script in /etc/cron.weekly that calls it once a
week.  I run it from there, and also run it from /etc/cron.daily to
daily backups to a different location.  This is the script that MCC
used to install in /etc/cron.weekly for automated backups:

root at mypvr:/etc/cron.weekly# cat mythtv-database
#!/bin/sh
# /etc/cron.weekly/mythtv-database script - check and backup
mythconverg tables
# Copyright 2005/12/02 2006/10/08 Paul Andreassen
#                      2010 Mario Limonciello

set -e -u

DBNAME="mythconverg"
DEBIAN="--defaults-extra-file=/etc/mysql/debian.cnf"

/usr/bin/mysqlcheck $DEBIAN -s $DBNAME

/usr/local/bin/mythconverg_backup.pl

/usr/bin/logger -p daemon.info -i -t${0##*/} "$DBNAME checked and
backed up."

# End of file.


I use a modified version of it to do my backups to a different
directory from the standard location (which is
/var/lib/mythtv/db_backups).

In the /usr/share/doc/mythtv-backend/contrib/maintenance/ directory
you will find the optimize_mythdb.pl script.  To check if you have any
problems with your database, run that manually now.  Then copy
optimize_mythdb.pl file to /etc/cron.daily/optimize_mythdb (note the
different file name) to get it to automatically be run every day.  I
am guessing that the check will show that your recordedseek table is
crashed or at least damaged, and that is the reason that the progress
bar has been disappearing.  If that is the case, then I hope that
optimize_mythdb will simply fix the problem.  If not, then most of the
data in recordedseek can be recreated from the recordings if
necessary, so it is possible to drop the recordedseek table, re-create
it, and then use "mythutil --checkrecordings --fixseektable" to
recreate all the seek table data.  Then you would need to run
mythcommflag on all recordings to recreate the commflagging data.

If the database is being regularly checked and repaired, any problems
that do happen will usually be fixed in the daily check and will
likely not cause any trouble.  If they do not get fixed soon after
they happen, they can cause subsequent problems in the database that
are much more difficult to fix and ultimately can become completely
unfixable.  So I consider it vital that optimize_mythdb is run every
day.

You also need to know if the daily optimize_mythdb sees any problems -
it may not have been able to fix them.  So the best way to do that is
to allow cron jobs such as optimize_mythdb to have their results
emailed to you.  The simple way to do that is to install the command
line "mail" program, from the package "mailutils", or the alternate
version in the package "s-nail".  Installing that should trigger the
installation of postfix and everything else required to send and
receive emails.  It is a *long* time since I did it, but I think the
default config that gets installed does everything you need, and sets
up email accounts for all users on the system.  Then cron jobs will
automatically have their results emailed to root@<your system name>.
You can alter where they are sent to if you have another email
account, either via an email server on your network or an external
one, but I just keep a root terminal open on my MythTV box and type
"mail" every day to see the results.