[mythtv-users] Mythbackend recording glitches

[Allen Edwards]

On Tue, Oct 16, 2018 at 8:47 AM Allen Edwards <allen.p.edwards at gmail.com>
wrote:

>
>
> On Tue, Oct 16, 2018 at 2:11 AM Stephen Worthington <
> stephen_agent at jsw.gen.nz> wrote:
>
>> On Mon, 15 Oct 2018 22:18:45 -0700, you wrote:
>>
>> >As you know, I had a disk failure on my Mythbuntu 8 machine.  I did not
>> >migrate any files or databases to the new machine.  It is a straight
>> >Mythbunty 16.041 install with no intervention on my part.  I got whatever
>> >partitions it selected and nothing was carried over. We just lost a few
>> >shows we wanted to watch but found them online so it wasn't a big deal.
>>
>> The standard install is not the recommended install.  See below.
>>
>> >The antenna, cable, connectors are all professional quality.  The HDHR
>> >reports 100% signal strength.  I don't think that is the problem.
>>
>> Signal strength is different from signal quality and each tuner
>> measures it differently, so those values can only be compared with the
>> same exact model of tuner, not with anything else.  That said, 100%
>> signal strength is a good indicator.  But there can also be problems
>> with too much signal.
>>
>> >My next experiment is recording without mythcommflag running.  I strongly
>> >suspect that is the problem. I will post the result of whatever tests I
>> run
>> >tomorrow.
>>
>> Yes, that could indeed be the problem, depending on how mythcommflag
>> is running.  Mythcommflag uses one CPU core only.  If your system has
>> a modern multicore CPU like mine, then it is likely capable of running
>> one mythcommflag process per core and it will run them in real time -
>> each CPU core is capable of doing all the processing needed for
>> mythcommflag on one recording faster than the recording is arriving
>> from the tuner, with enough CPU left over for all the other normal
>> operations.  So for a system like that, you set up mythcommflag with a
>> maximum number set to either the number of CPU cores, or if you are
>> conservative, to one less than the number of CPU cores.  And you set
>> mythcommflag to run as soon as the recording has started, rather than
>> after it is finished.  That means that mythcommflag will process the
>> recording data it needs out of the disk buffers, before or at the same
>> time as that data is written to disk.  Run that way, mythcommflag only
>> increases the disk usage a little for when it writes its data to the
>> database.
>>
>> If your system is incapable of running mythcommflag processes in real
>> time, then you need a completely different strategy, as mythcommflag
>> will lag behind the recording and will need to re-read it from disk,
>> so each mythcommflag process will need as much disk access as each
>> recording does.  Mythcommflag is run as lower priority, so the
>> recording threads will be preferred over mythcommflag whenever there
>> is conflict, but once mythcommflag does get disk access, it will
>> likely have to move the heads to do its read - and that is what makes
>> up the bulk of the disk usage, and will interfere with the recording
>> process.  So you have to think of a non-real time mythcommflag as
>> around the same disk usage as a recording or playing back a recording,
>> and do your disk usage calculations accordingly.  In your case, with
>> only one hard disk shared between the system, database and possible
>> multiple recordings, there is a lot of scope for mythcommflag to cause
>> problems if it is not running in real time.  However, almost any
>> system should be able to do one recording at a time, and play back a
>> recording, and run mythcommflag once, all at the same time as there is
>> background system and database activity.  With only one recording at
>> once, I would only expect problems if the system was very old or had a
>> very slow hard disk.
>>
>> >New problem.  On MythWeb, the left column shows small screen shot out of
>> >whatever show was recorded. The oldest show is August 29 and there are 52
>> >recordings that show the clip, then 2 that do not, one that does and all
>> 4
>> >from today are without the clip.  Not that important but what is going
>> on?
>>
>> That sounds like mythpreviewgen may not been run on those recordings.
>> Do they show an icon on mythfrontend?  The ones in mythweb are derived
>> from the mythfrontend ones, and then cached in a web server cache.  I
>> do not use mythweb much, so I do not know exactly how all that works.
>>
>> >And don't forget that the progress bar is missing.  I think I need a new
>> >disk and new install.  Random things are happening.  It is near Halloween
>> >so maybe it is ghosts.
>>
>> Yes, the progress bar is an ongoing mystery.  I believe it can be
>> missing if the recordedseek data is not available for it to be
>> calculated from, which leads to the possibility your database has a
>> corrupt or crashed recordedseek table.  Recordedseek is the table most
>> likely to be affected if there are problems with the database.  See
>> below for how to check and fix it if necessary.
>>
>> >Question.  If I get a new HD and install Mythbuntu 16.04.5 on it to start
>> >over, how should I set up the partitions?  A 1T HD is more than big
>> enough
>> >for what we do so it is not a big expense, just another time sink.
>>
>> If you have your system and recordings on one hard disk, the
>> recommended way to do that is to have an ext4 system partition of
>> relatively small size - 50 Gibytes is sufficient unless you have a
>> monster database like I have.  I use 100 Gibytes now.  Then the rest
>> of the drive is a separate recording and video storage partition.  It
>> should be XFS or JFS - I recommend JFS because I have always used that
>> and know all about it.  But lots of other people use XFS or even some
>> other newer things.  In the recording partition, you create a
>> subdirectory where your recordings go (I call mine "recordings"), and
>> you may have another subdirectory to store video files such as
>> Internet downloads and ripped DVDs and BluRays.  You set up the
>> storage groups to point to those subdirectories.
>>
>> But much better these days is to have an SSD for the system drive, and
>> one or more hard disks for the recordings and videos and anything else
>> that needs big files or lots of files.  I use a super fast M.2 NVMe
>> SSD for my system drive now, but a normal SATA SSD is still much
>> better than a hard drive - there are no heads to move.
>>
>> >I do not understand your comment about recordedseek.  Where do you see
>> that?
>>
>> The recordedseek table stores data that is needed to skip forwards and
>> backwards quickly in variable bit rate recordings (which is all
>> recordings these days).  Also stored in the recordedseek table are
>> other things like bookmarks and the data from mythcommflag.  When you
>> are making a recording, as the data is received from the tuner, the
>> recording process calculates the seek table data from the incoming
>> data and regularly writes it to the recordedseek table.  This uses
>> quite a bit of disk access, and keeps on moving the heads back to the
>> recordedseek table file and its index file.  The recordedseek table is
>> usually the largest one in the database and is frequently larger than
>> all the other data in the database combined.  So if your old MythTV
>> system happened to be so ancient that the recordedseek table did not
>> exist, then adding all of that access to the recordedseek table could
>> be what is making the difference between recordings working or not.
>> But recordedseek has been around for a very long time - if my memory
>> serves me, it was there when I first used MythTV in 2007.
>>
>> As you have just installed a new MythTV setup, you will likely not
>> have set up database backups or automatic checking and repair of the
>> database.  That used to be done using Mythbuntu Control Centre (MCC),
>> but MCC is no longer supported.  Installs done on 14.xx or before
>> where MCC still worked and then upgraded will likely have the backups
>> and check & repair tasks installed by MCC and they will keep on
>> working as upgrades happen.  But now from 16.xx onwards, they need to
>> be manually installed.
>>
>> In the /usr/share/mythtv directory you will find the
>> mythconverg_backup.pl program which does the database backups.  It is
>> normally run by a script in /etc/cron.weekly that calls it once a
>> week.  I run it from there, and also run it from /etc/cron.daily to
>> daily backups to a different location.  This is the script that MCC
>> used to install in /etc/cron.weekly for automated backups:
>>
>> root at mypvr:/etc/cron.weekly# cat mythtv-database
>> #!/bin/sh
>> # /etc/cron.weekly/mythtv-database script - check and backup
>> mythconverg tables
>> # Copyright 2005/12/02 2006/10/08 Paul Andreassen
>> #                      2010 Mario Limonciello
>>
>> set -e -u
>>
>> DBNAME="mythconverg"
>> DEBIAN="--defaults-extra-file=/etc/mysql/debian.cnf"
>>
>> /usr/bin/mysqlcheck $DEBIAN -s $DBNAME
>>
>> /usr/local/bin/mythconverg_backup.pl
>>
>> /usr/bin/logger -p daemon.info -i -t${0##*/} "$DBNAME checked and
>> backed up."
>>
>> # End of file.
>>
>>
>> I use a modified version of it to do my backups to a different
>> directory from the standard location (which is
>> /var/lib/mythtv/db_backups).
>>
>> In the /usr/share/doc/mythtv-backend/contrib/maintenance/ directory
>> you will find the optimize_mythdb.pl script.  To check if you have any
>> problems with your database, run that manually now.  Then copy
>> optimize_mythdb.pl file to /etc/cron.daily/optimize_mythdb (note the
>> different file name) to get it to automatically be run every day.  I
>> am guessing that the check will show that your recordedseek table is
>> crashed or at least damaged, and that is the reason that the progress
>> bar has been disappearing.  If that is the case, then I hope that
>> optimize_mythdb will simply fix the problem.  If not, then most of the
>> data in recordedseek can be recreated from the recordings if
>> necessary, so it is possible to drop the recordedseek table, re-create
>> it, and then use "mythutil --checkrecordings --fixseektable" to
>> recreate all the seek table data.  Then you would need to run
>> mythcommflag on all recordings to recreate the commflagging data.
>>
>> If the database is being regularly checked and repaired, any problems
>> that do happen will usually be fixed in the daily check and will
>> likely not cause any trouble.  If they do not get fixed soon after
>> they happen, they can cause subsequent problems in the database that
>> are much more difficult to fix and ultimately can become completely
>> unfixable.  So I consider it vital that optimize_mythdb is run every
>> day.
>>
>> You also need to know if the daily optimize_mythdb sees any problems -
>> it may not have been able to fix them.  So the best way to do that is
>> to allow cron jobs such as optimize_mythdb to have their results
>> emailed to you.  The simple way to do that is to install the command
>> line "mail" program, from the package "mailutils", or the alternate
>> version in the package "s-nail".  Installing that should trigger the
>> installation of postfix and everything else required to send and
>> receive emails.  It is a *long* time since I did it, but I think the
>> default config that gets installed does everything you need, and sets
>> up email accounts for all users on the system.  Then cron jobs will
>> automatically have their results emailed to root@<your system name>.
>> You can alter where they are sent to if you have another email
>> account, either via an email server on your network or an external
>> one, but I just keep a root terminal open on my MythTV box and type
>> "mail" every day to see the results.
>> _______________________________________________
>>
>
> A few test results but little progress to report.
>
> 1) It isn't mythcommfill.  I killed it and there were still glitches in
> the recording. I had to wait 15 minutes for my wife to catch up to when I
> killed it in the recording but almost as soon as I looked, there was a
> glitch similar to what I called the bad one.
> 2) I added a 10dB pad in front of the TV so the TV was now getting about
> 1/5 the signal or more than two splitters.  Looked fine. But again, these
> signal levels have worked for about a decade and HDHomeRun does not report
> any issues when it does a recording.
> 3) MythWeb has a database check and is showed everything was OK.  I did an
> extended check and it was still showing OK.  Just to be sure I did a repair
> and optimize. Still no progress bar on a new recording. I also ran the
> script you suggested which I guess just did the same thing with no errors.
> (I made it a cron job as you suggested optimize_mythdb).
> 4) The little icons started working again. The larger version is viewable
> when clicking on Recording Details on Mythweb so just the smaller ones
> missing.  But both recordings this morning and one of the 5 from yesterday
> have the little screen capture as they should.
>
> Out of ideas.
>

Yesterday I found a loose connector on tuner 0. HDHomerun config showed
yellow signal quality that went away when it was tightened.  Fixed it.
Still glitches, no improvement this morning.

Small screen shots in Mytyweb sometimes there, sometimes not.  Progress bar
still missing. The missing screenshots are in the Recordings folder but the
link to them on Mythweb is to a different folder and file name.

Is there a way to reinstall myth without losing my programs?  I wish I
could go back to Mythbuntu 8.

Allen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-users/attachments/20181017/5ca95c82/attachment.html>