[mythtv-users] mythbackend not responding on port 6543

[Stephen Worthington]

On Sat, 19 Nov 2016 00:22:46 +1300, you wrote:

>On Fri, 18 Nov 2016 04:49:21 +0000, you wrote:
>
>>I assume top / iotop / df -h etc didnt show anything interesting? Swapping? 
>>CPU @ 400%? Full partition? etc.
>
>Nothing seen on any of them, except the swapper is in use just a
>little, but it seems to just be swapped out startup code and that sort
>of thing.  It seems to be mythfrontend that causes the swapping for
>me, probably due to the huge number of recordings I have.  But the box
>has 8 Gibytes of RAM, and was fine up until the day before my problem
>started.
>
>I said already that the problem happened when I rebooted to get a
>failed USB tuner going again.  I did not mention that this was just
>about 20 minutes after the big 7.8 earthquake hit us here in New
>Zealand.  Where I am (Palmerston North), it was a "felt" earthquake,
>not a damaging one, but it had us wondering for a while as it went on
>for a long time.  I have since checked all the cables and so on to see
>if anything was affected, but in any case MythTV runs just fine with
>my old upgrade 0.26 database from September 2013, so I do not think
>that hardware is the problem.

Aha!  I found the problem!

Mark, thanks for your post.  It triggered me to rechecking the
hardware, and when I did "ll /mnt" I got a directory listing that
failed to complete - it just sat there trying to access something and
not being able to.  Which got me to thinking about the mountpoints and
how they work - which has changed with systemd.  And about how I had
been trying to set up two of my drives (arc1 and arc2) to have them
automount when I switched them on, but not to be needed when I boot in
the first place.  Those two drives are shingled 8 TByte Seagate
ST8000AS0002 drives which I use for archival storage.  I put all the
old recording files on them, and mostly have them turned off.  If I
want to play anything that is on them, I turn them on, then when I am
finished I unmount them and turn them off again.  They are in a USB 3
dual external drive mount.

So what went wrong seems to be that I got my fstab settings wrong for
those drives with systemd, and the result was that the mountpoints for
those two drive under /mnt were in a state where they would hang
anything that attempted to access them.  I am not sure yet whether the
hang would eventually timeout, or if it would go on until that task
was killed.  In any case, that is what was causing mythbackend to do
strange things.  Both those drives were set up in the storagegroup
table:

MariaDB [mythconverg]> select * from storagegroup where
groupname='Archives' order by groupname,id;
+----+-----------+----------+-----------------------+
| id | groupname | hostname | dirname               |
+----+-----------+----------+-----------------------+
| 37 | Archives  | mypvr    | /mnt/arc1/recordings/ |
| 45 | Archives  | mypvr    | /mnt/arc2/recordings/ |
+----+-----------+----------+-----------------------+
2 rows in set (0.00 sec)

So every time mythbackend went to look for a recording, it scanned all
the storagegroup entries for places to look, and then went and tried
to access them to see if it could find the file it wanted.  Whenever
it tried to access the arc1 or arc2 drives, it encountered the hang
situation - the thread doing the call to access via the hanging
mountpoint either never completed that call, or only did so after a
long timeout.  Hence mythbackend had all of its functions that
accessed storagegroups not working.

So the difference between my old working database and the new ones
that did not work was not in the settings table.  It was in the
storagegroup table, which has more entries for all the new drives I
have added between 2013 and 2016, including the two arc drives.

I also have to wonder about whether mythbackend has recently had some
changes in the code it uses to scan mountpoints, as in the past I have
had a hanging mountpoint caused by a drive that needed to be fscked.
But I do not remember any problems with mythbackend - the system
booted fine, and as soon as I had done the fsck and mounted the drive,
MythTV was able to access it.  So perhaps the devs need to review the
code used to access storagegroups, to see if there is some new
vulnerability there, as this problem is a very nasty one to work out
if it hits you.

You might think that the simple solution for me to get my MythTV going
again immediately would just be to turn on the dual drive mount and
have the arc1 and arc2 drive available for systemd to find at boot
time.  But no, it turns out there is another completely different
serious problem with my system in doing that.  Either the BIOS or grub
seems to have a bug where it can not cope with too many drives online
at boot time.  The result is that instead of the grub menu appearing,
I was getting a grub emergency prompt and a message saying that the
grub boot partition UUID was not found.  I have 7 recording drives,
the SSD and the two arc drives on the MythTV PC, which is quite a few
drives and possibly more than the BIOS or grub have been tested with.
Or the problem may lie with the USB 3 drive mount - it does take a lot
longer for the arc drives to become visible in that mount than when I
tried them on a SATA cable.

So after much experimentation (and many curses!) I finally discovered
that I could only get a grub menu to come up if I had the dual drive
mount turned off.  It turns out the only way I can get a reliable boot
at present is to have the dual drive mount off until I get a grub
menu, then turn it on and allow the boot to continue.  If the dual
drive mount is on, I will occasionally get a grub menu to appear, but
it is rare.  I will have to work out just what systemd wants in the
way of settings so that I can get it to be happy with the arc drives
not being present at boot, but still automount them when they do
appear.  And not to be unhappy when they are unmounted again.

But for now, my MythTV box is finally running again.