[mythtv-users] mythbackend not responding on port 6543

[Stephen Worthington]

Mythbuntu 16.04
MythTV Version : v0.28-78-g1434df8
MythTV Branch : fixes/0.28
Network Protocol : 88
Library API : 0.28.20160309-1
QT Version : 5.5.1

Last night, I rebooted my main MythTV box to get my DVB-S2 USB tuner
working again.  But after the reboot, mythbackend is no longer working
properly.  It starts up as normal, but once it completes startup,
nothing works properly.  It does not make recordings, and no further
messages are sent to the /var/log/mythtv/mythbackend.log file.  If I
have mythfrontend running while I start mythbackend, it sees that the
backend is up, but after that is only occasionally able to get some
small snippets of communication with mythbackend.  So if I go into
Information Centre > System Status, I do get the Listings Status page,
after a long wait, but the Schedule Status and Tuner Status pages
never show any real information.  The Schedule Status does show the
first few lines (with the numbers of rules) eventually, but shows "0
matching showings" and nothing after that.  The Tuner Status takes
even longer to ever display anything - several minutes - but
eventually does display the tuner list.  While mythbackend is running,
mythfrontend does not pop up the message to say it can not connect to
the backend.  But it does not get sensible or timely replies from the
backend.

To try to work out what is wrong, I used the "mythutil --resched"
command to try to connect to the backend.  I added the "-v all
--loglevel debug" options and what I found was that mythutil was able
to talk to the mysql and get things from the database, but when it
tried to use the connection information it got from the database to
talk to mythbackend, it was not working.  The log from mythutil was
showing that mythbackend was allowing TCP connections on port 6543,
but was not responding with any data when a query was sent.

I then tried mythutil from a remote PC instead of locally, and was
able then to use Wireshark to capture the packets.  That showed that
mythutil was connecting to mythbackend port 6543 (it got a TCP ACK),
but when it sent a query message, there was no reply, just a keepalive
ACK with no data from the mythbackend box.  Eventually mythutil timed
out and closed the TCP connection, then gave a "Protocol version check
failure" message.

I have put the logs from this testing on my web server:

http://www.jsw.gen.nz/mythtv/6543/mythbackend.log
http://www.jsw.gen.nz/mythtv/6543/mythutil_lith_to_mypvr_failed.log
http://www.jsw.gen.nz/mythtv/6543/mythutil_lith_to_mypvr_failed.pcapng
http://www.jsw.gen.nz/mythtv/6543/mythutil_mypvr_to_lith_worked.log
http://www.jsw.gen.nz/mythtv/6543/mythutil_mypvr_to_lith_worked.pcapng
http://www.jsw.gen.nz/mythtv/6543/netstat.log

mypvr = 10.0.2.4 = main MythTV box
lith = 10.0.2.12 = test PC (also a MythTV box)

The mythbackend.log file shows mythbackend being started at 22:38:46,
and all of that log looks normal (except for the high debug level
creating lots of messages).  The log stops at 22:39:51 after
mythbackend completes startup.  But in fact mythbackend was actually
shut down at around 01:00 - nothing more got logged after startup.

The netstat.log file is the output of the "netstat -apn" command with
mythbackend running and mythfrontend is running and trying to get the
Listings Status page from mythbackend.

The .pcapng files are the Wireshark captures of the IPv4 traffic at
the time the mythutil log was being captured.

I really need some help here please - I have about run out of ideas
for things to try.  Until I can figure out what is going wrong, I can
not use my main MythTV box.  I have managed to set up my test PC as a
backup MythTV box that can record my free-to-air programmes for a
little while until it runs out of disk space, but I can not record
from my subscription satellite without the main box.

I have tried restoring an older database from before the problem, but
that did not help at all, and I have rebooted several times.  It feels
like I have somehow managed to misconfigure the networking on that PC,
or that a recent update that got installed has caused the problem.  So
I have made sure that none of the other boxes will have any updates
done until I have this problem worked out.