[mythtv-users] Slave backend reboot procedure

Stephen Worthington stephen_agent at jsw.gen.nz
Tue Mar 26 02:47:18 UTC 2019


On Mon, 25 Mar 2019 22:08:41 +0000, you wrote:

>
>
>On 25/03/2019 15:47, Jan Ceuleers wrote:
>> On 25/03/2019 10:22, Stuart Auchterlonie wrote:
>>> It could be coming online before the networking is available
>>> and therefore doesn't come up properly.
>>>
>>> You've no said what distro you are running on, so depending on
>>> which one, and whether it's systemd, you may need to
>>> a) verify this is the issue
>>> b) work out how to ensure the slave backend service is started
>>>     after networking is online.
>> Stuart,
>>
>> Thanks. It's Ubuntu 16.04, i.e. with systemd. Here is the service file
>> on the slave backend:
>>
>> # cat /etc/systemd/system/mythtv-backend.service
>> # JCE modified this file:
>> # 1. set StartLimitInterval to 0 (was 10m)
>> # 2. add dependency on videop* devices
>> # 3. drop mysql.service dependency (since this is a slave backend)
>>
>> [Unit]
>> Description=MythTV Backend
>> Documentation=https://www.mythtv.org/wiki/Mythbackend
>> After=network.target dev-videop2.device dev-videop3.device
>> dev-videop5.device
>> wants=dev-videop2.device dev-videop3.device dev-videop5.device
>
>On my Fedora boxes there is another target `network-online` which, if it 
>exists (should do)
># systemctl list-units | grep network-online
>
>You should swap network.target to network-online.target, which waits 
>longer for the network
>to actually come online.
>
>
>Regards
>Stuart

The network.online target is much less useful than most people seem to
think, unfortunately.  In Ubuntu, you get network.online as soon as
one network device has an IP address.  That means it comes up as soon
as localhost comes up and gets its IP address.  That is a long time
before any real network device such as an Ethernet port has come up
and got an IP address, even if the IP addresses are static.  DHCP IP
addresses take longer.  WiFi devices take even longer.  For a slave
backend, I believe it needs to wait for two things: access to the
remote database, and access to the remote master backend.  Those are
not necessarily on the same machine and accessible via the same
network path, but they usually are.

Both MySQL and MariaDB are very good at starting up and binding their
required network interfaces.  They start up as soon as localhost does,
and after that register themselves for messages from the system about
network interfaces coming up and down.  On each of those messages,
they check the interface that has changed, decide whether they are
using it, and then bind or unbind it.  This method of binding network
interfaces is the gold standard way of doing it, but unfortunately
takes quite a bit of effort by the authors, and MythTV does not do
this.  MythTV simply looks at the network interfaces available at the
time it starts up, and binds any that are available and match what it
wants to bind to.  After startup, MythTV does not check for any
changes to the interfaces, either coming up or going down.  So while
MySQL and MariaDB can start at any time and will still get all their
network interfaces, MythTV requires that it only be started after all
its network interfaces are already available - they must be up and
have all their IP addresses assigned.  So if an interface has both
IPv4 and IPv6 addresses, or has more than one IP or IPv6 address, all
the ones MythTV wants must be up.

At this point, I run out of knowledge about MythTV's requirements for
slave backends as I have never used one.  If the slave backend gets
all its network interfaces at the time it starts up, it may or may not
also require that the master backend already be up and able to respond
to it.  I hope that is not the case and the slave backend will be able
to cope with the master backend going away and coming back again, and
vice versus.

This has been an ongoing problem for MythTV for some time, but has
been aggravated by Ubuntu changing to use systemd as that change has
caused mythbackend to be started much earlier and with the default
systemd .service files for mythbackend it generally only works if only
localhost is being used.  If networked tuners or external frontends
are being used, then the systemd unit needs to be changed so that it
will only start mythbackend once all the network addresses needed by
mythbackend are up and working.  And the networked tuners also need to
be up and working, as mythbackend tests them only at startup and will
not use any tuner that does not respond to that testing.

The most recent solution to this problem is to use another systemd
unit that will wait for the necessary interfaces to be up and the
networked tuners to be available by sending a ping to the networked
tuners, and will keep doing that until it gets a reply.  Only then
will that unit signal that it is up.  The mythbackend unit is changed
to wait on this new systemd unit before it starts.  To make this work,
I wrote a small python program called "wait-until-pingable.py" which
is available from here:

http://www.jsw.gen.nz/mythtv/wait-until-pingable.py

This was discussed in this mailing list thread:

https://lists.gt.net/mythtv/users/621911

The "full-internet.service" file is also available here:

http://www.jsw.gen.nz/mythtv/full-internet.service

Put that in /etc/systemd/system and edit the "<mytuner>" to be the IP
address or host name to wait for.  Run wait-until-pingable.py with the
--help option to see what options it has.  It needs to be run as root,
so if testing it, use sudo.  In the case of slave backends, it may be
that the master backend needs to wait for the slave box to be
pingable, and the slave backend needs to wait for the master box to be
pingable.  If you need to wait for more than one network connection
(eg multiple network tuners or a network tuner and a master backend),
then create multiple "full-internet.service" files (with different
names) and have the mythbackend .service file wait on all of them.
Note that full-internet.service itself waits for
NetworkManager-wait-online.service before it runs.  So you will need
to enable that service if you have not already.  And if you are not
using Network Manager, you will need to use a different target for
starting full-internet.service.

Unfortunately, I have yet to do a version of wait-until-pingable.py
that handles IPv6, so if you are using IPv6 you will need to find
another way of doing what wait-until-pingable.py does.  Let me know if
you need IPv6, as I think I know how to make the necessary changes to
wait-until-pingable.py now, but I just need to find the time to do it.


More information about the mythtv-users mailing list