[mythtv-users] Frontend locking up and unable to kill it.

[Allen Edwards]

On Fri, Jul 12, 2019 at 10:51 AM Stephen Worthington <
stephen_agent at jsw.gen.nz> wrote:

> On Fri, 12 Jul 2019 07:32:39 -0700, you wrote
>
> OK, I scanned that thread again and understand what you mean now.
>

Great


> >Regarding shutdown timer:
> >The evidence is that the standard shutdown timer is 90 seconds and I am
> >experiencing about 5 minutes for a shutdown. That suggests that there are
> >multiple things hung up and that they are being forced to shutdown in
> >series. I don't see a particular difference between forcing a hung program
> >to shutdown after 90 seconds or 4 seconds. The question then is there a
> >program that will shutdown gracefully and take more than 4 seconds to do
> >so.  The other evidence I have is that doing CTRL-ALT-BS restarts in less
> >than 4 seconds. That suggests that there is not a program that takes
> longer
> >than 4 seconds to gracefully shutdown and the 4 second timer should be
> fine.
> >
> >Allen
>
> Shutdown and Ctrl-Alt-BS (restart X) are two completely different
> things.  Shutdown will have to stop all the background tasks that run
> on the system.  Restarting X will only affect the GUI programs run
> under X.  For shutdown, mythbackend and MySQL/MariaDB have to be shut
> down.  They are not affected by restarting X.  So Restarting X taking
> only 4 seconds has no bearing on how long shutdown should take.  With
> X, the GUI programs are normally running independently from each
> other, so they do not need to be shut down in any order.  Indeed, I
> have a feeling that shutting down X may kill them without prior
> notification, but I have never researched that.
>
> Shutting down all the background / daemon programs and the system
> itself has to be done in a specific and orderly way unless you want to
> corrupt things.  So systemd works out what is dependent on what, and
> the correct order to shut them down in.  If you have a look at your
> /lib/systemd/system/mythtv-backend.service file, you will find that it
> has this line:
>
> After=mysql.service network.target
>
> That means that at startup, the mythtv-backend.service will not be
> started until mysql.service has been started and network.target has
> been reached (the first network interface is up).  On shutdown,
> systemd reverses that.  It will shut down mythtv-backend.service, and
> will wait for that to happen before it will shut down either
> mysql.service or the network.  If mysql.service was to be shut down
> before mythtv-backend.service, mythbackend might still be trying to
> use the database while it was shutting down, and it would be unable to
> do that, and that might mean that the database would wind up having
> conflicting data in it ie it would be corrupt.  Similarly, if you
> force mythtv-backend.service to be shut down before the mythbackend
> program has properly shut down, you could be causing corrupt data in
> the database.
>
> When you set the DefaultTimeoutStopSec=4s globally, rather than as an
> option for one systemd unit, you are telling systemd that any unit
> that takes longer than 4 seconds to shut itself down is to be killed
> with a kill -9.  So if, for example, a program is waiting for a disk
> to spin up to speed so it can close some files it has on that disk, it
> may never be allowed to do that, and those files may be corrupted.
> Another example is a program that is waiting for a network connection
> to come up again so it can write over it.  This is usually the thing
> that causes the longest shutdown delays.  It is not a good idea to
> force such programs close with kill -9 unless you know that it is safe
> to do so.  And these things depend on exactly what is going on in the
> environment.  If you unplug the Ethernet cable to a PC, it will
> normally cause a slow shutdown as there will be things waiting for the
> network.  And something that is waiting for the network may also be
> waiting until after it has used the network to be able to write
> critical data to disk.  So under normal conditions, it will shut down
> virtually instantly, but being without the Ethernet cable will cause
> data corruption unless it is allowed to timeout its network connection
> and go on after that.  There is a reason that the
> DefaultTimeoutStopSec value defaults to a generous 90 seconds.  So
> when you have slow shutdowns, you need to diagnose the problem, work
> out which programs or systemd units are causing it, and fix each of
> those programs or units individually, to ensure you are not breaking
> something.
>
> If everything is working right, systemd shuts down everything it can
> in parallel.  So any unit that is not dependent on other units will be
> shut down immediately systemd is told to shut down.  It works its way
> back down the tree of dependencies shutting things down as soon as
> nothing else is dependent on them.  That setup of dependencies and how
> it works is one of the things that 18.04 has had optimised versus
> 16.04, so 18.04 is capable of shutting down faster, and it also does
> not seem to do multiple waits for the same thing in the way 16.04
> seems to.
>
> One of the big culprits in slow shutdowns is mythbackend.  Systemd
> sends it only one kill -15 to request it to shut down.  It only partly
> shuts down, and then after the TimeoutStopSec timeout, systemd sends
> it a kill -9.  If mythbackend is given a second kill -15, like
> mythfrontend, it shuts down properly, and quickly.  Fortunately, in
> the case of mythbackend, the first kill -15 partial shutdown seems to
> be enough to ensure that nothing will be corrupted if systemd later
> does a kill -9.  So it appears to be safe to just let that happen, or
> to reduce the timeout before the kill -9 is done by overriding the
> DefaultTimeoutStopSec value with a TimeoutStopSec setting in the
> .service file.  But it is better to workaround the bug by giving
> mythbackend two kill -15 requests with a short time between them, as I
> posted the code for.  I tried giving two kill -15 signals with no
> delay between and that did not work, but it is entirely likely that a
> 1 second delay between the kill 15 signals is overkill, so if you want
> to experiment, you may be able to find a smaller value that will
> always work safely.  The sleep command in Ubuntu takes a floating
> point number for seconds, so you can wait for a fraction of a second,
> but other Linux or Unix versions may not support non-integer values.
>
> Similarly, with your frozen mythfrontend, if it is not completely
> locked up and unkillable as you seem to have seen sometimes, it needs
> two kill -15 signals with a short delay between to get it to shut
> down.  I get the locked up but not completely frozen problem with
> mythfrontend fairly frequently when playing old .wmv files.  And I
> have had it once that I can recall when playing a damaged DVB-T
> recording file.  Each time, I have just gone to a command prompt,
> found the PID for mythfrontend.real and given it two kill -15 signals
> and it has shut down.  I am guessing that mythbackend and
> mythfrontend.real will have the same problem code that is causing
> this, and there is some evidence, from what is in the log files before
> and after the second kill -15, that it relates to how they set up
> their signal handling code.
>
>
I am making several assumptions.  When Mythfrontend becomes non responsive
and hangs it cannot be killed. CTRL-ALT-BS appeared to hang so I did a
shutdown, which took 5 minutes. My assumption was that CTRL-ALT-BS, which I
agree is a restart of X, might have timed out and worked if I gave it
enough time. I also assumed the change in the timer limit would help
CTRL-ALT-BS do it's thing faster. I shall see.

My system will run for years without shutting down under normal
circumstances. With Mythbuntu 8.04 we did an occasional CTRL-ALT-BS when
mythfrontend hung up or disappeared.

I did a test of how long it takes the system to shutdown and it is very
difficult to see.  I can see that all the entries in syslog happen within
one second. I do not know if things continue to happen past the last syslog
entry and the power button takes a long time to go off so not sure my test
is meaningful.

Interestingly, doing CTR-ALT-BS seems to end mythfrontend with a seg fault
so I find that strange.

My plan is to leave the 4 second timer in and see if CTRL-ALT-BS is
responsive next time that the system hangs.

This is where I got the idea
https://askubuntu.com/questions/972460/my-pc-doesnt-shut-down-with-ubuntu-17-10

Allen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-users/attachments/20190712/35e37484/attachment.html>