[mythtv-users] Mythcommflag hanging at midnight

[Stephen Worthington]

On Sat, 5 May 2018 10:27:30 -0400, you wrote:

>I'm running Mythtv 29.0-71 on mythbuntu/Ubuntu 16.04. Combination 
>FE/BE.  6 core processor.  Nvidia GT630.  System is on SSD.  Recordings 
>are on HDD "media" drives in RAID.
>
>I have a recent issue where mythcommflag jobs hang on late-night shows.  
>The job statuses show that they're "running", but they're frozen at a % 
>complete, and a certain frame rate.  Those numbers won't change for 
>days, and the queue will keep stacking up. Here's the tail of a recent 
>mythcommflag.log:
[snip]
>So, every day, at 24 or 25 seconds after midnight, this will happen.  
>And there will be no more logging until I do something.
>
>If I kill the two jobs - in this case 18887 and 18969 - the current 
>(hung) jobs are marked as failed, and mythcommflag happily and 
>successfully goes on to the backlog of queued jobs.  I will then 
>re-queue the hung jobs, and they complete successfully.
>
>On a related note, just based on the final error in that log (Decoder 
>timed out waiting for free video buffers); I had been using vdpau for 
>video playback, and mythfrontend would often randomly freeze, and 
>mythfrontend.log would show that same error. I recently changed to 
>OpenGL/ffmpeg, and the mythfrontend freezing stopped.  That might be 
>about the time this mythcommflag issue started.  Could the playback 
>decoder affect mythcommflag?  In the log, it's obviously mythplayer.cpp 
>showing the error.
>
>I allow 2 simultaneous jobs.  Mythcommflag never goes over 35% of the 
>processing capability.  So, could this not necessarily be a mythcommflag 
>problem, but something possibly hogging io resources at midnight?  I do 
>have a system backup which runs at midnight, but I don't see it in the 
>chron lists, so don't know how to play with that to troubleshoot.  Is 
>there perhaps an "io" log capability?  Is it possible to ionice either a 
>backup script or mythcommflagging?  Running top or iotop during 
>mythcommflagging while watching recordings show low usage of processors 
>and io, but maybe whatever is happening at midnight is just too much.
>
>So, I think I have a couple directions I could go to diagnose this, but 
>I'm thinking someone will have some good insight in where to start!
>
>Thanks.

Does your backup job backup the mythconverg database?  Or run an
optimise on it?  If so, that is likely the problem.  I have had a
similar problem in the past with database backups causing mythcommflag
to lock up, probably because it is timing out on database access and
not handling the problem properly.  Both database backups and
optimising the (huge) recordedseek table cause the other database
accesses to become extremely slow.

I have changed my backup jobs quite a bit from the standard ones,
starting with putting "/usr/bin/nice -n 2" at the start of the
mythconverg_backup.pl command in the cron job.  I also moved my system
onto a super fast NVMe SSD at around the same time, and those changes
seemed to fix the problem for me.  But as I no longer run
mythcommflag, I can not say if it is still fixed.