[mythtv-users] Two files after transcoding
Michael T. Dean
mtdean at thirdcontact.com
Tue Mar 18 14:25:25 UTC 2014
On 03/17/2014 04:42 PM, Mark Perkins wrote:
>> The successive runs that still showed orphans were simply showing files that the backend had not yet finished deleting. Had you just waited, they would have disappeared, eventually, with only the one run.
> Mike is making quite an important point here that took me few goes to understand when I first used find_orphans.py. (IIUC) find_orphans.py still uses the slow delete function that is built into Mythtv, so even though the script returns back to the mini-menu or appears complete the delete process is just starting and could still run for hours if there are a lot if files.
Yes, and that's an important clarification, too--it could take hours
(and many, many of them), depending on your system configuration and the
size of the files being deleted.
> On my first go I just assumed it would be instantaneous like 'rm xxxx_xxxxxxxxx.mpg'.
> I'm not sure but I assume that it would be safe for the system to shutdown and the delete would just continue at next restart?
If you have slow deletes enabled, shutting down the system would result
in all the deletes completing "instantaneously" (read "as quickly as
your file system can handle"--which may mean several seconds to many
minutes, depending on size of files being deleted, the file system, and
the file system options in use). Generally in this situation, system
start/stop scripts will time out waiting for the kernel's process
cleanup to complete (note that mythbackend is shut down already, and
we're simply waiting on the I/O of the deletion process to complete, so
it's all in the kernel's hands, now) and report an error shutting down
mythbackend. If you wait, patiently (more so than your start/stop
script) until the process completes shutdown (when it's no longer
visible with ps -efw | grep mythbackend), then all is well, and you can
restart mythbackend without issues.
If, instead, you kill -KILL <pid> (or do the same thing, but
incorrectly, with kill -9 <pid>), you may be creating inconsistencies in
your file system. Similarly, this could occur if you shut down the
system (or, worse, hard power it off) before the "as fast as your file
system allows" deletion occurs (depending on your system stop scripts
and how they deal with a slow-exiting process). In theory, a good
journaling file system will ensure the deletions complete properly, even
if you do any of the above, but at this point, it's completely out of
MythTV's hands (from the point you attempted to shut down mythbackend in
the first place).
Note, too, that until the process cleanup completes, system start/stop
scripts could have issues restarting mythbackend. These issues could
result in either error messages from the start script (which would be
good) or "smart" start scripts could do things to try to work around
issues when they notice the old process is still there--possibly even
issuing a kill -KILL.
And, finally, if you do not have slow deletes enabled and you delete a
copious amount of data all at once through mythbackend then "quickly"
shut down mythbackend, it may result in the files not being deleted. In
the case of orphaned files that were being deleted, you'd be no worse
off than you were before. In the case of actual recording files (with
valid recording metadata in the database), you could actually get some
"stuck" recordings that are in a limbo between deleted and not
deleted/between orphaned and not orphaned, but still take up gigabytes
of space on the hard drive. My recommendation is actually to enable
slow deletes, regardless of your file system (and how 1337 it is at
deleting large files), just to prevent this race--at least until I have
a workaround in place that would clean up the mess created by shutting
down a "fast deletes" mythbackend at exactly the wrong time after
deleting a huge number of recordings. (Note, though, that this race
condition is pretty hard to trigger, as there's a very small window
during which you'd need to shut down mythbackend and--TTBOMK--it has to
be a "hard" shutdown, with kill -KILL or hard power off.)
More information about the mythtv-users