[mythtv] Expiry problem Ticket #10704

[Ian Dall]

I have pretty much figured out what is going on and have filed Ticket
#10704. I have a tentative solution I'm currently testing.

This is a big which would affect all recordings made on a slave backend.

On Sat, 2012-04-28 at 13:00 +0930, Ian Dall wrote:
> On Mon, 2012-04-23 at 21:50 -0400, Michael T. Dean wrote:
> > On 04/23/2012 06:55 PM, Ian Dall wrote:
> > > On Sun, 2012-04-22 at 18:37 -0400, Michael T. Dean wrote:
> > >> On 04/22/2012 11:26 AM, Ian Dall wrote:
> > >>> On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote:
> > >>>> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote:
> > >>>>> On 04/17/2012 07:10 PM, Ian Dall wrote:
> > >>>>>> I have been running a recent 0.25 pre-release(*) and have run into
> > >>>>>> problems with expiry. What happens is that the disk fills up, the expiry
> > >>>>>> thread runs, deletes the entry from the database, but does not delete
> > >>>>>> the actual files containing the recording. Since the disk is still full,
> > >>>>>> it goes ahead and deletes the next show and in a short space of time
> > >>>>>> there is a full disk, and all programs which allow auto expire have been
> > >>>>>> deleted from the DB. I recover the DB from backup and manually delete
> > >>>>>> files which have been recorded since the backup, but obviously it is
> > >>>>>> annoying to have to do this.
> > >>>>>>
> > >>>>>> Possibly relevant is that a) I have a separate master and slave backend
> > >>>>>> and b) am using NFS for the storage. The logs don't show anything
> > >>>>>> interesting. Mythtv *thinks* it is doing the right thing. There are
> > >>>>>> messages about expiring recordings, but nothing to do with failing to
> > >>>>>> delete a file.
> > >>>>>>
> > >>>>>> I'd appreciate any clues about what could cause this or how to debug it.
> > >>>>>>
> > >>>>> Are the expired shows in the Deleted recording group?  Watch Recordings,
> > >>>>> MENU|Change Group Filter, then select "Deleted" and see if any that the
> > >>>>> backend logs say were expired are there.
> > >>>> [iptv problem]
> > > Unfortunately it seems that the iptv problem is not the cause of the
> > > fail to delete option.
> > >
> > > So, the disk filled up again, and yes, it ended up with about 90
> > > recordings in the "Deleted" recording group. Restarting the master
> > > backend causes all the files to be deleted. I notice in the logs that
> > > there are successful regular deletions, (signalled by "About to delete
> > > file" messages) up until about 2:30pm then nothing until after the
> > > backend has been restarted at 11pm (after the disk space is exhausted),
> > > when all the files in the "Deleted" recording group get deleted.
> > >
> > > I notice that the delete thread does check that the file has gone (and
> > > would produce an error message if it isn't) but I don't see any of these
> > > error messages.
> > >
> > > It looks like the DeleteThread is simply not running. A check with gdb
> > > doesn't seem to show any deadlocked threads, but it is not always easy
> > > to easy to identify which thread is which.
> > 
> > It sounds to me like MythTV starts an iptv recording and "stops" it, but 
> > the recording keeps going without MythTV's "conscious" knowledge.  These 
> > recordings fill up the file system, and MythTV begins to expire 
> > recordings, but because they're still open/in use, they don't actually 
> > get removed and, perhaps, cause the delete thread to hang.  Restarting 
> > mythbackend closes all open files and means that next time you restart 
> > it will resume deleting the files it had in the queue, which are no 
> > longer open, and will get removed.
> 
> As noted, I have ruled out iptv by deleting the tuner.
> 
> I have also been capturing the network traffic between master and slave.
> I have also set gdb breakpoints in the slave, and /still/ I don't
> understand what is going on ;-(
> 
> What I can say for sure is that the slave gets a DELETE_RECORDING
> command. The recording is, at this stage in the "Default" recording
> group so MainServer::DoHandleDeleteRecording (mainserver.cpp:2468)
> figures that "justexpire" is true, changes the recording group to
> "Deleted" and returns. The code to start the DeleteThread is after the
> return and is never executed (on the slave at least).
> 
> I'm having trouble figuring out which threads are supposed to persist,
> which get started on demand and which get started from timers.
> Particularly in the case of the DeleteThread, how is it supposed to be
> started? I also am having trouble figuring out why delete involves the
> slave at all. The recording is available to the master so why doesn't
> the master just delete it?
> 
> Regards,
> Ian
> 
> 

-- 
Ian Dall <ian at beware.dropbear.id.au>