[mythtv] [mythtv-commits] Ticket #6726: Patch mythrename.pl to add '--hardlink' option

[Michael T. Dean]

On 07/20/2009 09:34 AM, Scott Mazur wrote:
> On Wed, 15 Jul 2009 12:11:54 -0400, Michael T. Dean wrote
>   
>> On 07/14/2009 11:05 PM, MythTV wrote:
>>     
>>> #6726: Patch mythrename.pl to add '--hardlink' option
>>>
>>>  This is a patch to mythrename.pl to add a '--hardlink' option.  When
>>>  scanning an NFS share over the network with a device such as a Popcorn
>>>  Hour, symbolic linked files will list in the directory contents, but can't
>>>  be played (retrieved) due to the way NFS handles (or doesn't handle)
>>>  symbolic links.  The patch adds a new option '--hardlink' to the
>>>  mythrename.pl script that works exactly the same as the '--link' option,
>>>  only it creates/deletes hard links instead of symbolic links.
>>>       
>> This behavior seems like it would be very dangerous.
>>
>> The first issue with it is that hard links can only be created on 
>> the same device as the source file.  So, if you use multiple 
>> filesystems via Storage Groups (even if they're all physically on 
>> the same host), hard linking to all recordings would be impossible.
>>     
> Yes.  This is a limitation of hardlinks.

Yes, that's what I was trying to say.  It's not a problem with your 
script, but a problem with the idea of using hard links.

>   It's not an option for every case.
>   

Yeah, seems it would only be an option for users with a single 
device/filesystem for all recordings (i.e. those still using LVM/RAID 
multi-disk configurations).  If nothing else, this should be mentioned 
in the --help output.

>> The second--and /much/ larger issue--is that hard links provide /no/ 
>> information about the relationship between source and destination 
>> files.  Therefore, when you delete a recording in Myth, Myth would 
>> be unaware that the hard linked filename exists and would be unable 
>> to delete the additional filename--even if you enable "Follow 
>> symbolic links when deleting files" (as it's /not/ a symbolic link 
>> and can /not/ be followed).  Therefore, someone would have to 
>> manually go through and delete the hard linked filename for the 
>> filesystem space to actually be freed.
>>     
> OK, first of all, if you're using mythrename.pl with a '--link' option (or now
> '--hardlink') you're specifically telling the script to make it's own link to
> the original Myth files.  There's no connection between the linked files and
> Myth what so ever, and there is no implied relation that Myth is responsible.
>  Hardlinked or symlinked, if Myth decides to delete a recording, the link is
> broken and dead.  Myth does not need to know that the file was hardlinked else
> where.  As far as Myth cares, the file is deleted and no longer visible to it
> (even if the physical data still exists under a different directory).  The
> only '/much/ larger' issue here is that Myth deletes the file, the file link
> count goes down to 1 (becomes regular file, no more links) and mythrename.pl
> no ignores it.  Again, it's not myth's job to go chasing down external file
> links.  It never was.
>   

No, but it shouldn't be the user's job, either.  If an app (in this case 
a script) creates data, that app should manage it.

>> Currently, mythrename.pl /only/ deletes "files" in the pretty link 
>> dir if they're symlinks--a safety feature done to minimize the 
>> chance of its deleting content it didn't create.  If changed to 
>> delete all files, it could delete something it shouldn't.
>>     
> Maybe you didn't look at the code closely?

Actually, didn't look at the code at all.  It's not the code I have an 
issue with--it's the functionality.

>   My mythrename.pl patch only
> deletes files that have a link count greater than 1 (hard links).  It can't
> delete something it shouldn't.
>   

OK, so now we have safety, but that means that it will only delete 
hard-linked filenames for recordings *that still exist*.  Therefore, 
when the user watches a show in Myth and deletes it (or when Myth 
autoexpires it or ...), the link count on the hard-linked filename goes 
to 1.  Then, the script will no longer delete it.  That means it will 
leave garbage data on the filesystem (either a multi-gigabyte recording 
file or a <4MB "scrap" after truncation if slow deletes are enabled).

And, speaking of which, since Myth's deleting the original filename will 
have no effect on filesystem usage, creating hard links to the recording 
files completely breaks autoexpire.

Both of these issues would definitely need to be mentioned in the --help 
text, too.

>> And, if nothing else, on some filesystems (like ext3) where deletes 
>> can take a long time, if myth had deleted the original filename 
>> (which would happen in almost no time if you do not enable, "Delete 
>> files slowly," since the file still exists/inodes are still in use), 
>> mythrename.pl's deleting the hard-linked filename would take a long 
>> time (potentially problematic if mythrename.pl is run frequently in 
>> a cron job).  However, if you enable, "Delete files slowly" (as most 
>> ext3 users would), Myth would go through and truncate the file until 
>> it was less than about 4MB before it deletes the original filename.  
>> Therefore, the hard-linked filename would reference the remaining 
>> (less than) 4MB file, which would leave little bits of garbage in 
>> the filesystem.  Even if you checked filesize before deleting, 
>> deleting a non-link file may allow the script to delete files it shouldn't.
>>     
> And how the heck is that any different then creating a symlink?  As I
> previously explained, the script doesn't delete files, it deletes (hard)links
> to files.
>   

OK, see above...

>> And, there's a simple workaround to "the way NFS handles (or doesn't 
>> handle) symbolic links"--simply have the filesystem layout mirrored 
>> on all systems.  And, since that's the recommended approach for 
>>     
> <snip>
>
> OK, maybe you just don't understand the way media boxes like Popcorn hour
> work.

I definitely don't--don't have any of them.

>   These are appliance devices, there's no control over things like
> file systems.  They're not user configurable computers that you can just setup
> anyway you like.  You plug it into your network, and you browse for network
> shares then play the files you find.  If I want to watch Myth recordings on my
> popcorn hour box from an NFS share, the files have to exist, either physically
> or as hard links.  My hardlinks patch isn't for everyone, but I guess that's
> why it's a command line option and not a default setting no?

If your device is really that limited, then just choose a filesystem 
that allows server-side symlink resolution--i.e. CIFS with unix 
extensions disabled--or continue to use your modified version of 
mythrename.pl.  However, I suggest that the change not be made in the 
contrib script distributed with MythTV as it's likely to cause more 
people more problems than it will solve (especially since you can still 
get usable symlinks on any device that allows CIFS shares, and can do so 
without modifying mythrename.pl).

That said, I do appreciate your working to improve Myth and making your 
changes available to the project (as the license says you should).  I 
just don't think the project should choose to use your changes.  
However, I won't be the one making the decision, so the changes may well 
be accepted/incorporated into mythrename.pl.

Thanks,
Mike