[mythtv-users] Storage groups behaving unexpectedly with slave backend.

[Michael T. Dean]

On 03/13/2010 02:56 PM, Andre Newman wrote:
> On 13 Mar 2010, at 17:45, Michael T. Dean wrote:
>    
>> On 03/13/2010 08:09 AM, Andre Newman wrote:
>>      
>>> I have a master backend with four tuners and a couple of slave backends with a tuner each and some local storage, all running 2:0.22.0-fixes23670-0ubuntu2 on Unbuntu 9.10, one is 64 bit the rest 32bit but that doesn't seem to affect my problem.
>>>
>>> Yes I've searched this list and the website and the wiki and anything else that google can find for me and I can't see any clues.
>>>
>>>
>>> I have three drives on the master backend each configured as a separate storage group in default, database and everything non mythtv is on a separate raid 1.
>>> Slave BE 1 has two drives each as a separate storage group and used to have the three master backend drives mounted on the same paths as the master over nfs.
>>> Slave BE 2 has 1 drive and used to have the three master backend drives mounted on the same paths as the master over nfs.
>>>
>>> It seems since disabling the nfs mounts I can no longer play any recordings that were made by the slave be's onto those mounts! I can see the recordings on the master backend, the files are present and MythWeb can find them to let me download them! A frontend on either the master backend or on any other frontend reports that it can't find the file!
>>>
>>> I realise I've messed the system around by removing the nfs mounts but I thought that MythTV searched for files in any storage directory, I've read many stories of people moving files between storage directories and myself I've moved files from remote backends to the master with impunity to make space on the slave disks and have recordings available when the slave is off.
>>>
>>> So is there some sort of cache somewhere that's telling MythTV to look for those recordings only on the Slave BE via the nfs mount and not to look locally? If so how do I flush this cache? I've tried removing the now unused directories from the slave be's and restarting all the backends, master last. If it's relevant I have thumbs for these "missing" shows in MythWeb and in the frontend.
>>>
>>> Before anyone says "just restore the nfs mounts" I can't right now, I'm running a 2.6.33 ppa kernel to try to resolve a sata driver problem that's been dropping disks and corrupting files on the master backend ever since the upgrade to 9.10 (so far so good) and that's completely broken nfs also I never got good results with recording over nfs (although that may have been the sata driver problem) so I'd rather not have to re-instate it.
>>>
>>> Any ideas gratefully received.
>>>
>>>        
>> To know for sure, you would need to list all the directory paths and the hosts for which they're configured (all the information you entered in Storage Directories in mythtv-setup).
>>
>> However, my best guess is that you (like 99.999999999999999% of users) didn't understand how Storage Groups work and configured them in such a way as to cause issues when you change the layout.
>>      
> I did spend some time trying to understand the "right way" to configure it but failed and went with what seemed to be the consensus method on the list, evidently wrong :-(
>    

Well, yours wasn't wrong--it just made the configuration you used (with 
NFS mounts) "mandatory for forever".

>> Every storage group directory defined on the master backend is inherited by *all* remote backends.  Therefore, you defined some directories on the master backend as part of the Default storage group (say, /srv/mythtv/recordings/mbe/1 through 3), then you NFS mounted those directories on the remote backends.  Then you defined some directories as part of the Default storage group for the remote backends (say, /srv/mythtv/recordings/sbe1/1 and 2 and /srv/mythtv/recordings/sbe2/1).  That meant that sbe1 had (and may have used for recording) the directory list:
>> /srv/mythtv/recordings/mbe/1
>> /srv/mythtv/recordings/mbe/2
>> /srv/mythtv/recordings/mbe/3
>> /srv/mythtv/recordings/sbe1/1
>> /srv/mythtv/recordings/sbe1/2
>>      
> That's about the size and shape of it, how did you guess ;-)
>    
>> and that sbe2 had (and may have used for recording) the directory list:
>> /srv/mythtv/recordings/mbe/1
>> /srv/mythtv/recordings/mbe/2
>> /srv/mythtv/recordings/mbe/3
>> /srv/mythtv/recordings/sbe2/1
>>
>> So, any recordings the remote backends made to the /srv/mythtv/recordings/mbe/* directories is now unavailable to the recording host when you unmount the NFS mounts.  Now, sbe1 only has available the directories /srv/mythtv/recordings/sbe1/1 and 2 and sbe2 only has available /srv/mythtv/recordings/sbe2/1 .
>>
>> I.e. it's as if you moved the recording file without telling the backend you did so, so it's expecting the wrong host to have that file.
>>      
> Spot on, which I _would_ expect to be a problem, except for the many comments about backends searching all local groups and my experience of being able to move recordings around, evidently with limitations that I'm now discovering!
>    

Right, you can move recordings around however you like, as long as 
they're still locally available to the recording host.

> I had mistakenly thought that setting "Master Backend Override" would make the mbe find all those files locally but it seems it would have done if only I hadn't messed up the storage group configuration.
>    

"Master Backend Override" in mythtv-setup (and it's cousin, "Always 
stream recordings from the backend" in mythfrontend settings) are also 
commonly misunderstood.  The effects of various combinations are always 
confusing.  If you're really interested, feel free to read through 
http://www.gossamer-threads.com/lists/mythtv/users/370317#370317 (note 
that we've now fixed SG inheritance to work properly even for 
non-Default Storage Groups), but it's probably not that important (as it 
boils down to, "if a recording is no longer available to the recording 
host, you need to edit its recording host in the DB").

>> This would definitely have happened if you did not have local storage on the remote backends originally (and, so, all recordings were made to the file server through NFS).
>>      
> They were at first but that didn't work out well so those recordings are all deleted, evidently some remain from an intermediate stage of workingness
>>   If you had local and NFS-mounted storage on the remote backends, it would be especially likely to happen if you have specified Balanced Free Space for the mythtv-setup setting (in General):
>>      
> I've been using Balanced Disk I/O, I have had big problems with simultaneous HD recordings so have been trying to throw spindles at the problem.
>    

Yes, IMHO, the best approach for stable and reliably MythTV is to have 
more file systems available (and to the right hosts) than you will ever 
have concurrent recordings, and a separate file system per spindle/disk 
only helps to make it more stable and more reliable.

>> Storage Group Disk Scheduler
>> - Balanced Free Space
>> - Balanced Disk I/O
>> - Combination
>> This setting controls how the Storage Group scheduling code will balance new recordings across directories.  'Balanced Free Space' is the recommended method for most users.
>>
>> (see http://www.gossamer-threads.com/lists/mythtv/users/424443#424443 ) where the default for users who created their MythTV databases with post-0.21-fixes r21225 or higher is Balanced Free Space and the default for users who created their MythTV databases before trunk r21225 and upgraded to 0.22-fixes or higher is Combination.  Combination would be the one least likely to result in the problem mentioned above, but it could happen even with Combination or Balanced Disk I/O, depending on configuration.
>>      
> Mine's an awful lot older than that!
>    
>> To fix it, you'll have to change the hostname for all the recordings that were recorded by remote backends onto the master backend file systems to the master backend hostname.
>>      
> Ok, I know how to find out how to do that, after backing up said database.
>
> Thanks for taking the time on a 99.999999999999999% faq or is a fmq frequently misunderstood question?
>
> So..... to make it all good going forward (assuming I get reliable nfs when my sata problem gets fixed) I should have:
>
> All mbe storage group directories nfs mounted on all slaves with identical paths but not explicitly configured in the slave be because they will be inherited and therefore understood to be the same. Only sbe local storage configured locally.
>    

Whether you configure the directory paths that are NFS-mounted to the 
remote backends locally or only on the master backend, they're still 
usable by all remote backends for recording.  So, basically, any 
directory in your master backend storage group definition is always 
considered for use by all remote backends--whether you've configured the 
directory in the remote backend Storage Group definition or not.

Basically, it's the switching from NFS-mounted configuration to the 
no-NFS-mounts configuration that caused the issue.  With the NFS mounts 
recordings were recorded by the remote backend (and, therefore, got the 
remote backend's hostname) to the NFS-mounted file system.  Since after 
removing the NFS mounts, the file system is no longer available to the 
remote backend, the recording can't be found.

If the issue is what I suspect (bad hostname on some recordings), all 
your new recordings should work fine since the remote backends can now 
only write to the directories they use.

Note, though, if you still have the mount points from your old NFS file 
systems (i.e. /srv/mythtv/recordings/mbe/1 through 3 in my example, 
above) on your remote backend, it will still attempt to use those 
directories for recordings.  If the file system (likely the root file 
system) doesn't have room, it will cause problems.

Basically, make sure you back up the DB, then edit the hostname for each 
recording whose hostname is "wrong" for the purposes of file access, and 
you should be good from now on.

Mike