[mythtv-users] scan for changes

[Raymond Wagner]

On 3/25/2012 22:51, Joe Mythtv wrote:
> My content is all local to a single backend. It's not striped across an
> array, it all lives on two 5400rpm spindles. Maybe the issue is volume?
> It takes me about 35 seconds to rescan 15k pictures, 15k mp3s, and 10k
> video files.

The Video Library only does video, so unless you've got those 15k 
pictures and 15k mp3s all jumbled into one massive folder, the 10k 
videos are the only ones you need to concern yourself with.  Low spindle 
speed might be a problem with the hashing, but the filesystem metadata 
lookup for the scan itself should be very fast.

How is this system configured? If this is a combo frontend/backend, or 
you are using storage groups on a remote frontend, the file list will be 
produced by scanning on whatever machine contains those files locally. 
If instead you are using the old local folder definitions on a remote 
frontend, you may have significant delays from scanning over NFS/CIFS.

> As Mark mentioned, linux caching could be the reason some scans are much
> faster - if I clear out the cache (by rebooting or with echo 1 >
> /proc/sys/vm/drop_caches) the scan takes about 5 minutes to finish.

I would expect the speed at which that data could be retrieved would be 
filesystem dependent. At least on FreeBSD using ZFS, I don't see any 
slow scanning, even if I haven't touched those folders since restarting. 
However, ZFS is extremely aggressive in its caching and prefetching.

> I wish there were a command line way to initiate the scan - I could run
 > my own script to look at the 'new footage' folder and rescan when I 
see new
> files.

In 0.25, there is an option in 'mythutil' to trigger a scan in the 
master backend. Prior to that, JAMU had a scanner, and there was a 
scanner built into the Python bindings. What I'm getting at is that none 
of that should be necessary, and the interactive scanner should be 
plenty fast. I'm interested in whether there might be some flaw in the 
scanner that needs to be resolved.

How much does memory consumption jump when running a scan? Do you 
actually have all 40k of those files in folders that the Video Library 
would be running through? It's possible there might be some component in 
the scanner with poor scaling efficiency that blows up with such a large 
file list.