[mythtv-users] scan for changes

Joe Mythtv joe.mythtv at gmail.com
Mon Mar 26 16:35:16 UTC 2012


I had everything under the same two folders, so it looked like this:

/disk1/import/home_videos <- 5000 files
/disk1/import/music <- ~1000 folders underneath, 15k files
/disk1/import/photos <- 15k files
/disk2/import/home_videos <- 5000 files

I set mythtv to look at /disk1/import for videos. Once I moved the other
media from there, the scan became much faster. I do really like the idea of
being able to kick off the scan without the interface, either with a script
that looks for new files in a scratch folder that holds new stuff or with a
shortcut button on the remote.

Is there a way to consolidate folders with the same name? so that the
interface would show me a single "home_videos" folder that consists of the
files in /disk1/home_videos + /disk2/home_videos ?

On Sun, Mar 25, 2012 at 11:45 PM, Raymond Wagner <raymond at wagnerrp.com>wrote:

> On 3/25/2012 22:51, Joe Mythtv wrote:
>
>> My content is all local to a single backend. It's not striped across an
>> array, it all lives on two 5400rpm spindles. Maybe the issue is volume?
>> It takes me about 35 seconds to rescan 15k pictures, 15k mp3s, and 10k
>> video files.
>>
>
> The Video Library only does video, so unless you've got those 15k pictures
> and 15k mp3s all jumbled into one massive folder, the 10k videos are the
> only ones you need to concern yourself with.  Low spindle speed might be a
> problem with the hashing, but the filesystem metadata lookup for the scan
> itself should be very fast.
>
> How is this system configured? If this is a combo frontend/backend, or you
> are using storage groups on a remote frontend, the file list will be
> produced by scanning on whatever machine contains those files locally. If
> instead you are using the old local folder definitions on a remote
> frontend, you may have significant delays from scanning over NFS/CIFS.
>
>
>  As Mark mentioned, linux caching could be the reason some scans are much
>> faster - if I clear out the cache (by rebooting or with echo 1 >
>> /proc/sys/vm/drop_caches) the scan takes about 5 minutes to finish.
>>
>
> I would expect the speed at which that data could be retrieved would be
> filesystem dependent. At least on FreeBSD using ZFS, I don't see any slow
> scanning, even if I haven't touched those folders since restarting.
> However, ZFS is extremely aggressive in its caching and prefetching.
>
>
>  I wish there were a command line way to initiate the scan - I could run
>>
> > my own script to look at the 'new footage' folder and rescan when I see
> new
>
>> files.
>>
>
> In 0.25, there is an option in 'mythutil' to trigger a scan in the master
> backend. Prior to that, JAMU had a scanner, and there was a scanner built
> into the Python bindings. What I'm getting at is that none of that should
> be necessary, and the interactive scanner should be plenty fast. I'm
> interested in whether there might be some flaw in the scanner that needs to
> be resolved.
>
> How much does memory consumption jump when running a scan? Do you actually
> have all 40k of those files in folders that the Video Library would be
> running through? It's possible there might be some component in the scanner
> with poor scaling efficiency that blows up with such a large file list.
>
> ______________________________**_________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://www.mythtv.org/mailman/**listinfo/mythtv-users<http://www.mythtv.org/mailman/listinfo/mythtv-users>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mythtv.org/pipermail/mythtv-users/attachments/20120326/364cbc7a/attachment.html>


More information about the mythtv-users mailing list