I had everything under the same two folders, so it looked like this:<br><br>/disk1/import/home_videos <- 5000 files<br>/disk1/import/music <- ~1000 folders underneath, 15k files<br>/disk1/import/photos <- 15k files<br>
/disk2/import/home_videos <- 5000 files<br><br>I set mythtv to look at /disk1/import for videos. Once I moved the other media from there, the scan became much faster. I do really like the idea of being able to kick off the scan without the interface, either with a script that looks for new files in a scratch folder that holds new stuff or with a shortcut button on the remote.<br>
<br>Is there a way to consolidate folders with the same name? so that the interface would show me a single "home_videos" folder that consists of the files in /disk1/home_videos + /disk2/home_videos ? <br><br><div class="gmail_quote">
On Sun, Mar 25, 2012 at 11:45 PM, Raymond Wagner <span dir="ltr"><<a href="mailto:raymond@wagnerrp.com">raymond@wagnerrp.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 3/25/2012 22:51, Joe Mythtv wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
My content is all local to a single backend. It's not striped across an<br>
array, it all lives on two 5400rpm spindles. Maybe the issue is volume?<br>
It takes me about 35 seconds to rescan 15k pictures, 15k mp3s, and 10k<br>
video files.<br>
</blockquote>
<br></div>
The Video Library only does video, so unless you've got those 15k pictures and 15k mp3s all jumbled into one massive folder, the 10k videos are the only ones you need to concern yourself with. Low spindle speed might be a problem with the hashing, but the filesystem metadata lookup for the scan itself should be very fast.<br>
<br>
How is this system configured? If this is a combo frontend/backend, or you are using storage groups on a remote frontend, the file list will be produced by scanning on whatever machine contains those files locally. If instead you are using the old local folder definitions on a remote frontend, you may have significant delays from scanning over NFS/CIFS.<div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
As Mark mentioned, linux caching could be the reason some scans are much<br>
faster - if I clear out the cache (by rebooting or with echo 1 ><br>
/proc/sys/vm/drop_caches) the scan takes about 5 minutes to finish.<br>
</blockquote>
<br></div>
I would expect the speed at which that data could be retrieved would be filesystem dependent. At least on FreeBSD using ZFS, I don't see any slow scanning, even if I haven't touched those folders since restarting. However, ZFS is extremely aggressive in its caching and prefetching.<div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I wish there were a command line way to initiate the scan - I could run<br>
</blockquote>
> my own script to look at the 'new footage' folder and rescan when I see new<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
files.<br>
</blockquote>
<br></div>
In 0.25, there is an option in 'mythutil' to trigger a scan in the master backend. Prior to that, JAMU had a scanner, and there was a scanner built into the Python bindings. What I'm getting at is that none of that should be necessary, and the interactive scanner should be plenty fast. I'm interested in whether there might be some flaw in the scanner that needs to be resolved.<br>
<br>
How much does memory consumption jump when running a scan? Do you actually have all 40k of those files in folders that the Video Library would be running through? It's possible there might be some component in the scanner with poor scaling efficiency that blows up with such a large file list.<div class="HOEnZb">
<div class="h5"><br>
______________________________<u></u>_________________<br>
mythtv-users mailing list<br>
<a href="mailto:mythtv-users@mythtv.org" target="_blank">mythtv-users@mythtv.org</a><br>
<a href="http://www.mythtv.org/mailman/listinfo/mythtv-users" target="_blank">http://www.mythtv.org/mailman/<u></u>listinfo/mythtv-users</a><br>
</div></div></blockquote></div><br>