[mythtv-users] SchedulesDirect timing out during Mythfilldatabase

DeKay dekaymail at gmail.com
Mon Apr 18 02:10:41 UTC 2011


On Sun, Apr 17, 2011 at 1:42 PM, Michael T. Dean
<mtdean at thirdcontact.com> wrote:
> I meant to use XMLTV to test the download, not as a replacement for
> mythfilldatabase's downloading.
>
> Again, please try:
>
> mythfilldatabase --dd-grab-all

Same result.  As I mentioned in my last email, the slow database
insertions compared to the download rate introduce long pauses that
cause timeouts during wget.  --dd-grab-all doesn't help at all.  All
of this is because myth doesn't work like I first thought.  It doesn't
grab the whole file and then do the database thing.  Instead, it pipes
the .gz compressed file from wget through gzip and inserts the data
into the database in parallel with the download (do "ps -ef | grep
wget" in one window while doing a "tail -f mythfilldatabase.log" in
another after running "mythfilldatabase -v all > mythfilldatabase.log"
to see this).  --dd-grab-all grabs a much bigger file than the normal
route, so the chance of hitting a timeout is high or higher than the
normal route.

> and read the posts I linked about it, earlier--since it will likely fix
> any problem you may have due to drive seeking/swap thrashing/... that's
> causing problems when mysql and mythfilldatabase are both trying to hit
> the disk.

The disk I/O is all mysql and the insertions it is doing.  The file
download is insignificant I/O wise according to iotop.

BTW, iotop show mysql is doing about 150 kb/s of I/O as it fills the
database during the lineup.  I have no idea if this is fast or slow.
My grabber script combo is taking a little over an hour now to
download, stuff the database, and clean up.  I suspect this is still
slow compared to other folks.  But my suspicion as to the cause is
noted below.

> And, more importantly, you'll have better, more-current
> listings data every day if you set up your system to use it.

The grabber script I came up with gives me this now as well.  At least
I'm up and running again.

>
> Getting your database working properly is much better than sweeping the
> problem under the rug by using XMLTV to download the data and then
> hoping mythfilldatabase will be able to insert it without problems (when
> it was having problems before).

I know this.  But I also mentioned that I suspect a bug in my kernel
that is causing ext4 to do excessive journaling that is slowing down
the database insertions.  iotop is showing the ext4 journalling task
being responsible for over 99% of the I/O at times.

Unfortunately I can't fix this until a conflict between a newer kernel
and the nvidia binary blob is sorted.  So the best I can do right now
is work around the problem with a script that first calls the xmltv
grabber and then mythfilldatabase on the resulting file.  Once the
conflict is resolved in ArchLinux, I'll go back to the usual
mythfilldatabase method to see if the database insertion speed
improves.

> If you are having database issues, you may want to try:
>
> wget mysqltuner.pl
>
> then run the script to see if there are any things you can change to
> improve its performance.

Came across this already, thanks.  It likes to have mysql running for
a while before it comes to any conclusions, so I'll let mysql run for
a bit and try it again.


More information about the mythtv-users mailing list