[mythtv-users] Performance issues, or Why Mythtv hates Oprah
sdkovacs
sdkovacs at gmail.com
Thu May 31 00:24:08 UTC 2007
On 3/28/07, Yan-Fa Li <yanfali at gmail.com> wrote:
> On 3/28/07, sdkovacs <sdkovacs at gmail.com> wrote:
> > The Oprah Winfrey show records every day at 4pm EDT, but there is
> > something on my box that is creating performance problems, and when
> > Oprah is choppy, I'm in trouble(WAF). This is my setup:
> >
> > OS: Fedora Core 6, ATRPMS
> > Mythtv: .20-fixes (mythtv-0.20-153.fc6.at)
> > Computer: PIII-800Mhz, 384MB RAM.
> > Capture Cards: PVR-350, PVR-150
> >
> > The symptom in Mythtv is that the recording intermittently skips a few
> > seconds. mythbackend.log has iobound errors and Orca graphs show that
> > the box is obviously stressed. I have mythtv setup to flag commercials
> > immediately upon the start of the recording and I don't have any
> > performance problems at any other time of day whilst recording. The
> > first place I looked was cron(see output below), but there are no jobs
> > running at that time. I'm looking for some tips on how to identify
> > what is stressing my box - maybe a command I can put into cron that
> > will identify the processes that are causing the problem.
> >
> > Thanks in advance for any help.
> >
>
> Check wether it's running slocate. That hog takes up a lot of memory
> and forces processes into swap. Look at your cron.hourly, cron.daily
> directories under /etc. Redhat also tends to run sa, which keeps
> track of IO statistics and what not, which you might consider turning
> off.
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>
Ok, here's a bit of an update. I tracked the IOBOUND errors on
mythbackend.log and found that this problem only occurs every 7 days.
Oprah records every day at the same time, but I only get IOBOUND
errors every 7 days. Strange. I setup a script to track some
preformance statistics during that time, so I will post my script and
then the output of the results. Sorry if it's verbose, but I'm hoping
someone will catch something that I don't see.
#! /bin/bash
#
# get performance statistics
#
###########################################
PATH=${PATH}:/usr/sbin:/sbin
export PATH
echo "Started script at `date`"
echo "----------------------------------------"
echo "Running iostat -xt 3 5"
/usr/bin/iostat -xt 3 5
echo "Running vmstat 3 5"
/usr/bin/vmstat 3 5
echo "Running top"
export CPULOOP=1
top -b -d 3 -n 3
echo "Running fuser -vm /dev/hdb"
fuser -vm /dev/hdb
echo "Running lsof /videos"
lsof /videos
Running iostat -xt 3 5
Linux 2.6.19-1.2911.fc6 (localhost.localdomain) 05/30/07
Time: 16:00:01
avg-cpu: %user %nice %system %iowait %steal %idle
1.34 6.27 2.29 2.48 0.00 87.63
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
hda 0.29 1.34 1.02 0.66 23.57 16.07 23.66
0.04 21.20 3.46 0.58
hdb 0.04 0.18 2.49 1.69 394.01 339.95 175.35
0.03 8.03 5.05 2.11
hdc 0.00 0.00 0.00 0.00 0.00 0.00 16.94
0.00 459.00 451.76 0.00
Time: 16:00:04
avg-cpu: %user %nice %system %iowait %steal %idle
2.00 0.00 4.00 94.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
hda 0.33 17.33 13.00 7.00 765.33 200.00 48.27
3.05 127.52 50.00 100.00
hdb 0.00 0.00 0.00 1.33 0.00 9.00 6.75
3.76 996.25 686.50 91.53
hdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
Time: 16:00:07
avg-cpu: %user %nice %system %iowait %steal %idle
1.67 0.00 3.00 95.33 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
hda 0.00 0.00 8.67 0.00 309.33 0.00 35.69
2.88 312.58 114.69 99.40
hdb 0.00 0.00 0.00 1.33 0.00 360.33 270.25
3.68 2746.75 750.00 100.00
hdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
Time: 16:00:10
avg-cpu: %user %nice %system %iowait %steal %idle
1.33 0.00 3.00 95.67 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
hda 0.00 0.67 5.00 0.33 352.00 2.67 66.50
3.08 683.88 187.56 100.03
hdb 0.00 0.00 0.00 1.33 0.00 1365.33 1024.00
4.02 2764.75 750.50 100.07
hdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
Time: 16:00:13
avg-cpu: %user %nice %system %iowait %steal %idle
1.33 0.00 2.99 95.68 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
hda 0.00 1.33 6.98 1.99 140.86 31.89 19.26
2.80 322.48 111.15 99.70
hdb 0.00 0.00 0.00 1.99 0.00 773.09 387.83
4.42 2251.17 500.17 99.70
hdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
Running vmstat 3 5
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 2 163900 5012 2976 134588 1 1 209 177 69 94 8 2 88 2 0
0 2 163900 4996 3028 134532 0 0 23 0 1468 748 3 3 0 94 0
0 1 163900 5408 3068 133664 0 0 239 1576 1485 723 1 4 0 94 0
0 2 163900 4572 3084 134424 0 0 0 283 1474 709 2 3 0 96 0
0 2 163900 5116 3096 133652 0 0 8 19 1461 709 1 3 0 96 0
Running top
top - 16:00:32 up 3 days, 9:52, 1 user, load average: 2.11, 0.67, 0.23
Tasks: 100 total, 2 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 1.3%sy, 0.0%ni, 0.0%id, 98.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 384544k total, 379508k used, 5036k free, 3148k buffers
Swap: 524152k total, 163900k used, 360252k free, 133492k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2470 root 15 0 276m 25m 7556 S 1.5 6.9 34:37.13 mythbackend
1487 root 15 0 0 0 0 S 1.0 0.0 48:43.29 lirc_dev
32653 root 15 0 2160 980 780 R 0.2 0.3 0:00.03 top
1 root 15 0 2032 604 548 S 0.0 0.2 0:01.17 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.15 events/0
6 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 khelper
7 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
53 root 10 -5 0 0 0 S 0.0 0.0 0:00.15 kblockd/0
54 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
128 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0
129 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ksuspend_usbd
132 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khubd
134 root 19 -5 0 0 0 S 0.0 0.0 0:00.01 kseriod
152 root 15 0 0 0 0 S 0.0 0.0 0:04.34 pdflush
153 root 15 0 0 0 0 S 0.0 0.0 0:04.00 pdflush
154 root 10 -5 0 0 0 S 0.0 0.0 0:59.16 kswapd0
155 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
318 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
341 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 ata/0
342 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 ata_aux
345 root 10 -5 0 0 0 D 0.0 0.0 0:04.77 kjournald
373 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kauditd
407 root 15 -4 2384 416 368 S 0.0 0.1 0:01.09 udevd
701 root 10 -5 0 0 0 S 0.0 0.0 1:12.82 ivtv_vbi/0
702 root 14 -5 0 0 0 S 0.0 0.0 0:00.01 ivtv_yuv/0
710 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 kgameportd
1175 root 10 -5 0 0 0 S 0.0 0.0 1:58.91 msp34xx
1199 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 ivtv_vbi/0
1200 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 ivtv_yuv/0
1495 root 15 0 0 0 0 S 0.0 0.0 0:00.05 lirc_dev
1628 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/0
1634 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 kmirrord
1657 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 kjournald
1663 root 10 -5 0 0 0 S 0.0 0.0 0:05.36 xfslogd/0
1664 root 10 -5 0 0 0 S 0.0 0.0 0:13.87 xfsdatad/0
1665 root 10 -5 0 0 0 S 0.0 0.0 0:07.38 xfsbufd
1666 root 11 -5 0 0 0 S 0.0 0.0 0:00.08 xfssyncd
2065 root 16 0 2276 528 408 S 0.0 0.1 0:00.00 dhclient
2152 root 18 0 1696 528 480 S 0.0 0.1 0:01.83 syslogd
2155 root 15 0 1640 372 328 S 0.0 0.1 0:00.49 klogd
2200 dbus 17 0 2708 392 316 S 0.0 0.1 0:00.00 dbus-daemon
2238 root 25 0 6260 728 612 S 0.0 0.2 0:00.46 automount
2257 root 16 0 1640 448 444 S 0.0 0.1 0:00.00 acpid
2271 root 17 0 5176 596 500 S 0.0 0.2 0:00.02 sshd
2285 ntp 15 0 4128 992 868 S 0.0 0.3 0:00.43 ntpd
2348 root 25 0 4488 1020 1016 S 0.0 0.3 0:00.03 mysqld_safe
2384 mysql 18 0 163m 24m 3248 S 0.0 6.4 8:13.18 mysqld
2423 root 15 0 8556 792 596 S 0.0 0.2 0:00.32 sendmail
2431 smmsp 15 0 7544 636 512 S 0.0 0.2 0:00.02 sendmail
2443 root 18 0 1868 292 268 S 0.0 0.1 0:00.13 gpm
2455 root 18 0 26792 1512 1488 S 0.0 0.4 0:01.70 httpd
2482 root 15 0 5224 624 560 S 0.0 0.2 0:00.61 crond
2493 root 15 0 3016 376 332 S 0.0 0.1 0:00.16 lircd
2532 xfs 15 0 4348 936 808 S 0.0 0.2 0:00.59 xfs
2558 root 15 0 10184 2508 1252 S 0.0 0.7 0:04.12 fail2ban
2585 root 18 0 13456 700 688 S 0.0 0.2 0:00.02 smbd
2588 root 18 0 13456 164 152 S 0.0 0.0 0:00.00 smbd
2589 root 15 0 8744 936 800 S 0.0 0.2 0:00.56 nmbd
2615 haldaemo 17 0 5492 1992 1176 S 0.0 0.5 0:05.57 hald
2616 root 18 0 2980 788 692 S 0.0 0.2 0:00.02 hald-runner
2622 haldaemo 20 0 1968 520 516 S 0.0 0.1 0:00.00 hald-addon-acpi
2643 root 16 0 1920 620 544 S 0.0 0.2 0:20.87 hald-addon-stor
2663 nobody 15 0 1808 668 580 S 0.0 0.2 0:00.10 noip2
2666 root 34 19 40504 1512 1488 S 0.0 0.4 0:30.41 mtd
2669 root 18 0 6840 1432 788 S 0.0 0.4 0:19.63 procallator
2681 root 16 0 1952 624 384 S 0.0 0.2 0:01.00 smartd
2693 root 18 0 1628 368 364 S 0.0 0.1 0:00.02 mingetty
2696 root 19 0 1628 368 364 S 0.0 0.1 0:00.00 mingetty
2697 root 20 0 1628 368 364 S 0.0 0.1 0:00.00 mingetty
2698 root 21 0 1624 388 384 S 0.0 0.1 0:00.00 mingetty
2699 root 22 0 1628 368 364 S 0.0 0.1 0:00.00 mingetty
2700 root 23 0 1628 368 364 S 0.0 0.1 0:00.00 mingetty
3393 apache 15 0 31344 4036 2120 S 0.0 1.0 0:00.86 httpd
3394 apache 17 0 32260 2728 2516 S 0.0 0.7 0:01.44 httpd
16943 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 cifsoplockd
16944 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 cifsdnotifyd
17084 root 18 0 13896 2576 1860 S 0.0 0.7 5:23.41 smbd
19156 apache 16 0 31932 4756 2544 S 0.0 1.2 0:02.28 httpd
19157 apache 15 0 32108 2812 2224 S 0.0 0.7 0:01.52 httpd
19158 apache 17 0 31560 3536 2540 S 0.0 0.9 0:02.01 httpd
19266 apache 17 0 32532 2648 2212 S 0.0 0.7 0:01.74 httpd
21975 root 23 0 4484 996 992 S 0.0 0.3 0:00.00 prefdm
21978 root 15 0 15304 1716 1712 S 0.0 0.4 0:00.11 gdm-binary
22009 root 16 0 15624 1728 1688 S 0.0 0.4 0:00.02 gdm-binary
22013 root 17 0 132m 76m 3340 S 0.0 20.3 3:40.68 Xorg
22033 mythtv 15 0 229m 68m 17m S 0.0 18.3 17:10.88 mythfrontend
22089 mythtv 18 0 4284 324 224 S 0.0 0.1 0:00.17 ssh-agent
22092 mythtv 24 0 2740 492 408 S 0.0 0.1 0:00.00 dbus-launch
22093 mythtv 25 0 2712 240 160 S 0.0 0.1 0:00.00 dbus-daemon
22094 mythtv 15 0 3056 1160 952 S 0.0 0.3 0:00.09 ratpoison
28701 root 16 0 8000 1984 1684 S 0.0 0.5 0:01.37 sshd
28703 mythtv 15 0 8140 1424 1056 S 0.0 0.4 0:02.12 sshd
28704 mythtv 15 0 4624 1340 1056 S 0.0 0.3 0:00.10 bash
32634 root 32 17 48744 9.9m 7444 R 0.0 2.6 0:00.15 mythcommflag
32637 root 20 0 5752 984 740 S 0.0 0.3 0:00.00 crond
32640 root 20 0 2364 880 768 S 0.0 0.2 0:00.00 sh
32641 root 18 0 2368 928 812 S 0.0 0.2 0:00.00 get_perf
Running fuser -vm /dev/hdb
2470 2666 32634
Running lsof /videos
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
mythbacke 2470 root 11w REG 3,64 0 536871049
/videos/record/nfslockfile.lock
mythbacke 2470 root 17w REG 3,64 54413312 536871568
/videos/record/1021_20070530155900.mpg
mtd 2666 root 7w REG 3,64 173 941566553
/videos/vids/temp/mtd.log
mythcommf 32634 root 5r REG 3,64 54413312 536871568
/videos/record/1021_20070530155900.mpg
More information about the mythtv-users
mailing list