[mythtv-users] Heavy load troubles

Tue Dec 23 21:55:38 EST 2003

This is a long but detailed message, so stay with me...

I've been using mythtv since last Jan on an older machine,
a dual P2-450, 512MB, w/ 3x80MB/sec scsi 10k disks in software raid0, 
with very good success.  The machine had 3
tuner cards, a PVR-250 and 2 WinTV 401 software cards.  Since
the old machine was rather slow in the cpus, I could only set
the software cards to 640x240 at 200 RTjpeg, the pvr @ 480x480.
I could record all 3 cards, and watch a software recorded show
at the same time.  The last myth version I had was
mid August (pre .11) and an ivtv before that.
All was happy and stable, tho low quality viewing.  If I
tried to push things too hard, I'd get typical broken motion/audio
with pauses, but still stable.  I routinely ran the machine
with both cpus pegged at 90-100% for long periods with no jumps/skips
or errors.

Since Thanksgiving, I built up a new machine, a single 3Ghz xeon,
1GB ram, the same scsi matched 10K disks (now 4) in raid0, Nvidia FX5600,
and the same 3 tuner cards.  By all reasonable comparisons, the new
machine is many times the machine as the old.  I intentionally
overbuilt to handle a pcHDTV card, which is currently installed
in a test machine until this get more stable.

Over the past month, I have seen almost all the following symptoms
starting with Myth .12+ cvs, .13 release, back to .12 release,
and now .13+ cvs from Dec 17.  I've upgraded several ivtv versions
and am now using the 1125a release recommended for .13.  The machine
is running Mandrake 9.2 (2.4.22mdk_smp) with current patches, and is 
dedicated to myth so there are no other jobs running.

Myth is configured with the pvr as the first card, and the 401s
as cards 2 and 3.

Using only 2 cards, things are basically stable.  The troubles
below start when the 3rd card is fired up...

1.  With all 3 cards recording, the PVR recording will 
frequently stop at some random time into the recording; say 6
minutes into a 30 minute show.  The other recordings continue.
The interesting thing is many times the pvr card
recording before it stops, the audio will start getting
progressively ahead (yes ahead) of the video, perhaps
1 second.  It doesn't seem to help if I reduce the
recording quality settings.  There is no error messages
in syslog or the backend that correspond to where it stops.

2.  At the end of the recording period above, the backend
will almost always miss the next recording program completely,
if its 1,2, or 3 shows.  It just never starts to record.
If I restart the backend, it will then pick up recording 
where it should until the next 3-way.  The backend log
shows:
2003-12-23 18:00:00 Changing from RecordingOnly to None
2003-12-23 18:00:00 Changing from RecordingOnly to None
2003-12-23 18:00:00 Changing from RecordingOnly to None
2003-12-23 18:00:01 Found changes in the todo list.
But there should be more shows at 18:00 to record, no startup.
There have been other times when it misses a recording,
but never so consistantly as this.

3.  The software encoded recordings can have occasional 
audio/video stutters or repeats.  The stutter is in the recording,
not in playback because if I rewind it stutters every time
at the same point.  It affects both cards at the same time.
There are 2 related log messages:
/var/log/syslog shows:
Dec 23 17:43:26 silver kernel: bttv1: irq: SCERR risc_count=36df2008
Dec 23 17:45:53 silver kernel: bttv1: irq: SCERR risc_count=36df2008
Dec 23 17:45:53 silver kernel: bttv1: aiee: error loops
Dec 23 17:45:53 silver kernel: bttv0: irq: SCERR risc_count=36df2808
Dec 23 17:45:53 silver kernel: bttv1: resetting chip
Dec 23 17:45:53 silver kernel: bttv1: PLL can sleep, using XTAL (28636363).

and backend log shows:
VIDIOCSYNC: Input/output error
VIDIOCSYNC: Input/output error
VIDIOCSYNC: Input/output error
VIDIOCSYNC: Input/output error

So far, I have not confirmed any firm correlation between this 
problem and number 1 by time, other than its typical with 3 cards.
ie:  tonight the pvr stopped at 17:32 but nothing like these messages
at that time.

4.  Looking at "top" and xosview while running the 3 shows and playing
one of them, I notice a very strange cpu load behavior, its not
consistant, not even close.  Even using my old low-quality settings,
there is an aproximately 4 minute cycle where the cpu load slowly goes from
almost nothing up to 80-100%, followed by a sharp drop back to 5-10%
again.  About 4 minutes later the cycle repeats.  Changing recording quality
(even dramatically, 640x480 mpeg at 4500) doesn't change the behavior, 
only the cycle period and the cpu starting level.  Top shows the backend
and X server taking most of the cpu during the peak.

5.  If the pvr card stops as in #1, the backend is much more
likely to hang trying to delete a show, start a new one, etc
until its restarted.  Like some thread gets hung up and its a
slow cancer from there.

6.  The audio playback may get garbled after playing for many minutes.
Stopping the replaying the same show will correct this.  Its like
there are 2 copies of the same audio being played about 1 second apart.
BTW, the 401s are using btaudio, so this isn't the common audio 
card live vs play problem.  This only happened recently after one
of my pci card shifts.

7.  Sometimes I get the PVR tinny audio that's widely reported
in the ivtv forum.  I haven't tried to apply their patch yet.
This can happen regardless of how many cards are active.

What have I tried so far to improve this???  Not all these at the same
time, but over the last month:
1.  Shuffling the pci slots to balance out the irqs between the devices;
/proc/interrupts currently shows:
  0:   18136571          XT-PIC  timer
  1:       1260          XT-PIC  keyboard
  2:          0          XT-PIC  cascade
  4:       2731          XT-PIC  serial
  5:   21631388          XT-PIC  nvidia
  7:    1211456          XT-PIC  bttv, btaudio
  8:          1          XT-PIC  rtc
  9:    5094648          XT-PIC  bttv, btaudio, ivtv: iTVC15/16 mpg2 encoder 
chip
 11:     116514          XT-PIC  eth0
 12:      39882          XT-PIC  PS/2 Mouse
 14:         32          XT-PIC  ide0
 15:    1106159          XT-PIC  aic7xxx, Intel ICH5

I've got the pvr and 1 401 on irq 9, and haven't
been able to balance it out better yet.  I need 10, but 
can't get anything to use it yet.  I'll ask Asus on this.

2.  Changing recording settings and quality.  I keep dropping and 
dropping.  Tonight I ran all 3 cards in the same settings as my
old P2 machine used to do, and the PVR still quit after just 
2 minutes, and the 401s had 3-4 audio jumps.  And it missed 
the 18:00 shows.  There definately were fewer audio jumps
today at these low settings, but 3-4 in 30 minutes is still
pretty bad.

3.  I normally run the backend as root, to activate the renice
logic of the card reading thread to -10.  
I've even tried changing the source to make that
reniced thread a real-time process (SCHED_RR).  No apparent help.

4.  Turned on and off (mostly off) the cpu Hyperthreading.  I'm always
running a SMP kernel.

I'm stumped...
Brian