[mythtv-users] playback performance regression: how to fix

[Allan Stirling]

Sasha Z wrote:
> On 5/22/06, Guy Paddock <gap7472 at rit.edu> wrote:
>> I digress. For the features MythTV offers, the developers have done a superb
>> job making so much progress so quickly; I don't mean to throw their hard
>> work back in their faces. I just feel that it's at the point where, unless
>> some design changes occur, no more features can get added and no more bugs
>> can be fixed because things are so large and hard to maintain.
>>
>> --Guy Paddock
> 
> I agree completely. There's something to be said for code simplicity
> and leveraging of multiple classes. More substantial documentation of
> linkages, naming conventions, data flow, and general programming dogma
> within myth would be nice.
> 
> --Sasha

Since both of you appear to be programmers, rather than 
debating how best to refactor Myth, which gains you 
relatively little with _massive_ effort, I would suggest 
trying oprofile with both versions and finding out where 
exactly in the code is eating more CPU.

If you're encountering playback problems, bear in mind that 
these may throw off the results of oprofile, since error 
paths are being executed.

What you need to do:

1. If you have a recent AMD or Intel chip (don't know about 
the low-power ones), you need to enable the local APIC - 
Make sure your kernel options include it:

Processor type and features  --->
[*] Local APIC support on uniprocessors (NEW)

You may also need to boot with the 'lapic' kernel parameter. 
This allows access to the processor internal performance 
counters, which is much more accurate than "sampling" the 
running processes through a timer.

2. Enable oprofile support in the kernel. Unless you're 
profiling, this does not add any overhead - But when you 
need it, you probably won't want to reboot :)

Instrumentation Support  --->
[*] Profiling support (EXPERIMENTAL) 
                                <*>   OProfile system 
profiling (EXPERIMENTAL)

3. Install the oprofile user space tools from 
http://oprofile.sourceforge.net/download/
Or through your package manager

4. Get the source for current SVN and a version of Myth that 
you think has reasonable performance. As much as possible, 
configure them the same as each other and your current 
environment. You'll want the "--compile-type=profile" option 
and possibly "--enable-proc-opt" on SVN and similar options 
in the old settings.pro. Compile one with --prefix=/opt. The 
following scripts will allow you to run that version in 
parallel.

tfe:
#!/bin/sh
export MYTHCONFDIR=~/.mythtvtesting
export DISPLAY=:0.0
export LD_LIBRARY_PATH=/opt/lib
export PATH=/opt/bin:$PATH
/opt/bin/mythfrontend $@

tbe:
#!/bin/sh
export MYTHCONFDIR=~/.mythtvtesting
export LD_LIBRARY_PATH=/opt/lib
export PATH=/opt/bin:$PATH
/opt/bin/mythbackend $@

5. Set up parallel databases. First, configure the test 
(lower) version to be identical to your current install. 
Then, do the following:

mysqldump -p mythconverg -u mythtv -p > ~/mythconverg.sql
sed -e "s/mythconverg/mythconvergtest/g" <~/mythconverg.sql 
 >~/mythconvergtest.sql
mysql -p -u root -p <~/mythconvergtest.sql

Remember what you're testing for here - Performance 
regressions. Never change something in one database that you 
don't mirror in the other.

6. Create the mysql.txt in ~/.mythtvtesting. It should 
include the line:
DBName=mythconvergtest

7. Make a recording with the older version of Myth. 
Preferably a longer one, since this gives oprofile more time 
to see what's expensive. You don't need to run oprofile at 
this stage, unless the regression you're seeing is on the 
backend.

8. Start up oprofile.

opcontrol --no-vmlinux

Since we're not really interested in kernel timings here, we 
don't need the vmlinux file, although it does no harm to 
have it.

opcontrol --start

9. Run the lower version and playback the file, preferably 
from beginning to end. If you're having problems with 
seeking, etc being slower, do some seeking, but try and set 
a pattern (fastforward from 0:30 - 13:00 at 60x, for example).

10. Dump the oprofile data.
opcontrol --dump
opreport --demangle=smart --symbols `which mythfrontend` 
 >~/mytholdprofile.txt

11. Clean out the profiling data
opcontrol --reset

12. Repeat the above with the SVN version.

You should now have a dump of exactly where in the code is 
taking CPU time, with comparable results from the 
'regressed' and 'good' versions. Compare these and look for 
any anomalies.

The FAQ at http://oprofile.sourceforge.net/faq/ is quite 
good, as is the oprofile documentation.

Hope this helps.

Cheers,

Allan.