[mythtv] backend from SVN 9761 bangs on mysqld

[Daniel Kristjansson]

On Sun, 2006-04-23 at 23:58 +0200, Janne Grunau wrote:
> On Sunday 23 April 2006 18:07, Daniel Kristjansson wrote:
> > On Sun, 2006-04-23 at 17:32 +0200, Janne Grunau wrote:
> > > It is not the scheduler alone. One of the recent EIT-refactoring
> > > (iirc 9671) adds noticeable DB-load.
> > I don't think hat wouldn't create the 100% cpu use spikes he's
> > seeing.
> I think we were both wrong. Even with my patches mysql's load is over 
> 30% while inserting events (without rescheduling). Depending on the 
> I just spoke to him on irc. He uses an AthlonXP 2500+, that's not that 
> much slower than my Athlon64 3000. But he is not seeing spikes of high 
> cpu utilization but plateaus of it. It starts just after adding the eit 
Ahh, well if he's not seeing 100% utilization, but more like a short
flurry of 40 or so DB queries every few seconds when a block of 20
events is inserted or updated that sounds like EIT insertion.

> I think it's best to start with a clean cache but the problem persists 
> then too. If we miss the update to the maximum version number we won't 
> update until the next time.
You will have to discuss this with Stuart A, he understands the
low pid priority scheme in the DVB EIT much better than me.

> > BTW The data 
> > does need to be stored in the DB, even the EIT version cache becomes
> > too big after a while. (But not nearly as big as the cache of the
> > entire EIT event that 0.19 uses.)
> I don't think using the db for the cache is a good idea. The greatest 
> parts of the EIT-scanning are atm queries. So adding more is not the 
> solution even if these queries are cheaper.
But this is one query per channel, that saves us hundreds of megabytes
of RAM..

> If counted just my program table and the average number of events per 
> channel is clearly below 500. Using 8 kbyte (plus overhead of QMap) 
> memory per program is not that bad.
500 events is way low, just one set of half hour events for 14 weeks
is 672. If you then account for two retransmissions as the data gets
expanded closer to transmission time you are talking over 2000 events.
Assuming a red-black three the QMap overhead probably 48 bytes per
entry (including the key on a 64 bit system). This gives us more like
a minimum of 96 KB per channel, which if you have a few sets of
channels, one or more DVB-S setups with 1000 channels. It gets
significant, plus who wants to have to recollect all this data after
we restart the backend?

> No, not really. I changed score_match in such a way that only a perfect 
> match can archieve a score of 12000. But Stuart Auchterlonie and I are 
> both not sure what you want to archieve with the second part of that 
> function. See attached diff.
So if the ending time of the program changed you could pick this up by
looking at the title or the description? :)

> > I think a persistent cache will make things much more efficient,
> > but right after adding a new EIT source you wouldn't have a cache
> > and so would still have a lot of DB load if this is what is causing
> > Tino's problem.
> But the work has to be done.
> The only thing we can change is the rate of insertions.
Yup, reducing the rate of insertions is the only solution for his
problem. While desireable, none of this caching does anything for
his problem on an unresponsive system when we are inserting events.

-- Daniel