[mythtv] Scheduler needs table keys?

[f-myth-users at media.mit.edu]

    > Date: Sun, 28 Jan 2007 23:40:47 -0500
    > From: Chris Pinkham <cpinkham at bc2va.org>

    > * On Sun Jan 28, 2007 at 07:21:23PM -0800, Bruce Markey wrote:
    > > 2007-01-28 17:39:20.600 Scheduled 249 items in 1.2 = 0.01 match + 1.19 place
    > > 2007-01-28 17:40:54.279 Scheduled 249 items in 1.2 = 0.01 match + 1.19 place
    > > 
    > > The match times are negligible unless there is a full resched.
    > > 
    > > : bjm at nordtv ; mythbackend --resched
    > > ...
    > > 2007-01-28 18:28:08.892 Scheduled 249 items in 9.6 = 8.18 match + 1.40 place

    > I don't see anything like this on my production or dev systems.

Nor I.  I typically see things like:

  2007-01-28 05:39:31.865 Scheduled 1156 items in 31.1 = 0.00 match + 31.08 place
  2007-01-28 06:01:24.727 Scheduled 1156 items in 24.5 = 0.00 match + 24.54 place
  2007-01-28 06:01:38.281 Scheduled 1156 items in 13.4 = 0.00 match + 13.44 place
  2007-01-28 06:02:15.506 Scheduled 1156 items in 14.9 = 0.00 match + 14.94 place
  2007-01-28 06:03:03.928 Scheduled 1156 items in 12.8 = 0.00 match + 12.77 place
  2007-01-28 06:03:51.463 Scheduled 1156 items in 13.1 = 0.00 match + 13.13 place
  2007-01-28 06:04:11.226 Scheduled 1156 items in 12.7 = 0.00 match + 12.66 place
  2007-01-28 06:04:55.894 Scheduled 1156 items in 13.7 = 0.00 match + 13.73 place
  2007-01-28 06:05:21.509 Scheduled 1156 items in 13.5 = 0.00 match + 13.47 place
  2007-01-28 06:06:08.682 Scheduled 1156 items in 13.7 = 0.00 match + 13.66 place

(This was when deleting a bunch of recordings, one after another, in
the UI.  Interestingly, the times went -down- (previous scheduler
activity was at 05:34:14), perhaps due to caching of the DB tables
in the kernel's filesystem cache.  The deletions were primarily
throttled by the scheduler, e.g., I kept waiting 10+ seconds before
the UI would let me delete the next thing 'cause the scheduler kept
running.)

Every so often, I see a match around 0.17 or so.  And once in the last
day, I saw -this-:

  2007-01-28 06:14:03.248 Scheduled 1156 items in 13.5 = 0.00 match + 13.49 place
  2007-01-28 08:00:25.533 Scheduled 1155 items in 25.3 = 0.00 match + 25.33 place
  2007-01-28 08:00:47.310 Scheduled 1263 items in 19.7 = 6.89 match + 12.82 place
  2007-01-28 10:31:25.835 Scheduled 1263 items in 25.6 = 0.00 match + 25.61 place
  2007-01-28 13:37:15.677 Scheduled 1252 items in 23.5 = 0.56 match + 22.97 place

No idea what that 6.89 spike is doing there, although that particular
run seems the -fastest- of those 5.  (That 08:00 is just a normal
resched when a recording ended.)  I've got maybe half a dozen power
searches, tops.

    [ . . . ]

    For reference, I just started with a new copy of my production record
    table and the only thing I did was truncate the oldrecorded table.
    The scheduler run time was cut in half:

    2007-01-28 23:34:45.908 Scheduled 754 items in 16.4 = 0.74 match + 15.66 place
    2007-01-28 23:35:07.078 Scheduled 754 items in 8.2 = 0.78 match + 7.41 place

    So for people with large histories, that data alone can cause as much
    scheduler run time as using kAllRecord vs kChannelRecord does.

True, but I was seeing IOBOUND errors with -one month- of oldrecorded
data, e.g., the system was only a month old.  Fix the interaction with
the scheduler, and even if oldrecorded makes scheduling somewhat slow,
few would notice.  Don't fix it, and no reasonable truncation of
oldrecorded is likely to help... :)

(Also:  I would -imagine- [without proof] that any reasonable
implementation of a database should scale approximately as the log of
the size of any table.  Thus several years of oldrecorded should only
add a few seconds to the operation compared to only a few months.  If
this isn't happening, either something's wrong with the indexing (or
there isn't the right kind of index), or something's wrong with the
query.)

    > Another item to put on my production TODO list. :)

Remember some people want that history, untrimmed!  (This [research]
system needs to know everything it's ever recorded, and oldrecorded is
designed for that---I've also added some columns [beginning with "_"
so as not to collide with any other columns in Myth] that record how
that recording got archived and where, plus its compressed closed-
captioning data, etc.  If anything in Myth deleted rows from
oldrecorded -without- my telling it to do so, I'd consider that a
bug---especially since I asked around a year ago on the dev list if I
could count on that as a repinvariant so I could use the table in this
manner.  If I had to put the data in a separate table just to make the
scheduler happy, I could, but -I- wouldn't be happy. :)

(And I recall somebody else on the list a while back saying he
depended on oldrecorded to remind him that he'd seen some particular
movie, or whatever---having items vanish from it arbitrarily would
break that for him.)