[mythtv-users] Mythfilldatabase taking FOREVER

Tue Jun 14 01:12:57 UTC 2022

On Mon, 2022-06-13 at 20:54 -0400, Larry Kennedy wrote:
> 
> 
> On Tue, Jun 7, 2022 at 9:52 PM Larry Kennedy
> <lunchtimelarry at gmail.com> wrote:
> > 
> > 
> > On Sun, Jun 5, 2022 at 2:06 PM Gary Buhrmaster
> > <gary.buhrmaster at gmail.com> wrote:
> > > On Sun, Jun 5, 2022 at 3:06 AM Gary Buhrmaster
> > > <gary.buhrmaster at gmail.com> wrote:
> > > 
> > > > There are numerous checks to determine if the
> > > > existing program is *really* the same (with all the
> > > > details).  Sometimes it is, sometimes it is not
> > > > ("generic" programs such as news often get
> > > > updates as the time gets closer to the "now"
> > > > value).
> > > 
> > > And, as I recall, the generic (daily) news programs
> > > can sometimes be marked as repeats, and
> > > sometimes not, due to the logic that mythfilldatabase
> > > and the upstream scheduling source use.
> > > 
> > > > And mythfilldatabase does some additional
> > > > adjustments of the data that can (next run) result
> > > > in detections of changes.  Some of which was
> > > > improved with recent (supported) releases of
> > > > MythTV.
> > > 
> > > At the end of the run, there is a line of the form:
> > >       Updated programs: nnnnn Unchanged programs: nnnnnn
> > > which indicates how many programs were updated.
> > > 
> > > If you run the same mythfilldatabase load immediately
> > > again (which implies the same source guide data) you
> > > will likely see a small (but persistent) number of updates
> > > due to how MythTV manages the program data and
> > > detection of changes (which require updates).  For
> > > a daily run, ~10% changes for Schedules Direct is
> > > not atypical as generic placeholder show data get
> > > replaced with actual episode data for the particular
> > > showings and new data for an additional day.
> > > 
> > > Running a recent mythfilldatabase and a recent
> > > version of the grabber, which together support
> > > (mostly) accurate new (previouslyshown) marking,
> > > I tended to use --no-mark-repeats to prevent
> > > mythfilldatabase from "improving" the program
> > > data (you can also modify the database setting
> > > NewEpisodeWindow to accomplish something
> > > equivalent, both mostly only matter because
> > > Schedules Direct actually often has more than
> > > 14 days of program data, and using/changing
> > > either may cause different issues depending
> > > on the details (and --no-mark-repeats had
> > > a specific bug when used in older versions)).
> > > 
> > > Lastly, sometimes database tuning may be
> > > appropriate.  From time to time there are
> > > discussions on the MythTV email lists, or
> > > forum, regarding database tuning,  Running
> > > mysqltuner.pl can point out specific cases
> > > which may be causing issues (just be aware
> > > that not everything mentioned by that script
> > > is actually going to be an issue for MythTV).
> > > 
> > 
> > 
> > OK, Gary, I bit the bullet and upgraded to Ubuntu 20 LTS and Mythtv
> > 31. I'm sure that will help long term, but it sure created some
> > additional drama :)
> > 
> > I found some old tuning parameters in mysql.cnf that I think were
> > problematic (thanks Bill from another post of yours I stumbled
> > across).  I removed those and it seems to help.  MFDB logging shows
> > that inserts/deletes are now taking 1 second or less, as seen
> > here:  
> > 
> > 2022-06-07 21:47:44.317605 I  Removing existing program: 2022-06-
> > 13T14:00:00Z - 2022-06-13T15:00:00Z I57391.json.schedulesdirect.org
> > Little People, Big World
> > 2022-06-07 21:47:45.123297 I  Inserting new program    : 2022-06-
> > 13T14:00:00Z - 2022-06-13T15:00:00Z I57391.json.schedulesdirect.org
> > Little People, Big World
> > 2022-06-07 21:47:47.038876 I  Removing existing program: 2022-06-
> > 13T15:00:00Z - 2022-06-13T16:00:00Z I57391.json.schedulesdirect.org
> > Little People, Big World
> > 2022-06-07 21:47:47.927278 I  Inserting new program    : 2022-06-
> > 13T15:00:00Z - 2022-06-13T16:00:00Z I57391.json.schedulesdirect.org
> > Little People, Big World
> > 2022-06-07 21:47:49.853728 I  Removing existing program: 2022-06-
> > 13T18:00:00Z - 2022-06-13T19:00:00Z I57391.json.schedulesdirect.org
> > 90 Day Diaries
> > 2022-06-07 21:47:50.308645 I  Inserting new program    : 2022-06-
> > 13T18:00:00Z - 2022-06-13T19:00:00Z I57391.json.schedulesdirect.org
> > 90 Day Diaries
> > 
> > Still, this is going to take longer than one would think
> > necessary.  Especially, given that this is not the first run
> > following the upgrade.  
> > 
> > I have a few questions about why MFDB is slow:
> > 
> > 1.  Does the number of channels in the Sqlite database matter?  I
> > have 1,000 channels of which only 90 are marked as "selected." 
> > Should I/can I delete the unused channels?
> > 2.  Does the number of channels in the mythtv database matter?  I
> > have 1,300 channels, of which 100 or so are marked as "visible." 
> > Should I/can I delete the unused channels?
> > 3.  I noticed that the XMLTV download to /tmp had over 1 million
> > lines, per "cat /tmp/xxx | wc".  That seems large.
> > 
> > Which of these is possibly slowing me down?  I assume #1, so I
> > tried to delete "unselected" channels from the sqlite database, but
> > they got reloaded later.  I can't recall what command I ran that
> > caused the reload, or if this was automatic and unavoidable. 
> > Thoughts?
> > 
> > 
> > Larry
> > 
> > > _______________________________________________
> > > mythtv-users mailing list
> > > mythtv-users at mythtv.org
> > > http://email.mg.glenb.net/c/eJxNjk0OwiAUhE9TdhIovy5YuPEejwe0JEBNQRNvbzUmmsxmvplJJjhMEJQk2c3Jg2GcIUcBgdtkGUQjuTpLHwSLRoOJs5gpQo1lkmwpsXna4iCrS0mhlt4Hq7lIGJS2UlidvFeIXCIpbh3jNonLNF8PldxHp_U51vGg274cqEIuFdo3zC1tb_hpnO497p3s7t8eD377FwZoQTE
> > > http://email.mg.glenb.net/c/eJxFjTsOwjAQBU8Td1j-xqZwQUMFZ4jW9jqxSAIkGxC3JxIF0qtGM3o5pALZGlaDKhGckCLJpCFLX7wAdEbao4lZC3QtOFRa8QQTjo0R_Yhz5DMSG0JMmJTypgXrSizoIxyjwJxsFFKCZWMYiB6NPjXqvO9db5VPHxroxe9Lv5Mr1LHOfXepK3VI9bkhEbIl_KzDtuKy7q__6AtATD1p
> > > MythTV Forums: http://email.mg.glenb.net/c/eJxFjMsKwyAQAL8m3iqrrtnk4KGX_seqa1LIoxhT6N830ENhLsPA5JAKZ4_qGWyJTGAgmeQ4m6EMwEJo_IgxOxDqmcQ6qxOvsnQI0yJb1Js0NYc8gkuEnih5ZkvscCATgcwIiH1WS5hbex2du3f2cVH2eq56_bS5vfVeJ1XDT27nIfW49v_2BaqgMxw
> > 
> 
> 
> As recommended, I've upgraded: the O/S is LTS 20.04.4, MariaDB is
> 10.3.34, and Mythtv is V31 w/fixes.  XMLTV module is version 0.6.1
> and tv_grab_zz_sdjson_sqlite is version 1.66.
> 
> However, mythfilldatabase still takes over 6 hours daily.  Can anyone
> say if this is normal?  Top shows a low level of CPU for all related
> processes.  
> 
> One side effect of all this is the database wait timeout issue that
> nukes the upcoming recordings.  I think I've addressed that with a
> global change to bump up the default 8hour timeout. Time will tell.
> 
> Here is a snippet of the log showing where mythfilldatabase spent
> time:
> 
> 2022-06-13 01:01:15.313333 I  Cardutil: HDHomeRun Cablecard Present.
> 2022-06-13 01:01:15.333400 I  Cardutil: HDHomeRun Cablecard Present.
> 2022-06-13 01:01:15.353206 I  Cardutil: HDHomeRun Cablecard Present.
> 2022-06-13 07:33:51.007237 I  Updated programs: 5891 Unchanged
> programs: 32563
> 2022-06-13 07:33:51.865934 N  Data fetching complete.
> 2022-06-13 07:33:51.866108 I  Adjusting program database end times.
> 2022-06-13 07:33:53.149755 I      0 replacements made
> 
> I can add more logging if anyone has suggestions on what to look
> for.  I just can't see why this process should take more than 15
> minutes to perform a daily incremental change. 
> 
> Larry
> 
> 
> 
> 
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://email.mg.glenb.net/c/eJxNjk0OwiAUhE9TdhIovy5YuPEejwe0JEBNQRNvbzUmmsxmvplJJjhMEJQk2c3Jg2GcIUcBgdtkGUQjuTpLHwSLRoOJs5gpQo1lkmwpsXna4iCrS0mhlt4Hq7lIGJS2UlidvFeIXCIpbh3jNonLNF8PldxHp_U51vGg274cqEIuFdo3zC1tb_hpnO497p3s7t8eD377FwZoQTE
> http://email.mg.glenb.net/c/eJxFjTsOwjAQBU8Td1j-xqZwQUMFZ4jW9jqxSAIkGxC3JxIF0qtGM3o5pALZGlaDKhGckCLJpCFLX7wAdEbao4lZC3QtOFRa8QQTjo0R_Yhz5DMSG0JMmJTypgXrSizoIxyjwJxsFFKCZWMYiB6NPjXqvO9db5VPHxroxe9Lv5Mr1LHOfXepK3VI9bkhEbIl_KzDtuKy7q__6AtATD1p
> MythTV Forums: http://email.mg.glenb.net/c/eJxFjMsKwyAQAL8m3iqrrtnk4KGX_seqa1LIoxhT6N830ENhLsPA5JAKZ4_qGWyJTGAgmeQ4m6EMwEJo_IgxOxDqmcQ6qxOvsnQI0yJb1Js0NYc8gkuEnih5ZkvscCATgcwIiH1WS5hbex2du3f2cVH2eq56_bS5vfVeJ1XDT27nIfW49v_2BaqgMxw
i run mythfilldatabase as a systemd service outside of myth with 3
different sourceid's. i would suggest testing by adding a test sourceid
and using tvgrab sdjson as opposed sqlite. and setup a conf file with a
small amount of channels. and then maybe you might see whats going on
or be on a path to just run it outside of myth as i do. for example for
my over the air source my script looks like this:

/usr/bin/vendor_perl/tv_grab_zz_sdjson --days 12 --config-file ota.conf
--output ota.xml
mythfilldatabase --file --only-update-guide --sourceid 9 --xmlfile
ota.xml

the conf file (first few lines) looks like this (user / pass omitted)

cache=/home/g/.xmltv/tv_grab_zz_sdjson.cache
channel-id-format=mythtv
previously-shown-format=date
username=*****
password=*****
mode=channels
channels!USA-CA57315-X
channels!USA-NY31519-X
channels=USA-OTA-90066
channel=19567
channel!88571
channel!112431
channel!117036
channel=19568

channels  you want data for have '=' and channels you exclude have '!'

for a couple hundred channels, it runs very quickly


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-users/attachments/20220613/bb2d64fa/attachment.htm>