<html><head></head><body><div>On Mon, 2022-06-13 at 20:54 -0400, Larry Kennedy wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 7, 2022 at 9:52 PM Larry Kennedy <<a href="mailto:lunchtimelarry@gmail.com">lunchtimelarry@gmail.com</a>> wrote:<br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jun 5, 2022 at 2:06 PM Gary Buhrmaster <<a href="mailto:gary.buhrmaster@gmail.com" target="_blank">gary.buhrmaster@gmail.com</a>> wrote:<br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>On Sun, Jun 5, 2022 at 3:06 AM Gary Buhrmaster<br><<a href="mailto:gary.buhrmaster@gmail.com" target="_blank">gary.buhrmaster@gmail.com</a>> wrote:<br></div><div><br>> There are numerous checks to determine if the<br>> existing program is *really* the same (with all the<br>> details). Sometimes it is, sometimes it is not<br>> ("generic" programs such as news often get<br>> updates as the time gets closer to the "now"<br>> value).<br></div><div><br>And, as I recall, the generic (daily) news programs<br>can sometimes be marked as repeats, and<br>sometimes not, due to the logic that mythfilldatabase<br>and the upstream scheduling source use.<br></div><div><br>> And mythfilldatabase does some additional<br>> adjustments of the data that can (next run) result<br>> in detections of changes. Some of which was<br>> improved with recent (supported) releases of<br>> MythTV.<br></div><div><br>At the end of the run, there is a line of the form:<br> Updated programs: nnnnn Unchanged programs: nnnnnn<br>which indicates how many programs were updated.<br></div><div><br>If you run the same mythfilldatabase load immediately<br>again (which implies the same source guide data) you<br>will likely see a small (but persistent) number of updates<br>due to how MythTV manages the program data and<br>detection of changes (which require updates). For<br>a daily run, ~10% changes for Schedules Direct is<br>not atypical as generic placeholder show data get<br>replaced with actual episode data for the particular<br>showings and new data for an additional day.<br></div><div><br>Running a recent mythfilldatabase and a recent<br>version of the grabber, which together support<br>(mostly) accurate new (previouslyshown) marking,<br>I tended to use --no-mark-repeats to prevent<br>mythfilldatabase from "improving" the program<br>data (you can also modify the database setting<br>NewEpisodeWindow to accomplish something<br>equivalent, both mostly only matter because<br>Schedules Direct actually often has more than<br>14 days of program data, and using/changing<br>either may cause different issues depending<br>on the details (and --no-mark-repeats had<br>a specific bug when used in older versions)).<br></div><div><br>Lastly, sometimes database tuning may be<br>appropriate. From time to time there are<br>discussions on the MythTV email lists, or<br>forum, regarding database tuning, Running<br></div><div><a href="http://email.mg.glenb.net/c/eJwlzDsOgzAQANHT4C6WvxgKF2lyj_V6FyIZAsZE4vZBijTNayZHZMjeiXc0nCAorVCjhawHHhRQcNqPLmWrKPQQyFgjERYqnVNToTXJlZqYo-OUE3J2QyKNvWP0PGoMwD5bglGUOLe2dfbZmdfdch17aedKVW5F1LhcbW7fx3lQPe71n_JTpx97YDO3" rel="noreferrer" target="_blank">mysqltuner.pl</a> can point out specific cases<br>which may be causing issues (just be aware<br>that not everything mentioned by that script<br>is actually going to be an issue for MythTV).<br></div><br></blockquote><div><br></div><div>OK, Gary, I bit the bullet and upgraded to Ubuntu 20 LTS and Mythtv 31. I'm sure that will help long term, but it sure created some additional drama :)<br></div><div><br></div><div>I found some old tuning parameters in mysql.cnf that I think were problematic (thanks Bill from another post of yours I stumbled across). I removed those and it seems to help. MFDB logging shows that inserts/deletes are now taking 1 second or less, as seen here: <br></div><div><br>2022-06-07 21:47:44.317605 I Removing existing program: 2022-06-13T14:00:00Z - 2022-06-13T15:00:00Z <a href="http://email.mg.glenb.net/c/eJwljUsOwiAUAE9TdhIolM-ChRsTj_H6eJQabA1QE29vjclsJrOYGDBBnDRbw5hmsEIKlKggSpecALJaTl7PUQmyBiyNauQITyqDFkuhbeYbdZaDQqEpOZe0BxJovTQCLDr0RmrlDCsh9_4a1HUYbyf3ySov-aPtG2-YKR6FWlwrYed7XVgNz0_P_X05GtV2zv76a1-Dwje7" target="_blank">I57391.json.schedulesdirect.org</a> Little People, Big World<br>2022-06-07 21:47:45.123297 I Inserting new program : 2022-06-13T14:00:00Z - 2022-06-13T15:00:00Z <a href="http://email.mg.glenb.net/c/eJwljUsOwiAUAE9TdhIolM-ChRsTj_H6eJQabA1QE29vjclsJrOYGDBBnDRbw5hmsEIKlKggSpecALJaTl7PUQmyBiyNauQITyqDFkuhbeYbdZaDQqEpOZe0BxJovTQCLDr0RmrlDCsh9_4a1HUYbyf3ySov-aPtG2-YKR6FWlwrYed7XVgNz0_P_X05GtV2zv76a1-Dwje7" target="_blank">I57391.json.schedulesdirect.org</a> Little People, Big World<br>2022-06-07 21:47:47.038876 I Removing existing program: 2022-06-13T15:00:00Z - 2022-06-13T16:00:00Z <a href="http://email.mg.glenb.net/c/eJwljUsOwiAUAE9TdhIolM-ChRsTj_H6eJQabA1QE29vjclsJrOYGDBBnDRbw5hmsEIKlKggSpecALJaTl7PUQmyBiyNauQITyqDFkuhbeYbdZaDQqEpOZe0BxJovTQCLDr0RmrlDCsh9_4a1HUYbyf3ySov-aPtG2-YKR6FWlwrYed7XVgNz0_P_X05GtV2zv76a1-Dwje7" target="_blank">I57391.json.schedulesdirect.org</a> Little People, Big World<br>2022-06-07 21:47:47.927278 I Inserting new program : 2022-06-13T15:00:00Z - 2022-06-13T16:00:00Z <a href="http://email.mg.glenb.net/c/eJwljUsOwiAUAE9TdhIolM-ChRsTj_H6eJQabA1QE29vjclsJrOYGDBBnDRbw5hmsEIKlKggSpecALJaTl7PUQmyBiyNauQITyqDFkuhbeYbdZaDQqEpOZe0BxJovTQCLDr0RmrlDCsh9_4a1HUYbyf3ySov-aPtG2-YKR6FWlwrYed7XVgNz0_P_X05GtV2zv76a1-Dwje7" target="_blank">I57391.json.schedulesdirect.org</a> Little People, Big World<br>2022-06-07 21:47:49.853728 I Removing existing program: 2022-06-13T18:00:00Z - 2022-06-13T19:00:00Z <a href="http://email.mg.glenb.net/c/eJwljUsOwiAUAE9TdhIolM-ChRsTj_H6eJQabA1QE29vjclsJrOYGDBBnDRbw5hmsEIKlKggSpecALJaTl7PUQmyBiyNauQITyqDFkuhbeYbdZaDQqEpOZe0BxJovTQCLDr0RmrlDCsh9_4a1HUYbyf3ySov-aPtG2-YKR6FWlwrYed7XVgNz0_P_X05GtV2zv76a1-Dwje7" target="_blank">I57391.json.schedulesdirect.org</a> 90 Day Diaries<br>2022-06-07 21:47:50.308645 I Inserting new program : 2022-06-13T18:00:00Z - 2022-06-13T19:00:00Z <a href="http://email.mg.glenb.net/c/eJwljUsOwiAUAE9TdhIolM-ChRsTj_H6eJQabA1QE29vjclsJrOYGDBBnDRbw5hmsEIKlKggSpecALJaTl7PUQmyBiyNauQITyqDFkuhbeYbdZaDQqEpOZe0BxJovTQCLDr0RmrlDCsh9_4a1HUYbyf3ySov-aPtG2-YKR6FWlwrYed7XVgNz0_P_X05GtV2zv76a1-Dwje7" target="_blank">I57391.json.schedulesdirect.org</a> 90 Day Diaries<br></div><div><br></div><div>Still, this is going to take longer than one would think necessary. Especially, given that this is not the first run following the upgrade. <br></div><div><br></div><div>I have a few questions about why MFDB is slow:</div><div><br></div><div>1. Does the number of channels in the Sqlite database matter? I have 1,000 channels of which only 90 are marked as "selected." Should I/can I delete the unused channels?<br></div><div>2. Does the number of channels in the mythtv database matter? I have 1,300 channels, of which 100 or so are marked as "visible." Should I/can I delete the unused channels?</div><div>3. I noticed that the XMLTV download to /tmp had over 1 million lines, per "cat /tmp/xxx | wc". That seems large.<br></div><div><br></div><div>Which of these is possibly slowing me down? I assume #1, so I tried to delete "unselected" channels from the sqlite database, but they got reloaded later. I can't recall what command I ran that caused the reload, or if this was automatic and unavoidable. Thoughts?</div><div><br></div><div><br></div><div>Larry<br></div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>_______________________________________________<br>mythtv-users mailing list<br></div><div><a href="mailto:mythtv-users@mythtv.org" target="_blank">mythtv-users@mythtv.org</a><br></div><div><a href="http://email.mg.glenb.net/c/eJxNjk0OwiAUhE9TdhIovy5YuPEejwe0JEBNQRNvbzUmmsxmvplJJjhMEJQk2c3Jg2GcIUcBgdtkGUQjuTpLHwSLRoOJs5gpQo1lkmwpsXna4iCrS0mhlt4Hq7lIGJS2UlidvFeIXCIpbh3jNonLNF8PldxHp_U51vGg274cqEIuFdo3zC1tb_hpnO497p3s7t8eD377FwZoQTE" rel="noreferrer" target="_blank">http://lists.mythtv.org/mailman/listinfo/mythtv-users</a><br></div><div><a href="http://email.mg.glenb.net/c/eJxFjTsOwjAQBU8Td1j-xqZwQUMFZ4jW9jqxSAIkGxC3JxIF0qtGM3o5pALZGlaDKhGckCLJpCFLX7wAdEbao4lZC3QtOFRa8QQTjo0R_Yhz5DMSG0JMmJTypgXrSizoIxyjwJxsFFKCZWMYiB6NPjXqvO9db5VPHxroxe9Lv5Mr1LHOfXepK3VI9bkhEbIl_KzDtuKy7q__6AtATD1p" rel="noreferrer" target="_blank">http://wiki.mythtv.org/Mailing_List_etiquette</a><br>MythTV Forums: <a href="http://email.mg.glenb.net/c/eJxFjMsKwyAQAL8m3iqrrtnk4KGX_seqa1LIoxhT6N830ENhLsPA5JAKZ4_qGWyJTGAgmeQ4m6EMwEJo_IgxOxDqmcQ6qxOvsnQI0yJb1Js0NYc8gkuEnih5ZkvscCATgcwIiH1WS5hbex2du3f2cVH2eq56_bS5vfVeJ1XDT27nIfW49v_2BaqgMxw" rel="noreferrer" target="_blank">https://forum.mythtv.org</a></div></blockquote></div></div><br></blockquote><div><br></div><div>As recommended, I've upgraded: the O/S is LTS 20.04.4, MariaDB is 10.3.34, and Mythtv is V31 w/fixes. XMLTV module is version 0.6.1 and tv_grab_zz_sdjson_sqlite is version 1.66.</div><div><br></div><div> However, mythfilldatabase still takes over 6 hours daily. Can anyone say if this is normal? Top shows a low level of CPU for all related processes. <br></div><div><br></div><div>One side effect of all this is the database wait timeout issue that nukes the upcoming recordings. I think I've addressed that with a global change to bump up the default 8hour timeout. Time will tell.<br></div><div><br></div><div>Here is a snippet of the log showing where mythfilldatabase spent time:</div><div><br></div><div>2022-06-13 01:01:15.313333 I Cardutil: HDHomeRun Cablecard Present.<br>2022-06-13 01:01:15.333400 I Cardutil: HDHomeRun Cablecard Present.<br>2022-06-13 01:01:15.353206 I Cardutil: HDHomeRun Cablecard Present.<br>2022-06-13 07:33:51.007237 I Updated programs: 5891 Unchanged programs: 32563<br>2022-06-13 07:33:51.865934 N Data fetching complete.<br>2022-06-13 07:33:51.866108 I Adjusting program database end times.<br>2022-06-13 07:33:53.149755 I 0 replacements made</div><div><br></div><div>I can add more logging if anyone has suggestions on what to look for. I just can't see why this process should take more than 15 minutes to perform a daily incremental change. <br></div><div><br></div><div>Larry<br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div><div>_______________________________________________<br></div><div>mythtv-users mailing list<br></div><div><a href="mailto:mythtv-users@mythtv.org">mythtv-users@mythtv.org</a><br></div><div><a href="http://email.mg.glenb.net/c/eJxNjk0OwiAUhE9TdhIovy5YuPEejwe0JEBNQRNvbzUmmsxmvplJJjhMEJQk2c3Jg2GcIUcBgdtkGUQjuTpLHwSLRoOJs5gpQo1lkmwpsXna4iCrS0mhlt4Hq7lIGJS2UlidvFeIXCIpbh3jNonLNF8PldxHp_U51vGg274cqEIuFdo3zC1tb_hpnO497p3s7t8eD377FwZoQTE">http://lists.mythtv.org/mailman/listinfo/mythtv-users</a><br></div><div><a href="http://email.mg.glenb.net/c/eJxFjTsOwjAQBU8Td1j-xqZwQUMFZ4jW9jqxSAIkGxC3JxIF0qtGM3o5pALZGlaDKhGckCLJpCFLX7wAdEbao4lZC3QtOFRa8QQTjo0R_Yhz5DMSG0JMmJTypgXrSizoIxyjwJxsFFKCZWMYiB6NPjXqvO9db5VPHxroxe9Lv5Mr1LHOfXepK3VI9bkhEbIl_KzDtuKy7q__6AtATD1p">http://wiki.mythtv.org/Mailing_List_etiquette</a><br></div><div>MythTV Forums: <a href="http://email.mg.glenb.net/c/eJxFjMsKwyAQAL8m3iqrrtnk4KGX_seqa1LIoxhT6N830ENhLsPA5JAKZ4_qGWyJTGAgmeQ4m6EMwEJo_IgxOxDqmcQ6qxOvsnQI0yJb1Js0NYc8gkuEnih5ZkvscCATgcwIiH1WS5hbex2du3f2cVH2eq56_bS5vfVeJ1XDT27nIfW49v_2BaqgMxw">https://forum.mythtv.org</a><br></div></blockquote><div>i run mythfilldatabase as a systemd service outside of myth with 3 different sourceid's. i would suggest testing by adding a test sourceid and using tvgrab sdjson as opposed sqlite. and setup a conf file with a small amount of channels. and then maybe you might see whats going on or be on a path to just run it outside of myth as i do. for example for my over the air source my script looks like this:</div><div><br></div><div>/usr/bin/vendor_perl/tv_grab_zz_sdjson --days 12 --config-file ota.conf --output ota.xml</div><div>mythfilldatabase --file --only-update-guide --sourceid 9 --xmlfile ota.xml</div><div><br></div><div>the conf file (first few lines) looks like this (user / pass omitted)</div><div><br></div><div>cache=/home/g/.xmltv/tv_grab_zz_sdjson.cache</div><div>channel-id-format=mythtv</div><div>previously-shown-format=date</div><div>username=*****</div><div>password=*****</div><div>mode=channels</div><div>channels!USA-CA57315-X</div><div>channels!USA-NY31519-X</div><div>channels=USA-OTA-90066</div><div>channel=19567</div><div>channel!88571</div><div>channel!112431</div><div>channel!117036</div><div>channel=19568</div><div><br></div><div>channels you want data for have '=' and channels you exclude have '!'</div><div><br></div><div>for a couple hundred channels, it runs very quickly</div><div><br></div><div><br></div><div><span></span></div><img width="1px" height="1px" alt="" src="http://email.mg.glenb.net/o/eJwly8EOwiAMANCvkZuktGC3Ax_TQtlMtpkwNPHvXeLxHV7NpUlN0T0zNhWGACUUkhqmNoEYx5DmqJXA-CFsSOiL7LbdIiybHeoPG27NGA0SA5oRaGSdDQtV1WulhCSu5_071vG5v0_r57X_9K--_AAMMyiC"></body></html>