[mythtv-users] slow mythfilldatabase

[Bill Meek]

On 1/8/21 7:25 PM, David Newman wrote:
> 
> 
> On 1/6/21 7:49 AM, Bill Meek wrote:
>> On 1/6/21 2:50 AM, Stephen Worthington wrote:
>>> On Tue, 5 Jan 2021 22:03:16 -0800, you wrote:
>>>
>>>> MythTV Version : v31.0-v31.0 on Ubuntu 20 LTS, mysql 8.0.22
>>>>
>>>> Greetings. After upgrading from Ubuntu 18 LTS to 20 LTS,
>>>> mythfilldatabase takes 45-60 minutes to run, sometimes more. It
>>>> previously took less than three minutes, and the system didn't have this
>>>> problem when upgrading from Ubuntu 16 LTS to 18 LTS.
>>>
>>> Where do you get your EPG data from?  Previous reports of similar
>>> problems turned out to be caused by getting a huge data set from
>>> Schedules Direct - all of the EPG rather than just the new data.  As I
>>> understand it (not being a Schedules Direct user), you are supposed to
>>> only get the full EPG data on the first download, and then after that
>>> limit how much you get to only the new data or maybe only one week of
>>> data.  If you have recently changed to Schedules Direct or to the new
>>> way of downloading the Schedules Direct data, that might be your
>>> problem.
>>
>> @dn, please mention that you completed the XMLTV conversion and that exactly one
>> copy of the data exists. 
> 
> Thanks, Bill. I did no conversion of any kind. There's one
> mythfilldatabase.SQL database. It's the same one that updated quickly
> when the system previously ran Ubuntu 18 LTS.

I don't recognize that DB. mythconverg is the norm.

> If I missed some steps in upgrading to Ubuntu 20 LTS, please let me know.

https://www.mythtv.org/wiki/Release_Notes_-_31#Major_Changes The 2nd
bullet point. DataDirect code has been removed. You could have been
running with a grabber before the v31 conversion (which is a really
good idea).

> (tl;dr: It seems to work better now; details below.)
> 
> 
> Here's the SQLite example, which should be similar to
>> the other method:
>>
>>   https://forum.mythtv.org/viewtopic.php?f=36&t=3480&start=15#p19508
> 
> That forum post seems to be mainly about SQLite. The grabber on this
> system is:
> 
> tv_grab_zz_sdjson

Exactly, and since it was my post and I selected the SQLite version,
that's what the example is for. Nothing wrong with using the other
grabber (OK, there is a security issue that the SQLite version fixes
by encrypting the password.) The point was having a single copy of
the files used, as opposed to links to other copies.

>> With an up-to-date* MythTV version, you can get more info by running
>> mythfilldatabase -v xmltv. 
> 
>  mythfilldatabase version: fixes/31 [v31.0-v31.0] www.mythtv.org
> 
> 
> Do that for the backend by using: mythbackend --setverbose xmltv
>> and waiting for the backend to run MFDB or just running: mythfilldatabase -v xmltv
>> from the command line *as user mythtv*.
>>
>> *I don't remember when the commit went in that improves the MFDB/XMLTV logging,
>> but if you do this,  you'll get it:
>>
>>   sudo add-apt-repository ppa:mythbuntu/31
>>   sudo apt update
>>   sudo apt dist-upgrade
> 
> This upgraded 13 packages:
> 
>   libmyth libmyth-python libmythtv-perl mythtv mythtv-backend mythtv-common
>   mythtv-database mythtv-dbg mythtv-doc mythtv-frontend
> mythtv-transcode-utils
>   mythweb php-mythtv
> 
>>
>> Also, verify that this exists (should have been installed automatically):
>>
>> $ cat /etc/mysql/conf.d/mythconverg.cnf
>> # Disable Binary logging for the mythconverg database. Binary logging
>> # for all databases was off by default prior to MySQL V8.
>> [mysqld]
>> binlog-ignore-db=mythconverg
> 
> yup
> 
> 
>>
>> If it doesn't, add it and restart the DB. I'd stop the backend first.
>>
>> FYI, I don't use any of the innodb_... or log_... changes in my .cnf file.
> 
> Yeah, I don't know that any of this made a difference. That .cnf file is
> all stuff that were suggested optimizations from the MythTV wiki. I had
> the same issue before using this version of the .cnf file.
> 
> But...it looks like these changes made a difference!
> 
> I ran mythfilldatabase manually after these steps, and it was *much*
> faster. I didn't prepend a time command, but it was something like two
> minutes or less. Let's see how it goes when mythfilldatabase runs
> programmatically.

I'd check to see if new listings are coming in. With the grabbers, expect
around 18 days of EPG.

-- 
Bill