[mythtv-users] Populating the .xmltv file for my grabber from mythtv database?
watkinshome at gmail.com
Wed May 5 08:16:49 UTC 2021
On Wed, 5 May 2021 at 05:06, Stephen Worthington <stephen_agent at jsw.gen.nz>
> On Tue, 4 May 2021 13:32:55 -0400, you wrote:
> >On Tue, May 4, 2021 at 12:58 PM David Watkins <watkinshome at gmail.com>
> >> On Sun, 2 May 2021 at 14:55, Larry Kennedy <lunchtimelarry at gmail.com>
> >> wrote:
> >>> I am just now attempting the upgrade to version 31, and the cutover
> >>> DD to XMLTV.
> >>> I successfully configured the sqlite grabber to pull from SD, and
> >>> that it downloaded all of the channels on my SD lineup, even though
> >>> are not selected. I guess this is intended behavior?
> >>> Seeing how tedious it is to mark channels as "selected" in the Sqlite
> >>> database, I came up with a slightly different way to transfer the
> >>> I currently care about.
> >>> I run this query in my existing mythconverg database:
> >>> SELECT CONCAT('update channels set selected = 1 where channum = ',
> >>> channum ,";") AS combined FROM channel where visible=1
> >>> Pipe this output to a file named selected.sql which creates the sqlite
> >>> update statements to set the correct channels to "selected" in the new
> >>> grabber database to match what you now have as visible channels in
> >>> Then, just use the grabber tool to make all the channels as "not
> >>> selected" by default, and then pipe the file from above into the sqlite
> >>> query as seen on the xmltv wiki:
> >>> sqlite3 $HOME/.xmltv/SchedulesDirect.DB < selected.sql
> >>> This seems simpler than some of the other methods I've seen on the wiki
> >>> and elsewhere...
> >>> Moving on....I need some advice before I go past the rubicon on this
> >>> + xmltv upgrade:
> >>> After I do the 31 upgrade, and since I only use an HDHR, I plan to just
> >>> blow away my channels and repopulate. I think all the channel ids will
> >>> line up the same as before.
> >>> Currently, in v30, I run mythfilldatabase as a cron job. Should I
> >>> continue to do that, or switch to another method? I've never let
> >>> natively run the mythfilldatabase process. I'm having a hard time
> >>> following the intent of the setup-video-sources wiki page on this
> >>> Advice appreciated!
> >>> Thanks,
> >>> Larry
> >> I have a combined BE/FE which shuts down when not being used and wakes
> >> for recordings. I run this script from crondaily and it seems to work
> >> OK.
> >> #!/usr/bin/bash
> >> /usr/local/bin/mythshutdown --lock
> >> tv_grab_zz_sdjson --days 10 --config-file
> >> --output ~/sd_listing.xml 2>/dev/null
> >> /usr/local/bin/mythfilldatabase --only-update-guide --max-days 10 --file
> >> --sourceid 2 --xmlfile ~/sd_listing.xml 2>/dev/null
> >> /usr/local/bin/mythshutdown --unlock
> >> It's not an ideal solution because I have a low powered ION motherboard
> >> with only 4GB RAM and this more or less guarantees that MFD will run
> >> a recording is in progress when I'd prefer to ensure it ran when the box
> >> wasn't recording. Also, if I ever go more than 10 days without a
> >> scheduled recording the whole thing will come to a stop because it will
> >> out of guide data and never wake up to load any more. In practice
> >> of those things causes a problem.
> >> Although SD returns 18 days of guide data I've found that many channels
> >> just have 'boilerplate' programme information after 10 days or so, so I
> >> limit both SD and MFD to 10.
> >I attempted an upgrade over the weekend, but ended up rolling it back.
> >Kudos to the myth devs that created the backup and restore scripts!
> >I'm still on v30 with Schedules Direct DataDirect and a cron job to run
> >mythfilldatabase. To get to v31 and the new xmltv grabber, I decided to
> >do the XMLTV upgrade now, followed by an upgrade to v31, say, a week
> >later. I chose this path since it appears that v31 would force the same
> >upgrade of XMLTV, and I wanted to decouple these two things. If I
> >approached this incorrectly, please let me know.
> >I ran the xmltv install, then followed the wiki to configure the sqlite
> >grabber. This all appeared to work, as I was able to see like 1,000 rows
> >of channel data in the sqlite database. As indicated in this thread, I
> >created a simple way to set the right channels as "selected".
> >After that, I went into mythtv-setup, channel editor, and deleted all my
> >channels. I only have one source that's by HDHR.
> >Then, I added a new video source that matches the name of the grabber
> >config, set this up as Multinational, and done.
> >Then I went into the other setup screen where I map the video source to my
> >HDHR tuners. Forget the name of that one. Capture cards?
> >After this, I ran mythfilldatabase. Looking at mythweb, I could see it
> >populated the channels back into the database, but I am pretty sure it put
> >in more than the ones I marked as "selected" in the Sqlite database
> >Strange. The listing started to appear, but VERY slowly.
> >Mytfilldatabase ran for like 4+ hours, and since I was getting close to
> >time I needed to finish or bail out, I killed it. I then tried running
> >script that does 3 days at a time, and this ran to full completion right
> >away, but never fully populated my listings. I still had major gaps in
> >listings. Some were there and some never showed up.
> >Any advice before I give this another go this coming weekend? Is my
> >strategy flawed? Did I miss a step? How best to run mythfilldatabase so
> >that it doesn't take all day? Is it the 3 days at a time script?
> As I understand it (not being an SD user), the first time you run
> mythfilldatabase after installing the new setup, it does take hours to
> run. Subsequent updates need to be limited to only a few days to make
> the time reasonable. This has been discussed here several times, and
> also on the forum, so a search here:
> should help.
> Unfortunately, it seems that Google searches these days are not
> finding the mailing list references much, unless you have a very
> specific keyword.
> Looking on as a disinterested party, I have never understood why the
> SD json EPG requires such long times to run. The actual process of
> putting the EPG data from my XMLTV generated sources into the database
> takes less than a minute. I have two sources, 141 channels and one
> week of EPG data.
When I configured xmltv for the Schedules Direct lineup at my location it
picked up about 140 channels, which is consistent with what MythTV picks up
with a channel scan and puts into the database.
In MythTV I've marked all but 51 channels as 'invisible' and I've disabled
those channels in xmltv as well, by replacing the '=' with a '!' in the
relative line in the xmltv config file.
I have pretty low powered hardware (Zotac ION / 4GB RAM). Running the
xmltv grab for 10 days takes 2 or 3 minutes to grab the file. Running
mythfilldatabase on that file takes about 20 minutes the first time (when
the guide database table is empty) and about 5 minutes when it only has to
update the 10th day (assuming that no earlier entries have changed).
So my daily mythfilldatabase run takes about 7 minutes. Reducing the number
of channels from 140 to 51 helped, as did reducing the days from 18 to 10.
I only have one data source. When I had two then I ran xmltv once to grab
the data, but I had to run mythfilldatabase twice(once for each source).
This slowed things down somewhat.
I have to say that working out the xmltv id's for each channel is not
trivial and there can be maintenance to do after a channel rescan. I have a
couple of SQL queries and an Excel spreadsheet to help keep track of
things, and I've seen people post similar schemes here and on the WiKi. I
believe that there's work in hand to preserve xmltvid, icon name and other
channel parameters after a rescan so things should get a bit easier
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the mythtv-users