[mythtv] Has anyone tested channel scanning in a us-cable Clear QAM environment?

Tue Feb 11 00:21:46 UTC 2020

On 2/10/2020 2:53 PM, Klaas de Waal wrote:
> Hi faginbagin,
> 
> On Mon, 10 Feb 2020 at 03:15, faginbagin <mythtv at hbuus.com <mailto:mythtv at hbuus.com>> wrote:
> 
>     On 2/8/2020 3:52 AM, Klaas de Waal wrote:
>     > Hi faginbagin,
>     >
>     > On Sat, 8 Feb 2020 at 02:39, faginbagin <mythtv at hbuus.com <mailto:mythtv at hbuus.com> <mailto:mythtv at hbuus.com <mailto:mythtv at hbuus.com>>> wrote:
>     >
>     >     On 2/7/2020 1:55 PM, Klaas de Waal wrote:
>     >     > On Fri, 7 Feb 2020 at 08:00, faginbagin <mythtv at hbuus.com <mailto:mythtv at hbuus.com> <mailto:mythtv at hbuus.com <mailto:mythtv at hbuus.com>> <mailto:mythtv at hbuus.com <mailto:mythtv at hbuus.com> <mailto:mythtv at hbuus.com <mailto:mythtv at hbuus.com>>>> wrote:
>     >     >
>     >     >     I ask because I can't get mythtv-setup to find any channels. I first
>     >     >     tried a scan before running mythfilldatabase as mentioned in this email
>     >     >     thread on mythtv-users:
>     >     >     http://lists.mythtv.org/pipermail/mythtv-users/2020-January/402291.html
>     >     >
>     >     >     I just tried another channel scan after running mythfilldatabase.
>     >     >     mythfilldatabase added lots of channels and listings to the database,
>     >     >     but the channel data looked incomplete. So I did a full channel scan,
>     >     >     which said it was scanning for ATSC channels, not QAM, and found no
>     >     >     channels. I can't find any channel scan settings that distinguish
>     >     >     between ATSC and QAM.
>     >     >
>     >     >     Not surprisingly, the channel data still looks wrong. For example, all
>     >     >     of the channel records have a NULL mplexid and the dtv_multiplex table
>     >     >     is empty. Has tuning a clear qam channel changed so much that channel
>     >     >     records don't need to point to valid dtv_multiplex records any more?
>     >     >
>     >     >     FWIW, both "General -> Locale Settings -> Channel frequency table" and
>     >     >     "Video Sources -> MyVideoSource -> Channel frequency table" are set to
>     >     >     us-cable and the listings grabber is "Multinational (Schedules Direct
>     >     >     JSON web services w..." and the listings data looks correct.
>     >     >
>     >     >
>     >     > It is interesting to know which version of MythTV you are running. There are lots of changes in the latest master / v31 compared with previous versions.
>     >     > You might have missed the Scan Type/ Full Scan settings, see the end of this thread:
>     >     > https://forum.mythtv.org/viewtopic.php?f=36&t=3567  
>     >     > Also described here:
>     >     > https://www.mythtv.org/wiki/Channel_Scanning#Scan_Type_2  
>     >     >
>     >     >
>     >     >
>     >     > _______________________________________________
>     >     > mythtv-dev mailing list
>     >     > mythtv-dev at mythtv.org <mailto:mythtv-dev at mythtv.org> <mailto:mythtv-dev at mythtv.org <mailto:mythtv-dev at mythtv.org>>
>     >     > http://lists.mythtv.org/mailman/listinfo/mythtv-dev
>     >     > http://wiki.mythtv.org/Mailing_List_etiquette
>     >     > MythTV Forums: https://forum.mythtv.org
>     >     >
>     >
>     >     Hi Klaas,
>     >
>     >     My apologies for not mentioning the version, it's v31 from the ubuntu PPA, 2:31.0~master.202002051824.e81d4cc~ubuntu19.10.1 and the distro is lubuntu 19.10 with the lxqt desktop.
>     >
>     >     Both the forum thread and images got me on the right track. Instead of scanning 51 ATSC channels, I'm now scanning 130+ QAM-256 channels. It's still in progress. I'll report back once I've done further testing.
>     >
>     >     But I must say the new settings UI is quite confusing and a departure from how other parts of mythtv's UI work. It hadn't occurred to me that right arrow would do something different than OK. Also, that left arrow behaves differently than ESC.
>     >
>     >     Another point of confusion: after navigating to the scan type page and changing the frequency table and modulation to cable and qam-256, followed by pressing ESC and being asked if I want to save my changes, and saying yes, why am I kicked out of not only the scan type page, but also out of the channel scan page? And when I go back into the scan type page, why are the frequency table and modulation settings back to the original broadcast and 8-vsb settings? Maybe something can be done to explain these points in the help text and the wiki?
>     >
>     >     And, if I've specified us-cable as the channel frequency table in the channel scan page and also in the Global -> Locale page, why does the frequency table in the scan type page always revert to Broadcast, not to mention 8-vsb modulation? Seems to me there's room for improvement in setting more reasonable defaults based on the us-cable setting as well as saving the settings the user specifies?
>     >
>     >     Food for thought?
>     >
>     >
>     > I am glad that the scanning does work for you now. 
>     > About your painful experiences with the GUI, you have made a good summary of what can be improved. It is largely in line with what I have in mind. For the time being I have documented the way it works in the WIki in  https://www.mythtv.org/wiki/Channel_Scanning but I do not expect people to actually read all of this, But it is there for when desperation sets in....
>     > Interesting is also that most of the changes in v31 are "under the hood", the GUI is not really changed since v30 except for a few new options.
> 
>     Hi Klaas,
> 
>     I haven't used mythtv-setup to do channel scans in years. My cable company has only made incremental changes over the years, most of which haven't been important to me. When they do make a change I care about, I've used hdhomerun_config to scan for changes, then crafted sql inserts and updates to the channel and dtv_multiplex tables. I tend to upgrade mythtv with every other release, e.g. 0.27 to 29 and now to 31. And I didn't need to spend much time in mythtv-setup to get 29 configured for my needs.
> 
>     Now I'm getting ready to upgrade to 31 and find it necessary to re-learn how to setup a video source because of the elimination of the DataDirect video source. So, I figured it was also time to see how channel scanning works these days. I had hoped, since the channel table was populated by the new Schedules Direct video source, that maybe the channel scan would be able to update those channel records. But it didn't happen, at least not in my case. Instead, it just added all new channel records for those found in the scan.
> 
>     In my case, the channel scan only found 68 channels when there are actually just over 100 channels. Maybe increasing timeouts would help. But if I've still got to populate the xmltvid columns, then I think I'm better off starting with my production system's channel and dtv_multiplex tables and using the switch_to_json.py script Bill Meek sent me to automate conversion from the old to the new xmltvids.
> 
>     Many Thanks for your help in getting channel scanning working.
>     _______________________________________________
> 
> 
> Thanks for testing on USA cable.  I appreciate getting feedback on this since I can test only on DVB sources. I have made changes over time that also can influence the ATSC behaviour but there has no serious regression testing been done on this.

I'd be happy to help.

> About finding  only 68 channels when there are actually just over 100 channels.
> Could you try with larger timeouts to see if that makes a difference? In v31 the default timeout values have increased from 1000/3000 to 3000/6000 but maybe even bigger timeouts are needed. Note that this new default is only for new capture cards, the timeout values of existing capture card entries are not changed.

I have tested with larger timeouts, 6000/9000. No change and I'm pretty sure timeouts aren't the issue. However, I made some changes to the tuner and network environment to rule out some questionable things. I was using an HDHomeRun DUAL Model: HDHR3-US tuner and it was separated from the backend by 2 gigabit switches and a gigabit router. Now I'm using an HDHomeRun CONNECT Model: HDHR4-2US and it's on the same gigabit switch as the backend.

> It could also be that there are frequencies used that are not being tested with a channel scan. The way it works is that a predefined list of frequencies is scanned but maybe that does not cover all frequencies.
> To figure this out, could you provide me with a list of channels as found by hdhomerun_config and also the output of a mythtv-setup channel scan with the "-v chanscan" option? E.g.  "mythtv-setup -v chanscan 2>&1 | tee ms-20200210-2033.log".

I have some data that may or may not interest you. If not, I'd be happy to perform whatever tests you want. I blew away the database and started over with a backup that I made before running mythfilldatabase. I changed the "capturecard" to the newer and closer HDHR4. I then ran a channel scan with the default 3000/6000 timeouts as well as the default scanning options like "Complete scan data required" enabled. Same results as before, Only 68 channels added, although I can see many more were found looking at the logs, 399 to be exact. Most are encrypted, but many aren't. They weren't added, presumably because they didn't meet the "Complete scan data required". I have the output from mythtv-setup -v general,chanscan (but not the stderr output which only complained about XDG_RUNTIME_DIR) and I'm pretty sure the missing channels were found, but were missing callsigns, for example. So I repeated the scan with only "Unencrypted only" enabled. When the scan completed, I answered in the affirmative to all prompts about adding or updating channels. Now there are 150 channels in the table. Some are duplicates my cable provider broadcasts on different frequencies, but others are unique channels that I'm recording on my production system. Also missing are the channel records that were added by mythfilldatabase and that I blew away when I restored an earlier backup. That's the reason why you may not want what I have so far.

As for an hdhomerun_config channel scan. I did one on Jan 12 on my production system, although I don't recall which HDHR tuner I used, probably the old one. The output clearly shows hdhomerun_config found call signs that mythtv-setup didn't. You may want a more recent scan, just in case my cable provider has changed something.

> About creating new channels vs. updating existing channels. 
> The v31 code does try to update existing channels more than before so that non-scanned channel information like xmltvid is preserved across scans.
> If I understand you correct then you have tested with a channel list created by/from Schedules Direct and then do a mythtv-setup channel scan.
> The expected/wanted behaviour is that the channels in the database are updated, but the actual behaviour is that mythtv-setup creates new channels.
> Correct?

Yes that's correct, but I blew away the database and logs with the schedules direct channels and the scan done at the time.

> To figure if the expected behaviour can be realized I need a dump of the tables channel and dtv_multiplex as created by/from SchedulesDirect and also as created by mythtv-setup. 
> 
> N.B. the default Channel frequency table for ATSC "Full Scan" is now obtained from the Video Source table value or, if that is "default", from the General / Locale settings table value, in master and in fixes/31. Of course it might take a few days before this is in a repository near you.

Sounds good, it will be nice not to have to drill down to the "Scan Type" page to make those changes. Are you also setting the default modulation as well?

> Thanks,
> Klaas.

If time permits, I will redo things: first running mythfilldatabase to populate the channel table with schedules direct channel data, then an hdhomerun_config scan, followed by a mythtv-setup scan, in the hopes of giving you something to look at tomorrow. But if time doesn't permit, I'll send the data I have now later tonight. In either case, I'll send the data offlist.

One other detail. I have since edited the sqlite database from schedules direct to unselect channels I can't get because they're encrypted. I assume it's a non-issue. If it is, I did save a copy of the original schedules direct database.

More offline...