[mythtv-users] Use of the HDHR Premium TV service with MythTV

[Gary Buhrmaster]

My (work in progress) instructions.

Use of the HDHR Premium TV service with MythTV.pdf
    https://drive.google.com/file/d/19wChqgPjPtt5tw7seUl-zySe-HrH6cJV/view?usp=sharing






The PDF (above) looks/reads better, but here is the text in
the case you do not have access to a PDF viewer app









Use of the HDHR Premium TV service with MythTV


Caveats:


As with everything MythTV related, this is all just a work in
progress, with no guarantees, by just another community member.  This
process worked for me when I started it a few weeks ago, but there is
no way to anticipate all the specialness that is the community (i.e.
caveat emptor), nor changes by others.  There is an old joke that the
presentation is never finished, it just escapes to the screen.
Consider these notes as having escaped.

Advanced users will almost certainly want to do things differently
than described below (I know I do things differently on my production
system) including changing the locations of the scripts and changing
the MythTV users PATH, and even automating service updates, but those
advanced users should be able to translate to their environment.  And
please share improved documentation (perhaps on that wiki page someone
will create).

I highly recommend testing these instructions in a test/development
environment.  While the steps provide some methods to fall back to
what was working before, there are no assurances it may not result in
some breakage (and if you break it, you own it).

Given that the SiliconDust HDHR Premium TV service is new I would
recommend you either join, or at least quite regularly review, the
SIliconDust forums where the service is discussed (there is a topic
area which announces service issues (yes, as a new offering, there
have been some operational service issues) that you can subscribe to
if you wish to receive notifications).

This example presumes you have an OTA tuner which you which to share
with the HDHR Premium TV service (i.e. continue to use for OTA
channels and get access to the HDHR premium channels).  If you use a
dedicated tuner for the HDHR Premium service you could simplify a lot
of the work below (left as an exercise to the reader).

The HDHR OTA tuner(s) and the MythTV BE must exist on the same logical
network (for detection purposes), and the HDHR tuners must have access
to the Internet for both streaming and authorization (so you may need
to adjust any firewall settings). and all must share the same external
address for device association and SiliconDust account validation
purposes.  Most residential networks using a gateway/router for IPv4
with an internal vLAN and using NAT for their external IPv4 address
will meet this requirement.

As part of this process you will migrate to using guide data from
Schedules Direct using the SD-JSON API and using MythTVs XMLTV guide
data import process (the legacy “DataDirect” download does not support
some of the required newer capabilities).  There are other (much more
detailed/extensive) posts on this list about doing this conversion,
and some of the changes one will experience. I would suggest it would
be prudent to review those posts. In particular, migration may result
in some changes in matching of programs for recording, or duplicate
detection,  and because this will be a new videosource, various
advanced recording rules such as “this channel” may need to be
adjusted. You should carefully review your recording rules at the end
of the process to insure they are producing the results you desire.

MythTV's native use of the HDHR tuners does not share (i.e. use
locking) with other apps using the HDHR tuners, nor provide pooling of
the available tuners, nor use the TCP transport protocol, all of which
I consider valuable in my environment which includes other solutions
occasionally attempting to use the tuners.  The HDHR Premium TV
service also requires tuning by virtual channel. The existing MythTV
code has support for tuning by virtual channel for the HDHR devices
(initially for CableCARD devices?), but it does not support the entire
newer set of API functions. This set of instructions/steps converts
your use of your gen 4 or 5 HDHR OTA tuners to using an MythTV
EXTERNAL recorder that provides sharing, pooling, and transport
changes, and uses virtual tuning.  Use of virtual tuning (regardless
of application) does mean that an OTA tuner can be used for only one
channel at a time (it cannot be used to tune/receive two stations that
share a mux (for example, recording CBS and Decades at the same time
using only one tuner)).

The MythTV project devs have suggested that they want to better
support the HDHR Premium TV service.  When that work is occurs, you
should expect (that you will want) to toss all these instructions and
migrate to the better integrated and documented solutions.  Most
importantly because anything regarding the scripts documented here
that has been integrated will probably enter EoS after any such
release (there is no desire on my part to maintain code that has been
supplemented by improved solutions directly from the developers).

The steps will result in you not using any current packaged versions
of a number of (git cloned) source files (the upstream sources have
been revised in some cases to support new or improved capabilities,
and not yet repackaged, or were never packaged by anyone in the first
place), so you will need to regularly remember to 'git pull' to get
the most recent fixes/updates (until/unless the project ships the
new/improved functionality and you migrate to it, or (in the case of
XMLTV) your packager starts to ship updated versions).

While I will try to review specific reports posted to the mythtv-users
email list, it would be useful for others to test, and if it works for
you to state that, and if it doesn't, please state that too, with as
much detail as possible.  Please do consider testing and respond
early, as my ability to test is time limited (I will not be renewing
my subscription to the service as I only subscribed in the first place
to improve the scripts and to test).

For the reason mentioned above, I would encourage some interested
party with ongoing interest to take ownership of these instructions
(and perhaps place them on the wiki when they are appropriately
tested/validated/improved, and have been wiki’fied).



Prereqs to MythTV configuration/setup:


These instructions presume you already have a working MythTV
environment.  If not, follow the docs for that.

You will need a 4th or 5th generation HDHR OTA tuner that supports the
modern tuning methods.  Earlier gen devices are not supported for the
new tuning methods by SiliconDust for the HDHR Premium TV service.

The HDHR tuner will need to have the most recent firmware (20180817 as
of this writing).  Since this is a new service, I would expect you
will need to track newer firmware (which would typically be announced
on the SiliconDust forums).  Note that the newer firmware supports the
ability to upgrade the firmware directly from the device itself off
the “System Status” web page if new versions are released.

You will need to be subscribing the the Schedules Direct guide service
as the guide data for the HDHR Premium TV service is only known to be
available from them at this time.

You will need to subscribe to the SiliconDust HDHR Premium TV service.
Part of the process will be to associate your tuner(s) with your
account (follow the SiliconDust instructions for all of that).  After
you subscribe and associate your tuner, the tuner needs to see the
subscription, which it will do when it calls home next, which may be
up to eight hours from now, but a restart of the tuner forces it to
call home now (power cycle or cli: hdhomerun_config <ip> set
/sys/restart self).  You likely should go to http://my.hdhomerun.com
to look at the tuners that SiliconDust believes are available on your
local network (it also provides a link to get to the tuner if you are
using non-stable DHCP addressing).  I believe it has been reported
that the Microsoft (Edge and/or IE) browsers do not reliably work with
that url (if you are primarily use Windows).

You will have to run a "Channel Listing/Detect Channels" scan on the
device itself (the tuners web page) to obtain a new list of channels
(OTA and the Premium channels) so that the tuner knows what it can
tune, and how to do that tuning.  Review the list to make sure all the
OTA channels you believe you should get are there (if not, first try
another scan. If the signal is marginal sometimes it will miss a
channel (or two)).

To obtain guide data you will need an XMLTV grabber that supports the
Schedules Direct SD JSON API.  All these docs presume you will be
using tv_grab_zz_sdjson_sqlite. XMLTV itself has not made a release
with the latest code, and while some packagers might repackage master,
many do not.  To obtain the latest you will need to issue (in your
/home/mythtv directory)


git clone https://github.com/garybuhrmaster/tv_grab_zz_sdjson_sqlite.git)


Typically, you should first install your distros xmltv-grabber
package, which will pull in the appropriate dependencies, for there
are many that the entire xmltv ecosystem requires. If your distros
xmltv version has not been recently repackaged you may need to
manually install the following perl dependencies: File-Which,
File-HomeDir, JSON-XS, DateTime-Format-SQLite, List-MoreUtils-XS,
Exporter-Tiny, List-MoreUtils, DBD-SQLite, LWP-UserAgent-Determined,
Time-HiRes.  Test to insure you have all the prerequisites by running
/home/mythtv/tv_grab_zz_sdjson_sqlite/tv_grab_zz_sdjson_sqlite --help
to get help information.

You will need the latest mythhdhrrecorder external recorder upstream
version.  To obtain the latest you will need to issue (in your
/home/mythtv directory)


git clone https://github.com/garybuhrmaster/mythhdhrrecorder.git


This external recorder is documented in the MythTV wiki at:
https://www.mythtv.org/wiki/Mythhdhrrecorder.  You may need to install
the python3 requirements which can include: sys, threading, time,
requests, json, re, queue, argparse, socket, inspect, os, io,
datetime, syslog, systemd.journal, natsort.  Test to insure you have
all the prerequisites by running:
/home/mythtv/mythhdhrrecorder/mythhdhrrecorder --help to get help
information.

You will need the latest upstream MythUtil utility set.  To obtain the
latest you will need to issue (in your /home/mythtv directory)


git clone https://github.com/garybuhrmaster/MythUtil.git


The individual scripts have wiki pages at:
https://www.mythtv.org/wiki/MythUtil-Channel-XMLTV-getLineup
https://www.mythtv.org/wiki/MythUtil-Channel-videosource
https://www.mythtv.org/wiki/MythUtil-Channel-HDHR-channelCheck
https://www.mythtv.org/wiki/MythUtil-Channel-HDHR-visibilityCheck and
https://www.mythtv.org/wiki/MythUtil-Channel-visibility.  These
scripts (too) have many python dependencies.  As the important one
with the most dependencies is the ..getLineup one, test to insure you
have all the prerequisites by running:
/home/mythtv/MythUtil/MythUtil-Channgel-XMLTV-getLineup --help to get
help information and install any missing dependencies.

MythTV configuration/setup steps (simplified, this is not a MythTV
tutorial, but intended for those with familiarity to the program):
Most of the steps are one-time only (except for the part about
enjoying your

hard work).


Note: it is presumed that you will be performing all these steps in
your /home/mythtv directory, and you performed the git clones steps
from above in that directory (you will need to adjust accordingly if
the programs/files are elsewhere).


First, run mythtv-setup and delete the input connection  between your
(existing) HDHR tuner and your videosource. This should preserve the
majority of your configuration so you can fairly easily revert to it
by simply adding the input connection (and deleting the input
connection we will create below).

Add the lineups to your Schedules Direct (SDJSON) account.  All on one
command line:


/home/mythtv/tv_grab_zz_sdjson_sqlite/tv_grab_zz_sdjson_sqlite --manage-lineups


select "Add Lineup", via the (guided) prompts select your location,
and add in your OTA lineup

select "Add Lineup", via the (guided) prompts select your location,
and add in the HDHR Premium TV lineup which is either
USA-0000001-CUSTOM for the east coast feed or USA-0000002-CUSTOM for
the west coast feed.  Which one you need to choose depends on your
zipcode, as SiliconDust assigns your location based on your
registration with them and you get that feed. There is no option to
get both feeds. If you think your location should get the east coast
feed, or the west coast feed (i.e. you are in the middle of the
country and you disagree with SiliconDust's choices), you need to work
with them directly.

(if not already done so sometime in the past) select "Initialize DB"
(typically using the defaults)

exit from manage lineups

Add the external recorder to your MythTV system mythtv-setup, add new
capture card, external (blackbox).  The path should be:


/home/mythtv/mythhdhrrecorder/mythhdhrrecorder --devicename 0


For the confident, create another external recorder with the same
path, changing the --devicename to be 1 (each external recorder must
have a unique --devicename due to a different artifact in the recorder
and MythTV).  One recorder per active recording will be required, so
if you have one HDHR 2-tuner device, you need two external recorders,
if you have two 2-tuner devices, you need four.

Add a new videosource named HDHRPremium with the grabber  "No Grabber"
(this is an artifact of needing to start the BE laster)

Connect your new External Recorder(s) to the videosource HDHRPremium

Configure the grabber (this can be done within mythtv-setup under some
scenarios, but we do it here, now, to allow some validation and
testing).  All on one line command line:


/home/mythtv/tv_grab_zz_sdjson_sqlite//tv_grab_zz_sdjson_sqlite
--config-file /home/mythtv/.mythtv/HDHRPremium.xmltv --configure


select the default database location (unless set differently during
the initialize DB function).

select your OTA and HDHR Premium lineups as "yes" to be included (and
no for anything else).

(typically) enable MythTV categories

Test the the grabber by doing a download only (you should see the
progress, and not get any errors).  All on one command line:


/home/mythtv/tv_grab_zz_sdjson_sqlite/tv_grab_zz_sdjson_sqlite
--config-file /home/mythtv/.mythtv/HDHRPremium.xmltv --download-only

Make sure your backend is running (the following steps use the
Services API, which requires a running BE).

Adjust the video source for your new grabber, and configure it.  While
using mythtv-setup is possible, because the cloned repo may make it
hard to select the grabber (depending on how your set up your paths),
we will use an alternative method (this will not be required when
xmltv updates their packages and the distros repackage the
grabber(s)).  All on one command line:


/home/mythtv/MythUtil/MythUtil-Channel-videosource --backend
<your_backend> --update --videosource-name "HDHRPremium" --grabber
/home/mythtv/tv_grab_zz_sdjson_sqlite//tv_grab_zz_sdjson_sqlite
--config /home/mythtv/.mythtv/HDHRPremium.xmltv

Add the channels to your lineup.  Unfortunately, when mixing OTA and
HDHR Premium TV lineups mythfilldatabase currently presumes ATSC
channel numbers are scannable and not auto-added, which they are not
in this virtual channel environment.  Instead we do the following all
on one command line:


/home/mythtv/tv_grab_zz_sdjson_sqlite/tv_grab_zz_sdjson_sqlite -q
--get-lineup --config-file ~/.mythtv/HDHRPremium.xmltv |
/home/mythtv/MythUtil/MythUtil-Channel-XMLTV-getLineup --backend
<your_backend> --videosource-name "HDHRPremium" --add


This command needs only to be done when your local OTA channels or the
HDHR Premium TV service changes their lineup.  In some locations the
FCC repack may delete, add, or rearrange channels. At this time
SiliconDust has not indicated that they have any current plans to
change their lineup, but it is prudent to watch for changes.

Run mythfilldatabase and review its output.  There should be no
significant errors reported (if your OTA lineup has duplicated
channels, and if you have local translators you may get some unknown
xmltvid messages, and this could be "normal").  This may take in
excess of 30 minutes just for the new videosource, especially for a
first run as there is a lot of new guide data to add. Also note that
this will take around 2GB of memory due to the way mythfilldatabase
operates (anyone who wants to refactor that code to do so would likely
be thanked by many).  If your system is very memory constrained, plan
to run mythfilldatabase with --no-allatonce and --refresh-all options
(which can be set in mythtv-setup).

You may want to go into the mythtv-setup channel editor (or mythweb)
to set some channels as invisible (especially if your OTA reception is
spotty, and the Scheduled Direct channel list is different from what
you can actually receive (note that DX’ers may need to combine even
more OTA lineups, beyond the scope of this set of instructions).

Test the new guide and HDHR Premium TV channels (and try a test
recording, perhaps of a classic movie on TCM?).  LiveTV has worked in
a simple test case, but since I almost never use LiveTV, its testing
is very limited.