[mythtv-users] Corrupt recording using HDHR Quatro

[Stephen Worthington]

On Fri, 20 Dec 2019 08:24:46 -0500, you wrote:

>On 12/20/19 7:39 AM, Stephen Worthington wrote:
>> On Fri, 20 Dec 2019 07:07:42 -0500, you wrote:
>>
>>> I was experiencing a lot of corrupted recordings that my Raspberry Pi4
>>> mythtv-backend was making from my HDHR Quatro tuner. I tested by also
>>> recording the same program on another Mythtv backend using a PCIe tuner
>>> card and saw no issues there.
>>>
>>> If you do the math, 4 HDTV OTA mpeg2 streams from a HDHR should be no
>>> more than ~50Mb/s.  The HDHR has a 100Mb/s port. So I thought that it
>>> shouldn't be a problem to have that network traffic on a network that
>>> was all Gbe devices except for the HDHR.
>>>
>>> But I tried a test by putting in a Gbe switch with just the HDHR Quatro,
>>> the RPi4 and an uplink connection to the main switch.  I recorded 4 HDTV
>>> programs at the same time all during prime time last night and no
>>> corruptions.
>>>
>>> In the original configuration the RPi4 and my Shield TV were on a
>>> network segment with a Gbe switch linked to the main home network switch
>>> and the HDHR was on that same main home switch.  So 2 HDTV streams from
>>> Youtube.TV were going through the main switch that the HDHR was on and
>>> one of those HDTV stream went to the Shield TV on the same network
>>> segment as the RPi4 recording all the HDHR traffic.
>>>
>>> I still think the math works out but maybe the 100Mb/s HDHR port
>>> degrades the switches so all traffic is affected???
>>>
>>> Jim A
>> No, having a 100 Mbit/s device on one port of a switch will not
>> degrade any other port.  All that will happen is that the traffic on
>> the 100 Mbit/s connection will be received and transmitted at that
>> rate.  The packets will be buffered in the switch and it will transmit
>> them at 1 gigabit/s to the other ports.
>>
>> What other traffic is happening on your network?  If you are copying
>> things between the RPi4 and other devices on the network at the same
>> time as the HDHR traffic, they all compete for use of the one gigabit
>> port on the RPi4.  Copying a file from a fast hard drive to a fast
>> hard drive is now capable of using up all of the available 1 gigabit/s
>> bandwidth of that port.  So is other similar traffic.  So unless your
>> switch is capable of according the HDHR traffic higher priority on
>> that port, you may be losing some of the UDP packets for the recording
>> traffic from the HDHR.  If a TCP connection loses a packet, it will
>> just be resent, and that process will probably cause the TCP
>> connection to do a bit of contention processing where it will back off
>> its speed.  But if a UDP packet for a recording goes missing, the HDHR
>> has no protocol to resend it and it is lost.  And the recording gets
>> damaged.
>>
>> The HDHR traffic is marked with a DSCP flag saying that the packets
>> are higher priority.  However, that only helps if everything that
>> transmits those packets checks and obeys the DSCP field in the IP
>> header.  If your switch is a normal home user type, it will not be
>> doing any DSCP priority checking, so it will receive the HDHR packets
>> and send them on in the same way it does with all other packets, and
>> they will have to contend for the bandwidth on the transmitter of the
>> switch port that connects to the RPi4.  If, however, you have a
>> commercial grade switch, they almost all support doing DSCP priority,
>> and you should enable that option.  With DSCP priority enabled, the
>> higher priority HDHR packets will be sent, and if there is then
>> insufficient bandwidth for the lower priority packets that are
>> flooding the connection, some of them will be dropped.  But as long as
>> the HDHR traffic remains less than the 1 gigabit/s capability of the
>> port, those packets will never be dropped.
>
>Well my switches are the home category so no prioritizing of DSCP traffic.
>
>I thought Mythtv v30 used the TCP/IP method of talking to the HDHR so 
>the ports could be properly shared between devices on the whole 
>network?  As far as I know there was no other traffic in the house 
>talking to any other HDHR certainly not the Quatro under test as all 
>it's ports were busy with my RPi4.

As far as I know, HDHRs do not have a TCP protocol option, just UDP.
That is because UDP is necessary when you are doing live TV.  If TCP
was used, any retransmission that occurred would cause a pause in the
playback, which is hated by any human watching it.  I do think that it
would be a good idea to have a TCP option for when you are recording
rather than watching live, but it does require rather more buffer
space in the HDHR RAM, so it may not be possible to add it if they
wanted to.  The problem is that since the data keeps arriving at a
high rate from the tuners, there must be enough buffer space to keep a
copy of that long enough to cover the maximum possible TCP error
recovery and retransmission time.  Which is often several seconds. And
if there keep on being errors, the effective TCP transmission speed
can easily drop below the tuner reception speed, and the buffer use
will overflow the available RAM.

I believe HDHRs support sending traffic to multiple destinations at
the same time from the same tuner, but it is done by sending separate
UDP streams to each destination.

There is a TCP/IP protocol that allows the same stream to be sent to
multiple destinations.  It does that by sending out the stream as an
IGMP broadcast stream.  Clients subscribe to the broadcast, which is
available over the whole network.  However, for this to work without
sending the traffic to ports on the switch that are not subscribed to
it, the switch needs to support a feature called IGMP snooping, where
each port sees the subscription packets and decides whether that port
should get a copy of the IGMP broadcast packets sent to it or not. And
IGMP snooping is usually only found on commercial grade switches.

Again, as far as I know, HDHRs do not support IGMP broadcasts.  That
is probably because of the need for a switch with IGMP snooping, which
is not usually available on home networks.

>Your explanation of how it's supposed to work is my understanding as well.
>
>If you look at YouTube.TV's requirements according to them if there is 
>other traffic on the network you need 13+ Mbps for a HD show.  Most 
>evenings we have a Roku and Shield TV both streaming Youtube.TV content. 
>My original configuration would have had the RPi4 receiving ~50Mbps from 
>the HDHR and the Shield receiving 13 Mbps from the internet in the same 
>network segment through the Gbe switch.  So the math still seems like I 
>should not have a problem.
>
>However, putting the HDHR and the RPi4 on the same segment solved the 
>problem, I think. More testing to be done.

I think it only lessens the likelihood of the UDP packets getting
swamped by other traffic, rather than preventing it.  All the other
traffic is coming into the HDHR/RPi4 switch over a single gigabit/s
port, so it is being rate limited by that to only 1 gigabit/s.  When
everything is on the same switch, several different ports can talk to
the RPi4 port at 1 gigabit/s speed, creating the potential for much
greater contention.  An 8 port switch could have up to 7 gigabits/s
destined for the RPi4 port in the worst case.  The bigger the switch,
the worse the worst case it.

>I've never research home network switches before, just went to Amazon 
>and ordered a name brand that's cheap and 1000b/s. Could I have a switch 
>that doesn't buffer the 100b/s traffic like your described?

No.  Switches have to buffer the traffic in order to work at all.  The
buffering is usually a good thing.  It certainly is with speed
conversions.  When a switch buffers packets, it does not have to
buffer the whole packet before it decides to start retransmitting it
out the destination port.  It does need to get all the headers that it
uses to make decisions on.  So it will always need to buffer the
Ethernet header, and a commercial grade switch will normally buffer
the IP header as well.  When the port speeds are matched, that is all
that must be buffered.  For a slow to fast speed conversion, much more
needs to be buffered, as the transmitter will run out of data if it
has to wait for the slower receiver.

Whenever there is more than one packet being received for the same
destination port, the destination port will decide on one to be
transmitted next and that packet does not need any more buffering and
can go out as it comes in.  The others will be buffered until the
first packet completes transmission and then one of them will be
chosen to start transmission while the others continue to be buffered.

This all works very well with TCP connections, as the TCP protocol has
builtin contention and speed control mechanisms that slow down the
transmission rate as necessary.  So if there are say 6 TCP connections
going from gigabit/s receivers to one single gigabit/s transmitter,
TCP will slow down all 6 TCP streams to one sixth speed, so that when
the streams have slowed down properly, the buffers on the transmitter
will no longer overflow and no more packets will be dropped there. UDP
has no corresponding mechanism and UDP packets will continue to be
sent at whatever rate they are being sent at, regardless of how many
of them are being dropped.  If there are TCP connections over the same
transmitter, they will be slowed down to allow the UDP packets through
most of the time, but that will be too late as before that steady
state can be achieved, lots of UDP packets will have been dropped
along with the TCP packets that are being dropped.  And it is only
most of the time - there will still be occasional UDP packet drops, as
the TCP contention mechanism is only an approximate thing - it can not
measure and report the exact problems that are happening at the
overloaded transmitter.  And new traffic starting will again cause
higher packet drop rates until it all adjusts again.

In the context of recording from an HDHR, the long timeframe of a
recording (typically half an hour or more) works against getting a
perfect recording using UDP packets.  All it takes is one full speed
gigabit/s TCP connection to occur to the RPi4 during the recording,
and there will be dropped UDP packets at that time, unless the TCP
connection is very short.  So the proper solution to this problem is
either to have the HDHRs on their own separate Ethernet port on the
MythTV device, or to use the DSCP priority mechanism and have a
commercial grade switch that supports it.  And having the HDHRs on
their own Ethernet port or ports prevents them from being used by
other devices on the network unless you make the MythTV device into a
full routing box.  And even if you do that, only the MythTV box will
be guaranteed not to get dropped packets - anything connecting via the
MythTV box to the HDHRs will have contention problems again.