[mythtv] AVSync2 Improvements

Tue Feb 19 21:18:05 UTC 2019

On 2/20/2019 7:13 AM, David Engel wrote:
> On Tue, Feb 19, 2019 at 01:53:26PM -0500, Peter Bennett wrote:
>> The problems with AVSync that I tried to address with AVSync2 -
>> 1. If the frame rate from ffmpeg is incorrect or the actual fra,e rate rate
>> changes, AVSync takes no heed of that, just uses the specified frame rate.
>> The only reason it actually gets it right in these cases is because it
>> doubles or drops frames to keep in sync with the audio. So if a video is
>> given as 30fps by ffmpeg but the time codes are actually 25 fps it will
>> display as 30fps and then double some frames to keep up with the audio. If
>> there was a video without audio it would effectively ignore video time
>> stamps and play at the wrong rate.
>> 2. When getting the video in sync with audio the only strategies it uses are
>> doubling or halving frame interval for selected frames.
>> 3. If the audio device is unreliable at telling us how much audio is in the
>> buffer (e.g. OpenMax digital audio), avsync produces unacceptably stuttery
>> audio and video. This is addressed in avsync with a complex arrangement of
>> averaging the error over many frames if OpenMax audio is in use.
>>
>> Avsync2 addresses the problems in this way -
>> 1. Uses individual frame timestamps to determine when to display frames,
>> rather than using the frame rate.
>> 2. If audio and video are out of sync adjusts each frame by a small amount
>> (from settings - defaults to 10ms) until it gets in sync. This solves
>> problem 3 above without the complex averaging scheme.
>> 3. Recent change - if audio and video are out by more than 200ms - speeds up
>> the correction so that it corrects over 15 frames at most.
> I understand all that.  Don't get me wrong.  I am *very* much in favor
> of what you're trying to do with avsync2.  For example, my local, ABC
> station broadcasts in 1080i (most ABC stations broadcast in 720p).
> This station is also one of those that exposed the Nvidia/mpeg2
> corruption issue.  I do not know what else this station is doing but
> The Good Doctor will only plya smoothly with avsync2.  Also, all of
> the movies I'm transcoding are being converted to 23.94 fps and they
> need avsync2 to play smoothly.  I want avsync2 to succeed.  It's just
> that the other issues (skips, ff/rew, randoms loss of sync) are
> significant to me that avsync2 is not the clear winner, YET.
>
>> On 2/17/19 2:03 PM, David Engel wrote:
>>> I compared some live TV cases (entering and changing channels) with
>>> and without avsync2.  Without avsync2, there were cases where the
>>> audio and video would sync up fairly quickly.  Most of the time,
>>> though, it would take a little while to sync up.  In those cases it
>>> was comparable to avsync2.  Avsync2 was pretty constant in that it
>>> always took dozens of secongs to sync up.
>> When starting Live TV there is a different problem from the normal playback,
>> that is the system being starved of audio and video data, and stuttering
>> while catching up. I don't know why that would be different between avsync
>> and avsync2, there is not much change in that area.
>>> I noticed a couple of other issues caused by avsync2 that I hadn't
>>> previously tied to it.  I have one, 1080i, h.264 channel that I watch
>>> fairly regularly at >= 1.5x.
>>>
>>> 1. The avsync randomly goes out of phase for short periods (like Mark
>>> Spieth mentioned).  I figured the CPU load on my Shield was on the
>>> edge and just couldn't quite keep at those times.  The problem goes
>>> away completely after I turn off avsync2.  I suspect the logs would
>>> show the same inabiltiy to correct that other logs have shown.
>> Comcast has no 1080i h264 channels. If you can give me more details, a
>> verbose playback log or sample video I can look into this. Does it happen
>> with Linux? Does it happen with mediacodec or software decode or both?
> I'll get you a sample.  I have not tried this on Linux.  I will try
> tonight.  The problem is my only Linux frontend now is my development
> desktop.  It's not exactly comfortable to use for real watching.
>
>>> 2. I have my fast forward speeds set to 10x, 20x, 60x and 180x.  10x
>>> and 20x work normally.  60x and 180x have problems.  A video frame is
>>> displayed and then there is a very long pause before another video
>>> frame is displayed.  Basically those speeds are useless.  Like above,
>>> the problem goes away completely after I turn off avsync2.  I can
>>> provide logs for this if you want.
>> Is this only on shield? Where do you set these amounts? I seldom use fast
>> forward so I am not familiar with this.
> Yes, like above, only tested on Shields.
>
> There is no GUI for setting the ff/rew speeds.  They were left as
> Easter Eggs for the more adventurous to find.  The setting names are
> FFRewSpeed0 through FFRewSpeed7.  Each successive press of ff/rew
> increases or decreases the speed in the appropriate direction.  Speeds
> of 0 are ignored.  Mine are set to 10, 20, 60, 180, 0, 0, 0, 0.
>
>>> I've been using/testing avsync for long enough that I guess I'd gotten
>>> used to it's quirks.  After seeing the old avsync behavior again, I'm
>>> inclined to go back.  Skips and jump (which I do a lot of) feel like
>>> they are slower than with avsync2 but the lack of the other issues
>>> more than makes up for it.
>> General comment - If you are using mediacodec decoder with interlaced
>> content it automatically uses a frame doubling deinterlacer. In this case it
>> is not possible for MythTV to switch to a non-doubling deinterlacer when
>> using speedup, so it may work better to use software decoding in that case,
>> which will switch to a non-doubling deinterlacer when speeding up.
> I have been mainly using software decoding for several months.  The
> main reasons are the Nvidia/mpeg2 issue on one channel, the ff/rew
> pauses that frequiently occur while the mediacodec decoder is reset
> and the inability to timestretch 1089/h.264 at 2x without frequent
> stuttering.
>
> My movie transcodes are h.265 so will be using mediacode decoding more
> going forward.  I also mediacodec yesterday on 720p/mpeg2 at 1.75x and
> 1080i/h.264 at 1.5x.  I hadn't done that in a while and wanted to
> check it out.  I didn't notice any of the ff/rew pauses.  Maybe Nvidia
> fixed something in their Oreo updates.
>
>> Since I do not use speedup or fast forward, my testing in those areas has
>> been minimal. AVSync2 changes to improve smoothness are likely moot when
>> using speedup.
> Probably 2/3 of what I watch is stretched to 1.25, 1.5, 1.75 and even
> 2x.
>
Ive been playing with some mods to avsync2 too.

1. Changed time fetch to a QElapsedTimer. I think this solves the 
sporadic crazyness that Ive seen. This uses Monotonic time on those 
systems that support it so  no NTP non monotonicness

2. I tried mediacodec with 1920x1080 at 25i h264 broadcast shows. This 
plays much nicer and does not have the occasional 1/2 frame rate that 
S/W decoding (4 cpus) + S/W deint has.

3. Smoothing the av delta always. This results in less timing jitter 
which may or may not have any effect.

4. WaitForTime delays using a 2ms usleep chunks. Probably has no effect. 
I thought this may affect SW dec #2 above but probably not.

Mark