[mythtv] AVSync2 Improvements

Tue Feb 19 20:13:54 UTC 2019

On Tue, Feb 19, 2019 at 01:53:26PM -0500, Peter Bennett wrote:
> The problems with AVSync that I tried to address with AVSync2 -
> 1. If the frame rate from ffmpeg is incorrect or the actual fra,e rate rate
> changes, AVSync takes no heed of that, just uses the specified frame rate.
> The only reason it actually gets it right in these cases is because it
> doubles or drops frames to keep in sync with the audio. So if a video is
> given as 30fps by ffmpeg but the time codes are actually 25 fps it will
> display as 30fps and then double some frames to keep up with the audio. If
> there was a video without audio it would effectively ignore video time
> stamps and play at the wrong rate.
> 2. When getting the video in sync with audio the only strategies it uses are
> doubling or halving frame interval for selected frames.
> 3. If the audio device is unreliable at telling us how much audio is in the
> buffer (e.g. OpenMax digital audio), avsync produces unacceptably stuttery
> audio and video. This is addressed in avsync with a complex arrangement of
> averaging the error over many frames if OpenMax audio is in use.
> 
> Avsync2 addresses the problems in this way -
> 1. Uses individual frame timestamps to determine when to display frames,
> rather than using the frame rate.
> 2. If audio and video are out of sync adjusts each frame by a small amount
> (from settings - defaults to 10ms) until it gets in sync. This solves
> problem 3 above without the complex averaging scheme.
> 3. Recent change - if audio and video are out by more than 200ms - speeds up
> the correction so that it corrects over 15 frames at most.

I understand all that.  Don't get me wrong.  I am *very* much in favor
of what you're trying to do with avsync2.  For example, my local, ABC
station broadcasts in 1080i (most ABC stations broadcast in 720p).
This station is also one of those that exposed the Nvidia/mpeg2
corruption issue.  I do not know what else this station is doing but
The Good Doctor will only plya smoothly with avsync2.  Also, all of
the movies I'm transcoding are being converted to 23.94 fps and they
need avsync2 to play smoothly.  I want avsync2 to succeed.  It's just
that the other issues (skips, ff/rew, randoms loss of sync) are
significant to me that avsync2 is not the clear winner, YET.

> On 2/17/19 2:03 PM, David Engel wrote:
> > I compared some live TV cases (entering and changing channels) with
> > and without avsync2.  Without avsync2, there were cases where the
> > audio and video would sync up fairly quickly.  Most of the time,
> > though, it would take a little while to sync up.  In those cases it
> > was comparable to avsync2.  Avsync2 was pretty constant in that it
> > always took dozens of secongs to sync up.
> When starting Live TV there is a different problem from the normal playback,
> that is the system being starved of audio and video data, and stuttering
> while catching up. I don't know why that would be different between avsync
> and avsync2, there is not much change in that area.
> > I noticed a couple of other issues caused by avsync2 that I hadn't
> > previously tied to it.  I have one, 1080i, h.264 channel that I watch
> > fairly regularly at >= 1.5x.
> > 
> > 1. The avsync randomly goes out of phase for short periods (like Mark
> > Spieth mentioned).  I figured the CPU load on my Shield was on the
> > edge and just couldn't quite keep at those times.  The problem goes
> > away completely after I turn off avsync2.  I suspect the logs would
> > show the same inabiltiy to correct that other logs have shown.
> Comcast has no 1080i h264 channels. If you can give me more details, a
> verbose playback log or sample video I can look into this. Does it happen
> with Linux? Does it happen with mediacodec or software decode or both?

I'll get you a sample.  I have not tried this on Linux.  I will try
tonight.  The problem is my only Linux frontend now is my development
desktop.  It's not exactly comfortable to use for real watching.

> > 2. I have my fast forward speeds set to 10x, 20x, 60x and 180x.  10x
> > and 20x work normally.  60x and 180x have problems.  A video frame is
> > displayed and then there is a very long pause before another video
> > frame is displayed.  Basically those speeds are useless.  Like above,
> > the problem goes away completely after I turn off avsync2.  I can
> > provide logs for this if you want.
> Is this only on shield? Where do you set these amounts? I seldom use fast
> forward so I am not familiar with this.

Yes, like above, only tested on Shields.

There is no GUI for setting the ff/rew speeds.  They were left as
Easter Eggs for the more adventurous to find.  The setting names are
FFRewSpeed0 through FFRewSpeed7.  Each successive press of ff/rew
increases or decreases the speed in the appropriate direction.  Speeds
of 0 are ignored.  Mine are set to 10, 20, 60, 180, 0, 0, 0, 0.

> > I've been using/testing avsync for long enough that I guess I'd gotten
> > used to it's quirks.  After seeing the old avsync behavior again, I'm
> > inclined to go back.  Skips and jump (which I do a lot of) feel like
> > they are slower than with avsync2 but the lack of the other issues
> > more than makes up for it.
> 
> General comment - If you are using mediacodec decoder with interlaced
> content it automatically uses a frame doubling deinterlacer. In this case it
> is not possible for MythTV to switch to a non-doubling deinterlacer when
> using speedup, so it may work better to use software decoding in that case,
> which will switch to a non-doubling deinterlacer when speeding up.

I have been mainly using software decoding for several months.  The
main reasons are the Nvidia/mpeg2 issue on one channel, the ff/rew
pauses that frequiently occur while the mediacodec decoder is reset
and the inability to timestretch 1089/h.264 at 2x without frequent
stuttering.

My movie transcodes are h.265 so will be using mediacode decoding more
going forward.  I also mediacodec yesterday on 720p/mpeg2 at 1.75x and
1080i/h.264 at 1.5x.  I hadn't done that in a while and wanted to
check it out.  I didn't notice any of the ff/rew pauses.  Maybe Nvidia
fixed something in their Oreo updates.

> Since I do not use speedup or fast forward, my testing in those areas has
> been minimal. AVSync2 changes to improve smoothness are likely moot when
> using speedup.

Probably 2/3 of what I watch is stretched to 1.25, 1.5, 1.75 and even
2x.

David
-- 
David Engel
david at istwok.net