[mythtv] Jitter with time stretch

David Engel david at istwok.net
Sun Jul 15 20:50:56 UTC 2018

On Sat, Jul 14, 2018 at 08:52:30PM -0500, David Engel wrote:
> On Sun, Jul 15, 2018 at 10:14:35AM +1000, Mark Spieth wrote:
> > On 7/15/2018 7:27 AM, Peter Bennett wrote:
> > > Hi David
> > > 
> > > I do not use time stretch, so I do not know what is good and what is
> > > bad. I tested time stretch with the current master version on Linux (no
> > > patches), VAAPI and "-v playback" . With a 30fps recording it can be
> > > speeded up to 2x with no log messages about aout of sync audio. With a
> > > 60fps recording (720p) from Comcast, it shows hundreds of errors, a
> > > bunch of messages saying "Video is 20.0819 frames behind audio (too
> > > slow), dropping frame to catch up." followed by a bunch of messages
> > > saying "Video is 17.7729 frames ahead of audio, doubling video frame
> > > interval to slow down.". This repeats.
> I don't go by the playback, log messages except to try to narrow down
> the problem when there is one.  Rather, I use my eyeballs to see how
> smooth the news ticker is on news channels.  As long as the decoder
> can keep up, it should be generally, pretty smooth with just a little
> bit of jitter as frames need to be dropped or extended.  At 2x, it
> should be completely smooth as every other (60fps) frame gets dropped.
> The generally, pretty smooth is exactly what I see on my main frontend
> (Linux with VDPAU).  That's also what I see on my development system
> (Linux with VAAPI), albeit with a little more jitter.
> > > On the shield, this type of thing also happens, including with 30 fps
> > > interlaced content, perhaps because it is actually shown at 60 fps.
> On my Shields with current master and current Android patch. 720p
> starts to get a little more jittery than expected or wanted at just
> below 2x.  I believe it was good all the way to 2x before when there
> were more video buffers.  I need to retest that.  1080i has always
> been more jittery than expected or wanted at any speed greater than
> 1x.  This is *the* issue keeping from making the Shield my primary
> frontend on my main TV.
> > > On the shield if you set your audio device to NULL it seems to be
> > > perfectly smooth when speeding up to 2x.
> Did you try this with 1080i?  720p worked fine for me.  With 1080i,
> however, playback doesn't actually speed up for speeds greater than
> 1x.  It stays at 1x even though MythTV reports it being faster.  It
> dows slow down for speeds less than 1x, though.  Strange.
> See below for more comments on the NULL audio test.(*)
> > > I suspect what you are seeing is a pre-existing bug, not related to the
> > > shield specifically, but showing up because of the frame doubling that
> > > is happening on the shield.
> > > 
> > > I think the audio speedup may not be perfect, or maybe showing video at
> > > higher rates than 60fps is problematic.
> I disagree about the pre-existing bug part.  Once the processing gets
> to the rendering stage, there is no real difference between rendering
> video and audio that was originally 60fps from that which was
> originally 30fps and then doubled to 60fps.

I found it!  In MythPlayer::SetFrameInterval(), the frame_interval and
avsync_predictor_enabled are getting set wrong because frame_period
passed in is wrong.  It is still set the 30fps equivalent instead of
the 60fps equivalent after the frame doubling in decoding.

After forcing frame_interval and avsync_predictor_enabled to the
correct values, I get smooth playback of my sample, 1080i content on
my 1080p TV at speed upto 1.90x.  At 1.95x, a little bit of
non-predicted adjustment is needed, but it's barely noticeable.  At
2.00x, it a little more noticeable, but really isn't too bad.

On my 4k TV, the results aren't quite as good.  The video gets
noticeably jittry at 1.55x.  It seems the scaling from 1080 to 4k
impacts how fast the decoding and deinterlacing can go.  I wonder what
effect using Surface rendering would have on that.

Note that the above was all done with using the old numbers of buffers
(vbuffers.Init(31, true, 1, 12, 4, 2)).  I'm going to see how far I
can reduce them without degrading the timestretching performance.

Finally, I don't have a proposed fix for the correct setting of
frame_interval and avsync_predictor_enabled yet.  I'll defer to those
of you more familiar with the MythPlayer setup for the time being.  It
probably needs to be tied into the code that Peter added to detect the
frame doubling during decoding.


> (*)I think the NULL audio test points the blame at the Shield's audio,
> or more likely, our handling of audio on it.  Does the audio timing
> need to be adjusted in any way to compensate for the video frame rate
> being doubled.  Also, are we sure we are getting accureate timeing
> information back from the audio subsystem on the Shield?
> FWIW, I've often felt that something was a little off on the audio
> timing.  Sometimes it seems like my short skips ahead and back are a
> few seconds longer or shorter than they should be.  I realized that's
> purely anecdotal, so take it with a huge grain of salt.
> > I notice with timestretch it is sensitive to the number of audio buffers and
> > video buffers to absorb any delta in A/V presentation in the transport
> > stream.
> > Increasing the audio buffering time helps in one direction (A precedes V)
> > and video buffers where V precedes A.
> Peter suggested increasing the audio buffering to 300ms or more a few
> days ago.  I did see some improvement going to 300.  I didn't see much
> improvement, if any, going as high as 2000ms.  I have it set to 1000ms
> currently.
> > As a test, adjust audio sync to the point where you no longer get the Video
> > ahead of audio logs and that will give you an idea of how many buffers are
> > required.
> > Also with mediacodecs pipeline, this will exacerbate this V lagging audio
> > issue.
> > 20 frames is about 750 ms at 30fps. so up the audio extra buffer to 1 sec
> > should fix this.
> I hope to do some more testing on this tomorrow and Monday.
> David
> -- 
> David Engel
> david at istwok.net

David Engel
david at istwok.net

More information about the mythtv-dev mailing list