[mythtv] Frames Played issues

[Mark Kendall]

On Mon, 10 Feb 2020 at 00:31, Peter Bennett <pb.mythtv at gmail.com> wrote:
>
> These are commits related to frames played and audio sync
>
> * 0e7e57f9c2 2019/03/06 Peter Bennett : Playback: Fix jerkiness with codec-deinterlaced content

This commit appears to have been superseded by changes I made to the
framerate detection code. Although it did prompt me to look at the
framerate detection again - so I've pushed an unrelated fix for the
initial frame rate supplied to VideoDisplayProfile (we were always
giving it 0 as the frame rate had not been detected yet).

> * 6a5fb07fa1 2019/03/04 Peter Bennett : AVSync2: Use frame timestamps to determine elapsed play time

This is obviously the commit that is causing problems. See below.

> * 4089bf1ff0 2019/02/24 Peter Bennett : Playback: AvSync2 fix fast forward/rewind
>
> * 6b402ca5a3 2019/02/15 Peter Bennett : AVSync2: Improve resync speed and accuracy
>
> * 90b4ec347f 2019/02/15 Peter Bennett : AVSync2: Fix never-ending stutter on Live TV

I think these are mostly just tweaks to the new a/v sync code - some
of which may result from the 'problem' commit.

>
> The change in 6a5fb07fa1 was to address the issue of frame count and elapsed time getting out of sync and it worked surprisingly well, but there are side-effects..

The patch I posted to irc last night (https://pastebin.com/muDLvSGu)
does a few things:-

- remove the use of frame time stamps committed above
- remove David's 'fix' for live tv to fix the previous fix:)
- remove m_framesPlayedExtra - which seems to be redundant now as it
is never set to anything (irrelevant I think to this discussion)
- ensure that m_framesPlayed is incremented when we skip a frame (I
*think* this might be important to the issue at hand)
- added some verbose debugging of the various frames 'played' variables in play.

The debugging highlighted a few things:-

- in the decoder, framesRead runs at twice the frame rate for
everything I've looked at. Probably irrelevant - but looks odd (n.b.
not related to decoder deinterlacing).

- under normal playback, framesPlayed in the decoder - which is
actually the number of decoded frames - runs ahead of framesPlayed in
the player. This is what I was expecting - as framesPlayed in the
decoder actually refers to the last decoded frame - and not the 'head'
of the buffered, decoded frames. So the discrepancy is normally
approximately the number of video buffers. I'm pretty sure this is the
main problem with the commit that calculates the frames played based
on the latest time stamp - and then sets the decoder framesPlayed to
that value - which is not correct - and throws out short seeks etc.

- I have a couple of (I think transcoded) H264 clips that consistently
show a much lower number of decoded frames than played. This is, I
think, a side effect of the decoder holding back reference frames and
reusing them. One clip at least was used to fix an issue with h264
reference frames many years ago. This did prompt me to think that the
wider issue of playback length getting out of kilter might be due to
our handling of the repeat_pict variable in AVFrame - but I cannot
find any clip where repeat_pict is actually set to anything other than
0. I'm not however ruling this out as an issue. As a side note, FFmpeg
only very recently added support for repeat picture handling in their
HEVC parser. (just trying to join the dots on what Peter said - re the
issue seems to be related to transcoded H264 material and HEVC).

- the fix for m_framesPlayed when skipping a frame may be significant.
So on normal startup of playback, where the sync is out, we would skip
X number of frames to get back in sync. The worst case testing here is
when using vaapi decode only - which is I think a combination of VAAPI
startup times and sometimes re-allocating the software frame buffers
for 10bit decodes. framesPlayed in mythplayer is incremented in the
videooutput class (this should probably change - as the videooutput
classes don't actually use it for anything) - but if we skip a frame,
the value is not incremented and if we have a number of skips, there
can be a significant difference between what the player thinks has
been played vs reality. I can imagine that if playback is occasionally
skipping frames to keep up - which would be worse with HEVC, decoder
deinterlacing etc - then over a long, uninterrupted playback period,
the framesPlayed figure could diverge significantly. Would that tie up
with the symptoms seen?

> In my opinion playback should use timestamps rather than frame count to determine elapsed time, but that would be a major change.

As you say, that is a bigger change.

> I believe that David Engel , Mark Spieth and John Pilkington had problems with elapsed time going at the wrong rate before I made the fix.
>
> The problem was worst with the built in deinterlace of the shield and in particular if you were running at increased speed (e.g. 1.5x).

As an aside, I think there is still an opportunity to significantly
improve the initial av sync at the start of playback. At the moment,
we dump audio packets until we have seen the first video packet. The
audio player will then start playback as soon as it has its first
audio packet. The problem is that the video decoder may then have a
sizeable delay while it initialises and then buffers enough decoded
frames. By the time the player has enough frames to work with and we
start showing video frames, the audio can already be well ahead - and
we then have to dump video frames. I think we should look at either
dumping audio packets until there are enough decoded video frames
ready or pause audio playback until the video is ready. This is
relatively minor issue on a significant number of clips - but is a
major issue with some streams where the decoder takes forever to
return the first video frames. I have one clip that is 4k H264 main 10
(i.e. nothing will decode this in hardware) that takes around 10
seconds to show a video frame - but the audio starts after less than a
second. By the time the decoder returns video frames, we discard most
of them:) Obviously I wouldn't expect this clip to play well on most
systems, but it does highlight the problem quite well.

As a way forward, I propose I commit the patch I've posted (without
the debugging). As I see it, an intermittent problem with playback
length is a lesser problem than the various issues we have with the
current code. We can then try and figure out the root cause of the
problem (if it still exists with the fix for skipped frames?).

Thoughts?
regards
Mark