[mythtv] MythUI OSD branch and the way forward

[Mark Kendall]

Morning All,

There has been some discussion on irc recently regarding the
libmythui-osd branch and how to support XVideo (and XvMC) going
forward. It's not a straightforward issue and any resolution is likely
to have wider implications - so it's about time I tried explaining it
in a little more detail and the options I believe we have. Apologies
if some of the following is teaching some (or all!) of you to suck
eggs.

 - Background

The aim of the branch is to draw the OSD using the libmythui-code.
This gives users a consistent look and feel, gives themers
significantly more opportunity to theme the osd (and a consistent
theme format) and should allow us to remove significant amounts of OSD
code. Ideally a win/win/win.

My longer term aims have always been broader. By combining the
rendering code for the UI and video, I'm targetting a situation in
which video is just another UI widget to be toyed with by themers and
normal video playback is a just a fullscreen widget. Picture In
Picture might become just another OSD UI widget and with some
modifications to NuppelVideoPlayer it can become a generic media
player (i.e. music and video) with visualisations built in. I guess
the extreme case is where the media becomes the background (whether it
is livetv, music visualiser, album photo etc) and the UI is alpha
blended on top.

While admittedly it's all just eye candy, I'd like to think that most
people would think it's a good thing! (and for what it's worth, it's
probably not that far away)

 - The Problem

All of the above is, in general, pretty straightforward when using
OpenGL or Direct3D for the video and the UI. So probably no issue on
either Windows or Mac (both of which will also have useable hardware
drivers and most users will have sufficiently powerful GPUs).

On linux VDPAU will handle the rendering basics well enough but in the
future we may want to use OpenGL rendering directly. Any VDPAU capable
card also has sufficient OpenGL horsepower.

Very little of this will work well with XVideo however (XvMC is even
worse) (and in certain situations the hardware drivers just aren't
good enough to give the OpenGL performance we need on linux).

There are essentially three problems with XVideo:-

 - XVideo handles video only (or more precisely images in the YUV
colorspace and by and large planar formats), there is no API to
manipulate the images after they have been converted/scaled and there
is no way of reliably and consistently integrating them with other
hardware acceleration APIs.

 - we by default rely on creating a YUV osd image which is then
blended into the video frame (in software). This image will usually be
a different size to the main UI. As a result we either need to use OSD
specific theme elements or reload/resize the main UI.
(OpenGL/VDPAU/Direct3D all create the OSD at the screen resolution
which should match the UI size - although there will be exceptions
that need to be handled)

 - blending in software is not cheap and as the OSD becomes larger and
more complicated, there is a significant performance hit. For example,
a number of the newer themes use popups with essentially a fullscreen
alphamask (to darken the background) - this is simply not going to
work when trying to play h264 1080 material.

- Options

There are 1001 possible workarounds, hacks and solutions and I've
listed a few options below. I do however feel it's worth pointing out
that any long term solution that includes support for XVideo will be a
compromise for the rest of the UI and hence the rest of the
application.

Option 1 - Do nothing

i.e. do not merge in the branch when complete.

Option 2 - Provide XVideo (and XvMC?) OSD support only via Chromakeying.

The Chromakey OSD operates at the screen resolution and rendering on
screen is not as expensive as the software blend. I think the XVideo
chromakey was dropped by Nvidia for 7xxx series cards and later and is
by and large still available on Intel. Although i don't know about
ATI, this may be a useful short term option for users of 'legacy'
cards (but many of the longer term UI features will not be doable).

Option 3 - Use specific 'legacy' OSD themes for XVideo to avoid the
sizing issue.

Far from ideal from a themers perspective and does not provide a long
term solution.

Option 4 - Force resizing of the main UI when needed.

This is going to be extremely expensive and may simply not be doable
in certain situations (e.g. guidegrid during playback). XVideo
blending will still be expensive and still no future improvements.

Option 5 - Force an XVideo overlay

On systems without chromakey support, try and render the OSD directly
on top of the video frame using the Qt X11 pixmaps directly. I have no
idea whether this will work (it is impossible to test until some
threading issues have been resolved), though I think there will be
blending/compositing issues. Regardless - it will be ugly! There may
also be a solution here involving the X composite extension but given
we are trying to support older hardware...

Option 6 - Drop XVideo/XvMC support and standardise around OpenGL

There are numerous ways of approaching this but I would clean up the
current OpenGL video code, standardise around GLSL for shader support
(UI effects, deinterlacing, colorspace conversion), remove OpenGL code
based around immediate mode (for OpenGL ES compatability), provide
colourspace conversion only support for ARB_fragment_program (for
older OpenGL implementations which will probably not be capable of
using any advanced video shaders anyway) and use OpenGL as the
'frontend' for VDPAU.

It's probably worth dropping Direct3D support as it offers nothing
above and beyond OpenGL and simply represents duplicated code (esp.
around supporting HLSL).

There is then also a separate debate to be had around the Qt painter....

Option 6 - Drop support in stages

Move to Option 2 for the 0.24 release and Option 6 for 0.25.

Option 7 - Maintain a legacy branch

Cut a stable 'Legacy' branch with XVideo support and move to Option 6
for 0.24 (assuming it's ready). Provide X weeks/months/years of legacy
support...


Hopefully that's all sufficiently clear. If not, I'll crawl back into
my hole and brood on it some more:) In case it isn't obvious, I'd like
to move to option 6 as soon as possible - but it's not my decision and
I've spent so long working on it that I may have overlooked the
obvious...

Regards

Mark

PS Nigel - I've deliberately not mentioned OS X quartz/accelerated
support - mostly because I still can't get trunk to compile on Snow
Leopard. I do feel however that OS X support should probably be based
around OpenGL regardless to maintain consistency across platforms.

PPS I've almost finished converting the last of the video code to
using the mythui osd. After that the threading in video playback needs
to be changed signicantly. This will involve sizeable changes to
NuppelVideoPlayer and will take some time. With a fair wind, I'm
hoping that we could consider a merge in around 3-4 weeks time.