[mythtv] Cause of denoise3d black spots
Jeremy Green
jrgreen at MIT.EDU
Wed Jan 21 04:22:55 UTC 2009
I think I've figured out what causes the black spots when using
denoise3d with MMX and compiler optimization.
The MMX code makes use of macros in libavcodec/i386/mmx.h like the
following:
#define mmx_r2m(op,reg,mem) \
asm volatile (#op " %%" #reg ", %0" \
: "=m" (mem) \
: /* nothing */ )
[...]
#define movq_r2m(reg,var) mmx_r2m (movq, reg, var)
In filter_denoise3d.c these are used in code of this sort:
int16_t wbuf[16];
[...]
movq_r2m (mm0, wbuf[0]);
The problem here is that since wbuf[0] is of type int16_t, gcc assumes
that the asm statement will only output to two bytes at the address of
wbuf[0], whereas the output is actually eight bytes. When using -O3,
gcc stores other entries of wbuf[] into registers after the first time
this asm macro is called but after the second time, only the new value
for wbuf[0] is retrieved and not wbuf[1], etc.
A solution is to cast to a data type of the correct size:
movq_r2m (mm0, *((mmx_t *) &wbuf[0]);
I don't know if this should be done in mmx.h (a handful of changes) or
in filter_denoise3d.c (many changes).
Some cursory searches revealed many other places where care is not taken
to cast MMX asm input or output operands to the correct size.
I have tested that this does eliminate the black spots using denoise3d.
Jeremy
More information about the mythtv-dev
mailing list