[mythtv] Cause of denoise3d black spots

Jeremy Green jrgreen at MIT.EDU
Wed Jan 21 04:22:55 UTC 2009


I think I've figured out what causes the black spots when using 
denoise3d with MMX and compiler optimization.

The MMX code makes use of macros in libavcodec/i386/mmx.h like the 
following:
#define         mmx_r2m(op,reg,mem) \
         asm volatile (#op " %%" #reg ", %0" \
                               : "=m" (mem) \
                               : /* nothing */ )
[...]
#define         movq_r2m(reg,var)           mmx_r2m (movq, reg, var)

In filter_denoise3d.c these are used in code of this sort:
     int16_t wbuf[16];
[...]
             movq_r2m (mm0, wbuf[0]);

The problem here is that since wbuf[0] is of type int16_t, gcc assumes 
that the asm statement will only output to two bytes at the address of 
wbuf[0], whereas the output is actually eight bytes.  When using -O3, 
gcc stores other entries of wbuf[] into registers after the first time 
this asm macro is called but after the second time, only the new value 
for wbuf[0] is retrieved and not wbuf[1], etc.

A solution is to cast to a data type of the correct size:
             movq_r2m (mm0, *((mmx_t *) &wbuf[0]);

I don't know if this should be done in mmx.h (a handful of changes) or 
in filter_denoise3d.c (many changes).

Some cursory searches revealed many other places where care is not taken 
to cast MMX asm input or output operands to the correct size.

I have tested that this does eliminate the black spots using denoise3d.

Jeremy


More information about the mythtv-dev mailing list