seg fault when reading small PNG files with a palette

[s.kilvington@…]

there is a bug in libavcodec when it decodes small (eg 1x1) PIX_FMT_PAL8 format
images - the get_buffer function avcodec_default_get_buffer doesn't alloc enough
space for the palette entries, so when the palette data gets copied into the
data[1] array it overflows the buffer on the heap and causes a seg fault the
next time you use free/malloc
(actually it does alloc enough space in base[1], but data[1] points to the
middle of the buffer, so it overflows)

this is probably exploitable

here's a patch to fix it:

--- libavcodec/utils.c.orig 2005-11-17 15:13:57.000000000 +0000
+++ libavcodec/utils.c 2005-11-17 15:14:51.000000000 +0000
@@ -325,6 +325,15 @@

const int h_shift= i==0 ? 0 : h_chroma_shift;
const int v_shift= i==0 ? 0 : v_chroma_shift;

+ if(s->pix_fmt == PIX_FMT_PAL8 && i == 1)
+ {
+ buf->base[i] = av_malloc(256 * 4);
+ if(buf->base[i] == NULL)
+ return -1;
+ buf->data[i] = buf->base[i];
+ continue;
+ }
+

FIXME next ensures that linesize= 2^{x uvlinesize, thats needed}

because some MC code assumes it

buf->linesize[i]= ALIGN(pixel_size*w>>h_shift,

STRIDE_ALIGN<<(h_chroma_shift-h_shift));

you can trigger the bug by using avcodec_decode_video to read a 1x1 PNG file
with a palette, calling avcodec_close afterwards causes a seg fault in glibc
inside free

[s.kilvington@…]

patch to fix it

[diego@…]

Fixed by a commit from Michael.