Assertion failed in libmpcodecs/vf.c:259: vf_get_image, Mplayer crashed

[zlai88@…]

The fuzzed file 37-the-mummy3-trailer.mp4 (in the archive at the URL above) caused Mplayer to crash by signal 6 in module: decode_video. However, Valgrind reports zero error. It seems that the test file caused Mplayer to fail an assertion in libmpcodecs/vf.c:259: vf_get_image.

This bug is reproducible on Linux Debian Etch, with the latest Subversion head
mplayer (r27249). The machine used is VMWare Player.

Reproduce as follows:
wget ​http://www.eecs.berkeley.edu/~zhl210/7074-37-3179021022-Leak_PossiblyLost.tgz
tar xzf 7074-37-3179021022-Leak_PossiblyLost.tgz
Valgrind mplayer 37-the-mummy3-trailer.mp4

---

Here is the report by Valgrind:

==4652== Memcheck, a memory error detector.
==4652== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==4652== Using LibVEX rev 1854, a library for dynamic binary translation.
==4652== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==4652== Using valgrind-3.3.1, a dynamic binary instrumentation framework.
==4652== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==4652== For more details, rerun with: -v
==4652==
MPlayer dev-SVN-r27255-4.1.2 (C) 2000-2008 MPlayer Team
CPU: Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz (Family: 6, Model: 15, Stepping: 6)
CPUflags: MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
Compiled for x86 CPU with extensions: MMX MMX2 SSE SSE2

Playing 37-the-mummy3-trailer.mp4.
libavformat file format detected.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x863dc50]error reading header: -1
LAVF_header: av_open_input_stream() failed
Quicktime/MOV file format detected.
[mov] Video stream found, -vid 0
VIDEO: [mp4v] 368x720 24bpp 25.000 fps 0.0 kbps ( 0.0 kbyte/s)
Can't open /dev/fb0: No such file or directory
[fbdev2] Can't open /dev/fb0: No such file or directory
vo_cvidix: No vidix driver name provided, probing available ones (-v option for details)!
[cyberblade] Error occurred during pci scan: Operation not permitted
[mach64] Error occurred during pci scan: Operation not permitted
[mga] Error occurred during pci scan: Operation not permitted
[mga] Error occurred during pci scan: Operation not permitted
[nvidia_vid] Error occurred during pci scan: Operation not permitted
[pm3] Error occurred during pci scan: Operation not permitted
[radeon] Error occurred during pci scan: Operation not permitted
[rage128] Error occurred during pci scan: Operation not permitted
[s3_vid] Error occurred during pci scan: Operation not permitted
[SiS] Error occurred during pci scan: Operation not permitted
[unichrome] Error occurred during pci scan: Operation not permitted
[VO_SUB_VIDIX] Couldn't find working VIDIX driver.
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffodivx] vfm: ffmpeg (FFmpeg MPEG-4)
==========================================================================
Audio: no sound
Starting playback...
VDec: vo config request - 368 x 720 (preferred colorspace: Planar YV12)
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is 1.77:1 - prescaling to correct movie aspect.
VO: [null] 368x720 => 1274x720 Planar YV12

mplayer: libmpcodecs/vf.c:259: vf_get_image: Assertion `h == -1

h >= vf->h' failed.

MPlayer interrupted by signal 6 in module: decode_video

• MPlayer crashed. This shouldn't happen. It can be a bug in the MPlayer code _or_ in your drivers _or_ in your gcc version. If you think it's MPlayer's fault, please read DOCS/HTML/en/bugreports.html and follow the instructions there. We can't and won't help unless you provide this information when reporting a possible bug.

==4652==
==4652== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 21 from 1)
==4652== malloc/free: in use at exit: 438,808 bytes in 2,244 blocks.
==4652== malloc/free: 2,439 allocs, 195 frees, 2,020,655 bytes allocated.
==4652== For counts of detected errors, rerun with: -v
==4652== searching for pointers to 2,244 not-freed blocks.
==4652== checked 3,237,352 bytes.
==4652==
==4652== LEAK SUMMARY:
==4652== definitely lost: 32,880 bytes in 8 blocks.
==4652== possibly lost: 0 bytes in 0 blocks.
==4652== still reachable: 405,928 bytes in 2,236 blocks.
==4652== suppressed: 0 bytes in 0 blocks.
==4652== Rerun with --leak-check=full to see details of leaked memory.

---

Here is the backtrace by gdb:

[Thread debugging using libthread_db enabled]
[New Thread -1209677152 (LWP 5009)]
MPlayer dev-SVN-r27255-4.1.2 (C) 2000-2008 MPlayer Team
CPU: Intel(R) Core(TM)2 Duo CPU T7250 @ 2.00GHz (Family: 6, Model: 15, Stepping: 13)
CPUflags: MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
Compiled for x86 CPU with extensions: MMX MMX2 SSE SSE2
get_path('codecs.conf') -> '/home/user/.mplayer/codecs.conf'
Reading /home/user/.mplayer/codecs.conf: Can't open '/home/user/.mplayer/codecs.conf': No such file or directory
Reading /usr/local/etc/mplayer/codecs.conf: Can't open '/usr/local/etc/mplayer/codecs.conf': No such file or directory
Using built-in default codecs.conf.
Configuration: --enable-debug=3
CommandLine: '-v' '37-the-mummy3-trailer.mp4'
get_path('font/font.desc') -> '/home/user/.mplayer/font/font.desc'
font: can't open file: /home/user/.mplayer/font/font.desc
font: can't open file: /usr/local/share/mplayer/font/font.desc
Using MMX (with tiny bit MMX2) Optimized OnScreenDisplay
Using nanosleep() timing
get_path('input.conf') -> '/home/user/.mplayer/input.conf'
Can't open input config file /home/user/.mplayer/input.conf: No such file or directory
Can't open input config file /usr/local/etc/mplayer/input.conf: No such file or directory
Falling back on default (hardcoded) input config
get_path('37-the-mummy3-trailer.mp4.conf') -> '/home/user/.mplayer/37-the-mummy3-trailer.mp4.conf'

Playing 37-the-mummy3-trailer.mp4.
get_path('sub/') -> '/home/user/.mplayer/sub/'
[file] File size is 6472527 bytes
STREAM: [file] 37-the-mummy3-trailer.mp4
STREAM: Description: File
STREAM: Author: Albeu
STREAM: Comment: based on the code from ??? (probably Arpi)
LAVF_check: QuickTime/MPEG-4/Motion JPEG 2000 format
libavformat file format detected.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x863dc50]error reading header: -1
LAVF_header: av_open_input_stream() failed
Checking for YUV4MPEG2
ASF_check: not ASF guid!
Checking for NuppelVideo
Checking for REAL
Checking for SMJPEG
Checking for Nullsoft Streaming Video
Checking for MOV
ISO: File Type Major Brand: ISO Base Media
ISO: File Type Minor Version: 512
ISO: File Type Compatible Brand #0: mp41
MOV: Movie DATA found!
MOV: Movie header found!
Quicktime/MOV file format detected.
MOV: Movie header (100 bytes): tscale=90000 dur=9079200

---

MOV: Track #0:
MOV: Track header!
tkhd len=84 ver=0 flags=0x0 id=1 dur=9079200 lay=0 vol=0
MOV: Media stream!
MOV: Media header!
MOV: Handler header: /vide () VideoHandler
MOV: unknown handler class: 0x0 ()
MOV: Media info!
MOV: Video header!
MOV: unknown chunk: dinf 28
MOV: Sample info!
MOV: Description list! (cnt:1)
MOV: desc #0: mp4v (136 bytes)
MOV: Sample duration table! (1 blocks)
MOV: Syncing samples (keyframes) table! (229 entries) (ver:0,flags:0)
MOV: Sample->Chunk mapping table! (1 blocks) (ver:0,flags:0)
MOV: Sample size table! (entries=2522 ss=0) (ver:0,flags:134217728)
MOV: Chunk offset table! (2522 chunks)
MOV: unknown chunk: trak 35112
MOV: unknown chunk: ��� -264
stream_seek: WARNING! Can't seek to 0x6239BB !
MOV track #0: 2522 chunks, 2522 samples
pts=9079200 scale=90000 time=100.880
==> Found video stream: 0
[mov] Video stream found, -vid 0
MOV: Found MPEG4 movie Elementary Stream Descriptor atom (66)!
ESDS MPEG4 version: 0 flags: 0x000000
ESDS MPEG4 ES Descriptor (52Bytes):

-> ESId: 0
-> streamPriority: 0

ESDS MPEG4 Decoder Config Descriptor (44Bytes):

-> objectTypeId: 32
-> streamType: 0x11
-> bufferSizeDB: 0x001800
-> maxBitrate: 450.000kbit/s
-> avgBitrate: 450.000kbit/s

ESDS MPEG4 Decoder Specific Descriptor (29Bytes)
ESDS MPEG4 Sync Layer Config Descriptor (1Bytes)

-> predefined: 2

Image size: 368 x 720 (24 bpp)
Display size: 368 x 208
Fourcc: mp4v Codec: 'xvid'

---

MOV: longest streams: A: #-1 (0 samples) V: #0 (2522 samples)
VIDEO: [mp4v] 368x720 24bpp 25.000 fps 0.0 kbps ( 0.0 kbyte/s)
[V] filefmt:7 fourcc:0x7634706D size:368x720 fps:25.000 ftime:=0.0400
get_path('sub/') -> '/home/user/.mplayer/sub/'
using /dev/fb0
Can't open /dev/fb0: No such file or directory
[fbdev2] Using device /dev/fb0
[fbdev2] Can't open /dev/fb0: No such file or directory
vo_cvidix: No vidix driver name provided, probing available ones (-v option for details)!
vidixlib: PROBING: cyberblade
[cyberblade] Error occurred during pci scan: Operation not permitted
vidixlib: PROBING: mach64
[mach64] Error occurred during pci scan: Operation not permitted
vidixlib: PROBING: mga
[mga] probe
[mga] Error occurred during pci scan: Operation not permitted
vidixlib: PROBING: mga_crtc2
[mga] probe
[mga] Error occurred during pci scan: Operation not permitted
vidixlib: PROBING: nvidia
[nvidia_vid] Error occurred during pci scan: Operation not permitted
vidixlib: PROBING: pm3
[pm3] Error occurred during pci scan: Operation not permitted
vidixlib: PROBING: radeon
[radeon] Error occurred during pci scan: Operation not permitted
vidixlib: PROBING: rage128
[rage128] Error occurred during pci scan: Operation not permitted
vidixlib: PROBING: s3
[s3_vid] Error occurred during pci scan: Operation not permitted
vidixlib: PROBING: sis
[SiS] Error occurred during pci scan: Operation not permitted
vidixlib: PROBING: unichrome
[unichrome] Error occurred during pci scan: Operation not permitted
vidixlib: No suitable driver can be found.
[VO_SUB_VIDIX] Couldn't find working VIDIX driver.
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
INFO: libavcodec init OK!
Selected video codec: [ffodivx] vfm: ffmpeg (FFmpeg MPEG-4)
==========================================================================
Audio: no sound
Freeing 0 unused audio chunks.
Starting playback...
[ffmpeg] aspect_ratio: 1.769231
VDec: vo config request - 368 x 720 (preferred colorspace: Planar YV12)
Trying filter chain: vo
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is 1.77:1 - prescaling to correct movie aspect.
VO Config (368x720->1274x720,flags=0,'MPlayer',0x32315659)
VO: [null] 368x720 => 1274x720 Planar YV12
VO: Description: Null video output
VO: Author: Aaron Holtzman <aholtzma@…>

mplayer: libmpcodecs/vf.c:259: vf_get_image: Assertion `h == -1

h >= vf->h' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread -1209677152 (LWP 5009)]
0xb7fea410 in ?? ()
(gdb) bt
#0 0xb7fea410 in ?? ()
#1 0xbfffe038 in ?? ()
#2 0x00000006 in ?? ()
#3 0x00001391 in ?? ()
#4 0xb7e85811 in raise () from /lib/tls/i686/cmov/libc.so.6
#5 0xb7e86fb9 in abort () from /lib/tls/i686/cmov/libc.so.6
#6 0xb7e7efbf in assert_fail () from /lib/tls/i686/cmov/libc.so.6
#7 0x080e7e30 in vf_get_image (vf=0x89a8a28, outfmt=842094169, mp_imgtype=3,

mp_imgflag=4123, w=6, h=208) at libmpcodecs/vf.c:259

#8 0x080e4717 in mpcodecs_get_image (sh=0x89a87d8, mp_imgtype=3,

mp_imgflag=4123, w=368, h=208) at libmpcodecs/vd.c:340

#9 0x08199947 in get_buffer (avctx=0x89a8c90, pic=0x89ad020)

at libmpcodecs/vd_ffmpeg.c:614

#10 0x083421af in alloc_picture (s=0x89a9000, pic=0x89ad020, shared=0)

at mpegvideo.c:180

#11 0x08342581 in MPV_frame_start (s=0x89a9000, avctx=0x89a8c90)

at mpegvideo.c:868

#12 0x08412734 in ff_h263_decode_frame (avctx=0x89a8c90, data=0x89a8ba0,

data_size=0xbfffe454, buf=0x89ab290 "", buf_size=4368) at h263dec.c:615

#13 0x082ed600 in avcodec_decode_video (avctx=0x89a8c90, picture=0x89a8ba0,

got_picture_ptr=0xbfffe454, buf=0x89ab290 "", buf_size=4368) at utils.c:897

#14 0x081991da in decode (sh=0x89a87d8, data=0x89ab290, len=4368, flags=0)

at libmpcodecs/vd_ffmpeg.c:781

---Type <return> to continue, or q <return> to quit---
#15 0x080db7ab in decode_video (sh_video=0x89a87d8, start=0x89ab290 "",

in_size=4368, drop_frame=0, pts=0) at libmpcodecs/dec_video.c:369

#16 0x080795d9 in main (argc=3, argv=0xbffff714) at mplayer.c:2292
(gdb) disass $pc-32 $pc+32
Dump of assembler code from 0xb7fea3f0 to 0xb7fea430:
0xb7fea3f0: add %al,(%eax)
0xb7fea3f2: add %al,(%eax)
0xb7fea3f4: add %al,(%eax)
0xb7fea3f6: add %al,(%eax)
0xb7fea3f8: add %al,(%eax)
0xb7fea3fa: add %al,(%eax)
0xb7fea3fc: add %al,(%eax)
0xb7fea3fe: add %al,(%eax)
0xb7fea400: push %ecx
0xb7fea401: push %edx
0xb7fea402: push %ebp
0xb7fea403: mov %esp,%ebp
0xb7fea405: sysenter
0xb7fea407: nop
0xb7fea408: nop
0xb7fea409: nop
0xb7fea40a: nop
0xb7fea40b: nop
0xb7fea40c: nop
0xb7fea40d: nop
0xb7fea40e: jmp 0xb7fea403
0xb7fea410: pop %ebp
---Type <return> to continue, or q <return> to quit---
0xb7fea411: pop %edx
0xb7fea412: pop %ecx
0xb7fea413: ret
0xb7fea414: nop
0xb7fea415: nop
0xb7fea416: nop
0xb7fea417: nop
0xb7fea418: nop
0xb7fea419: nop
0xb7fea41a: nop
0xb7fea41b: nop
0xb7fea41c: nop
0xb7fea41d: nop
0xb7fea41e: nop
0xb7fea41f: nop
0xb7fea420: pop %eax
0xb7fea421: mov $0x77,%eax
0xb7fea426: int $0x80
0xb7fea428: nop
0xb7fea429: nop
0xb7fea42a: nop
0xb7fea42b: nop
0xb7fea42c: nop
---Type <return> to continue, or q <return> to quit---
0xb7fea42d: nop
0xb7fea42e: nop
0xb7fea42f: nop
End of assembler dump.
(gdb) info all-registers
eax 0x0 0
ecx 0x1391 5009
edx 0x6 6
ebx 0x1391 5009
esp 0xbfffe020 0xbfffe020
ebp 0xbfffe038 0xbfffe038
esi 0xbfffe0d8 -1073749800
edi 0xb7f89ff4 -1208442892
eip 0xb7fea410 0xb7fea410
eflags 0x200202 [ IF ID ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
st0 0 (raw 0x00000000000000000000)
st1 0 (raw 0x00000000000000000000)
st2 0 (raw 0x00000000000000000000)
st3 0 (raw 0x00000000000000000000)
st4 0 (raw 0x00000000000000000000)
st5 0 (raw 0x00000000000000000000)
st6 8 (raw 0x40028000000000000000)
---Type <return> to continue, or q <return> to quit---
st7 1.76923072338104248046875 (raw 0x3fffe276270000000000)
fctrl 0x37f 895
fstat 0x20 32
ftag 0xffff 65535
fiseg 0x73 115
fioff 0x80e4d42 135155010
foseg 0x7b 123
fooff 0x0 0
fop 0x5d8 1496
xmm0 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},

v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,

0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},

uint128 = 0x00000000000000000000000000000000}

xmm1 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},

v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,

0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},

uint128 = 0x00000000000000000000000000000000}

xmm2 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},

v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,

0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},

uint128 = 0x00000000000000000000000000000000}

xmm3 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},

v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,

---Type <return> to continue, or q <return> to quit---

0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},

uint128 = 0x00000000000000000000000000000000}

xmm4 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},

v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,

0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},

uint128 = 0x00000000000000000000000000000000}

xmm5 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},

v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,

0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},

uint128 = 0x00000000000000000000000000000000}

xmm6 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},

v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,

0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},

uint128 = 0x00000000000000000000000000000000}

xmm7 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},

v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,

0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},

uint128 = 0x00000000000000000000000000000000}

mxcsr 0x1f80 [ IM DM ZM OM UM PM ]
mm0 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,

0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}

mm1 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,

0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}

---Type <return> to continue, or q <return> to quit---
mm2 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,

0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}

mm3 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,

0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}

mm4 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,

0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}

mm5 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,

0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}

mm6 {uint64 = 0x8000000000000000, v2_int32 = {0x0, 0x80000000},

v4_int16 = {0x0, 0x0, 0x0, 0x8000}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0,

0x0, 0x80}}

mm7 {uint64 = 0xe276270000000000, v2_int32 = {0x0, 0xe2762700},

v4_int16 = {0x0, 0x0, 0x2700, 0xe276}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0,

0x27, 0x76, 0xe2}}

---

This bug was found as part of the SUPERB-TRUST 2008 project.

[zlai88@…]

Sorry, the bug was tested and reproducible in Mplayer version r27255 not r27249.