mencoder muxer leaks chunks until out of memory transcoding XVID

[rectalogic@…]

When using mencoder to transcode an XVID file to raw AVI, libmpdemux/muxer.c:muxer_write_chunk allocates a 278528 byte chunk each time it is called. After 12923 allocations we run out of memory and fail.

mencoder xvid-memleak.avi -ovc raw -vf format=i420 -oac pcm -o raw.avi

Sample XVID video is on upload.mplayerhq.hu/MPlayer/incoming/xvid-memleak/xvid-memleak.avi

Setting a breakpoint in malloc_error_break shows the following backtrace:

[mpeg4 @ 0x205bc00]Invalid and inefficient vfw-avi packed B frames detected
mencoder(6158) malloc: * mmap(size=278528) failed (error code=12)3027:0]
* error: can't allocate region
* set a breakpoint in malloc_error_break to debug

Breakpoint 6, 0x95512131 in malloc_error_break ()
(gdb) bt
#0 0x95512131 in malloc_error_break ()
#1 0x9550d11f in szone_error ()
#2 0x95434bb6 in allocate_pages ()
#3 0x9543540d in large_and_huge_malloc ()
#4 0x9542c508 in szone_malloc ()
#5 0x9542c2f8 in malloc_zone_malloc ()
#6 0x9542c28c in malloc ()
#7 0x0000f0f1 in muxer_write_chunk (s=0x1560d30, len=276480, flags=16, dts=544.08575241908579, pts=544.08575241908579) at libmpdemux/muxer.c:83
#8 0x0000edaa in put_image (vf=0x1560e80, mpi=0x1561800, pts=-9.2233720368547758e+18) at libmpcodecs/ve_raw.c:127
#9 0x00078898 in vf_next_put_image (vf=0x1560f20, mpi=0x1561800, pts=-9.2233720368547758e+18) at libmpcodecs/vf.c:665
#10 0x00085f4c in put_image (vf=0x1560f20, mpi=0x1533e50, pts=-9.2233720368547758e+18) at libmpcodecs/vf_expand.c:376
#11 0x00078898 in vf_next_put_image (vf=0x1560fc0, mpi=0x1533e50, pts=-9.2233720368547758e+18) at libmpcodecs/vf.c:665
#12 0x00067fbc in filter_video (sh_video=0x1533bf0, frame=0x1533e50, pts=-9.2233720368547758e+18) at libmpcodecs/dec_video.c:416
#13 0x00006d99 in main (argc=10, argv=0xbffff6a0) at mencoder.c:1349

Running mencoder under Instruments on Leopard to detect leaks shows 12923 allocations, each 278528 bytes, 3432.94 Mb cumulative total all with the following stack:

8 mencoder 3432.94 Mb start
7 mencoder 3432.94 Mb main mencoder.c:1349
6 mencoder 3432.94 Mb filter_video libmpcodecs/dec_video.c:416
5 mencoder 3432.94 Mb vf_next_put_image libmpcodecs/vf.c:666
4 mencoder 3432.94 Mb put_image libmpcodecs/vf_expand.c:376
3 mencoder 3432.94 Mb vf_next_put_image libmpcodecs/vf.c:666
2 mencoder 3432.94 Mb put_image libmpcodecs/ve_raw.c:128
1 mencoder 3432.94 Mb muxer_write_chunk libmpdemux/muxer.c:83
0 libSystem.B.dylib 3432.94 Mb malloc

[rectalogic@…]

compressed mencoder output