Memory leak when extracting PS from MPEG_TS

[mpla@…]

Significant mencoder memory leak... rapidly exhausts system swap space eventually causing OS to kill the process. Command is to extract a single PS from a file containing an MPEG_TS:

mencoder foo.m2t -tsprog 3 -ovc copy -oac copy-of mpeg -mpegopts format=dvd:tsaf:vbuf_size=500 -o foo.mpg

Runs but does give occasional errors:
ERROR: scr 149.681, dts 0.000, pts 149.649

ERROR: scr 149.690, dts 0.000, pts 149.683

and also the very occasional buffer underflow:
BUFFER UNDEFLOW at stream 0, raising muxrate to 23768 kb/s, delta_scr: 18611

For first 200 to 300 frames or so, process size is reasonable at around:
$ ps axl | grep menc

523 70941 41652 0 96 0 46052 16348 - R+ p0 1:02.74 mencoder foo.m2t -tsprog 3 -ovc copy -oac copy

However, after a while, without obvious error message, the process' memory footprint starts growing rapidly at a rate of 4096 bytes per second:
$ while :; do ps axl | grep menc; sleep 1; done

523 70941 41652 0 96 0 550884 408484 - R+ p0 2:23.35 mencoder foo.m2t -tsprog 3 -ovc copy -oac copy
523 70941 41652 0 96 0 554980 410104 - R+ p0 2:23.94 mencoder foo.m2t -tsprog 3 -ovc copy -oac copy
523 70941 41652 0 96 0 560100 410944 - R+ p0 2:24.48 mencoder foo.m2t -tsprog 3 -ovc copy -oac copy
523 70941 41652 0 96 0 563172 410288 - R+ p0 2:24.89 mencoder foo.m2t -tsprog 3 -ovc copy -oac copy
523 70941 41652 0 96 0 567268 411096 - R+ p0 2:25.55 mencoder foo.m2t -tsprog 3 -ovc copy -oac copy

and, as I said, eventually, the OS kills the process with "out of swap" error.

The input data is an MPEG_TS containing a 1920x1080 HD PS and possibly also some lower definition PSs too. In the example above:
Input #0, mpegts, from 'foo.m2t':

Duration: 00:29:59.71, start: 11864.375867, bitrate: 19389 kb/s
Program 3

Stream #0.0[0x31]: Video: mpeg2video, yuv420p, 1920x1080 [PAR 1:1 DAR 16:9], 65000 kb/s, 29.97 tb(r)
Stream #0.1[0x34]: Audio: liba52, 48000 Hz, 5:1, 448 kb/s

If I increase the mpegopts vbuf_size, I can delay and perhaps even avoid the error all together. In the case of the file here, increasing to 600 allows the PS extraction to complete OK.

I suspect, therefore, there's a bug handling the condition when the vbuf overflows.

[reimar]

Without a reproducible test case I see little chance of getting this fixed.
However a debugging tips for anyone who has this issue:
If you can easily reproduce it, try running mencoder in valgrind, which should show you where exactly the memory was lost.
Otherwise run mencoder (preferably a version with debugging symbols) in gdb (or attach gdb), then when the memory usage starts climbing press "CTRL+C" to get to the gdb prompt, then add e.g. "b malloc", "b calloc", "b memalign" and continue with "c". gdb will then stop at any memory allocation and you should get an idea where all that memory was requested from.