Opened 12 years ago

Last modified 12 years ago

#806 new defect

mencoder silently terminates early when used in pipes

Reported by: bjung50169@… Owned by: reimar
Priority: important Component: core
Version: unspecified Severity: major
Keywords: Cc:
Blocked By: Blocking:
Reproduced by developer: Analyzed by developer:

Description

I've tried using mencoder in a chain of piped commands to encode my first movie, but the mencoder processes seem to detect/signal end-of-stream much too early (less than 1 minute through the movie, always at the same point), and so the encoder at the end of the chain terminates as if all was OK. This happens with ffmpeg and ffmpeg2theora.

First hint at the problem came to me when running this (MPLAYER_VERBOSE is set to -99):

some_movie_feeder_that_needs_cycles_and_buffers_frames | mencoder -msglevel all=-1 - -demuxer rawvideo -rawvideo fps=25:w=768:h=576:format=rgb24 -vf format=444p -o - -ovc raw | mencoder -msglevel all=-1 - -vf format=yv12 -o - -ovc raw | ffmpeg -y -i - test.mpeg

If all the programs happen not to be already cached in memory, ffmpeg complains that it can't open the input file, but retrying a second time usually makes it start encoding the input! The same happens with ffmpeg2theora (which initially complains about the input format being bad). Clearly something is already fishy here.

Note that in the example there are 2 format conversions done, this is because the first conversion is also done when using denoise3d, but the bug happens also without denoise3d, so I hoped to simplify the example a bit more. (It also seems that I can't do denoise3d+conversion in the same filter chain, but that's another problem).

Now more hints to the problem, running this:

mencoder -msglevel all=-1 /dev/zero -demuxer rawvideo -rawvideo fps=25:w=768:h=576:format=rgb24 -vf format=444p -o - -ovc raw | mencoder -msglevel all=-1 - -vf format=yuy2 -o - -ovc raw | ( while true; do dd bs=1000000 count=100; done ) > /dev/null

You can see the last stage in the pipeline regularly printing messages when consuming its input stream. The problem happens when you let it run long enough. With the frame size in the example, it runs happily here for about 30 seconds, then seems to pause (the transferred data, not the CPU usage), then after about 1 minute, dd continuously says it has input/output 0 bytes: the input stream has been closed and the mencoder processes exited. With smaller frames it may happen that the whole thing runs for 30 seconds, pauses 10 seconds, continues for 1 or 2 minutes and then ends like explained above.

Now I don't really know where in the program the bug(s) might be, or if it may be libavcodec fault.

Just in case, here's the output I get (AMD K7 processor):

SwScaler?: reducing / aligning filtersize 1 -> 4
SwScaler?: reducing / aligning filtersize 5 -> 4
SwScaler?: reducing / aligning filtersize 1 -> 1
SwScaler?: reducing / aligning filtersize 1 -> 1
[swscaler @ 0x836c4d0]SwScaler?: BICUBIC scaler, from rgb24 to yuv444p using MMX2
[swscaler @ 0x836c4d0]SwScaler?: using 4-tap MMX scaler for horizontal luminance scaling
[swscaler @ 0x836c4d0]SwScaler?: using 4-tap MMX scaler for horizontal chrominance scaling
[swscaler @ 0x836c4d0]SwScaler?: using 1-tap MMX "scaler" for vertical scaling (YV12 like)
[swscaler @ 0x836c4d0]SwScaler?: 768x576 -> 768x576
SwScaler?: reducing / aligning filtersize 1 -> 4
SwScaler?: reducing / aligning filtersize 9 -> 8
SwScaler?: reducing / aligning filtersize 1 -> 1
SwScaler?: reducing / aligning filtersize 1 -> 1
[swscaler @ 0x836c4d0]SwScaler?: BICUBIC scaler, from yuv444p to yuyv422 using MMX2
[swscaler @ 0x836c4d0]SwScaler?: using 4-tap MMX scaler for horizontal luminance scaling
[swscaler @ 0x836c4d0]SwScaler?: using 8-tap MMX scaler for horizontal chrominance scaling
[swscaler @ 0x836c4d0]SwScaler?: using n-tap MMX scaler for vertical scaling (BGR)
[swscaler @ 0x836c4d0]SwScaler?: 768x576 -> 768x576

I'm using Debian Testing, plus the packages (very recently) compiled from CVS by Christian Marillat on Debian Multimedia when those don't exist in the official Debian (mencoder + libs).

That bug is a great pain since using huge temp files as a workaround is nearly impossible given the files and disks sizes. Except maybe if I knew how to do denoise3d + yuy2 conversion in the same filter chain it could work since it seems the bug only happens when there are *two* chained mencoder processes.
Or I'll have to code my own 444p->yuy2 converter.

Of course I'm available for further tests to run on my machine if needed.

Thanks.

Change History (4)

comment:1 Changed 12 years ago by compn

what error message do you get with -vf format=yv12,denoise3d ?
also... do you not like hqdn3d better?

or maybe mkfifo stream.yuv | mplayer -vf denoise3d -vo yuv4mpeg tv:// -tv ... | mencoder stream.yuv -options... might work better for you ?

devs reccomend using named pipes.

also, you just want to output to mpeg? why not mencoder -ovc lavc -lavcopts vcodec=mpeg2video -of mpeg -o done.mpg ?

comment:2 Changed 12 years ago by bjung50169@…

(In reply to comment #1)

what error message do you get with -vf format=yv12,denoise3d ?

Ah! That seems to work correctly now. Maybe I thought before that this filter chain didn't work because of the encoder failing to open its input when not cached (that problem still happens). Add to that the need to hide all console messages when using pipes and you can see the mess it was debugging my encoding script.

But it still doesn't work with yuy2 (see bug #581):

The selected video_out device is incompatible with this codec.
Try appending the scale filter to your filter list,
e.g. -vf spp,scale instead of -vf spp.
Cannot find codec matching selected -vo and video format 0x52474218.

yuy2 is my preferred format, and what I hoped to make work. yv12 loses too much chroma resolution and is a mess with interlaced frames, so I'd prefer leaving that conversion to the encoder itself. However in my case, first the movies are progressive, and second there would be difficulties using denoise3d on interlaced stuff anyway, so yv12 is OK for me now.

But... Using "-vf format=yv12,hqdn3d" does what it should, except the encoding process now terminates after about 8 minutes through the movie, so there's still a problem.

also... do you not like hqdn3d better?

Of course, but I didn't want to complicate matters more.

or maybe mkfifo stream.yuv | mplayer -vf denoise3d -vo yuv4mpeg tv:// -tv ... |
mencoder stream.yuv -options... might work better for you ?

I tried my test involving 'dd' using 3 intermediate named pipes (and directly referring to them as files), and the behaviour is the same except that at the last stage all involved processes are stopped indefinitely.

Using two intermediate named pipes when encoding the movie also terminates after 8 minutes, as does the unnamed pipe version.

devs reccomend using named pipes.

Why? I hope not as a permanent solution to the console messages going to stderr/stdout mess (#794). Why complicate something that's so easy, clean, and basic on Unix-like systems?

also, you just want to output to mpeg? why not mencoder -ovc lavc -lavcopts
vcodec=mpeg2video -of mpeg -o done.mpg ?

That works, with this message at the start:

Pos: 10.0s 250f ( 0%) 22.10fps Trem: 0min 0mb A-V:0.000 [899:0]
BUFFER UNDEFLOW at stream 0, raising muxrate to 1980 kb/s, delta_scr: 223418

BUFFER UNDEFLOW at stream 0, raising muxrate to 2178 kb/s, delta_scr: 203107

BUFFER UNDEFLOW at stream 0, raising muxrate to 2395 kb/s, delta_scr: 184643

But in fact I just want to encode with ffmpeg2theora. And I prefer using externally (outside mplayer) packaged encoders with pipes if possible. IMO that's much simpler and flexible, less bloated, and more robust on Unix-like systems.

As a last resort workaround I can always try merging the 8-minute parts with ogmmerge and hope that it's precise enough to not cause later problems with audio sync. Or encode with a codec inside mencoder...

To summarise, the problems I reported are still there, even with named pipes or only one mencoder process.

Thanks.

comment:3 Changed 12 years ago by bjung50169@…

After more tests done this Sunday, I think I've finally found what's the problem! It's libavformat fault: url_read_packet() in aviobuf.c incorrectly thinks that url_read(), which uses read() directly, never returns less bytes than asked, which leads the library to suppose it has read an entire frame when it hasn't! A test on my original problematic piped commands shows that after about 1 Gb were read, the last component starts to get very frequent 4-byte reads, which could explain the "pause". Premature EOF seems also to happen when exactly 232 bytes have been read by the encoder. I have no explanation for that. Maybe it'll disappear when the libavformat bug above is fixed.

I'll report this in the ffmpeg category.

comment:4 Changed 12 years ago by bjung50169@…

Well, at last I found the real culprit for the main problem above. After some tests and a hard look at libavformat source code, I've come to realise that it can't read its own raw (avi) video when streamed through a pipe. The library needlessly and carelessly seeks the stream when it finds idx1 and RIFF chunks, which are placed about every Gb. It doesn't even check if the stream is seekable, so that leads to premature EOFs, infinite loops and God knows whatever else in the video readers.

What a mess. I've programmed a very simple utility that can make an avi yv12 -> yuv4mpeg conversion in a pipe. It works great now. I can do "movie_feeder | mencoder vf=denoise3d | utility | ffmpeg2theora" in one go.

I'll file 2 bug reports about that in the ffmpeg category when I have more time.

Note: See TracTickets for help on using tickets.