mencoder segfaults when using software scaler

[dun1313@…]

I'm using mencoder to downscale a 1920x1080 ATSC mpeg2video stream from an HDHomerun using the following command:

$ mencoder -of mpeg -mpegopts format=dvd:tsaf -oac mp3lame -lameopts cbr:br=192 -srate 48000 -af lavcresample=48000 -vf scale=640:360,harddup -ovc lavc -lavcopts vcodec=mpeg2video:keyint=5:vrc_buf_size=1835:vbitrate=4000:vrc_maxrate=9800:vstrict=0 -ofps 30000/1001 -o dstfile.mpg srcfile.mpg

This usually works fine, but I occasionally get segfaults on some streams from a particular channel having a weaker signal. I'm guessing there is corruption in the received stream when this segfault occurs:

a52: CRC check failed!
a52: error at resampling
a52: CRC check failed!
a52: error at resampling
a52: CRC check failed!
a52: error at resampling
Segmentation fault (core dumped)

The segfault does not happen if I remove the "-vf scale" option.

I've uploaded the following "mencoder -v" and "gdb bt/disass/info" output to the incoming directory on the ftp server:

dun1313.mencoder.out (1.3MB)
dun1313.gdb.out (16KB)

In this particular stream, the segfault happens at 99.7s. I've tried to cut down the 6GB file with dd to be as small as possible for a reproducer. I can upload the resulting 180MB file if you want.

System info:

Debian 5.0.3 (Lenny)

$ uname -a
Linux myth 2.6.26-2-686 #1 SMP Wed Aug 19 06:06:52 UTC 2009 i686 GNU/Linux

$ ls -l /lib/libc[.-]*
-rwxr-xr-x 1 root root 1294572 Jan 4 2009 /lib/libc-2.7.so
lrwxrwxrwx 1 root root 11 Feb 1 2009 /lib/libc.so.6 -> libc-2.7.so

$ gcc -v
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.3.2-1.1' --with-bugurl=​file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-targets=all --enable-cld --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.3.2 (Debian 4.3.2-1.1)

$ ld -v
GNU ld (GNU Binutils for Debian) 2.18.0.20080103

$ as --version
GNU assembler (GNU Binutils for Debian) 2.18.0.20080103
Copyright 2007 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `i486-linux-gnu'.

$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Pentium(R) 4 CPU 1.80GHz
stepping : 2
cpu MHz : 1794.572
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm up pebs bts
bogomips : 3594.15
clflush size : 64
power management:

[compn]

yes we would like the 180mb sample file.

[dun1313@…]

It looks like that sample file has been deleted. I'll follow-up again if I'm able to find it, or come across another show that segfaults early in the stream.

[dun1313@…]

I've found the 180MB sample for reproducing this segfault and have uploaded it to the incoming directory on the ftp server as "bug-1583.sample.mpg". I've also verified it still segfaults as of r32957.

[cehoyos]

Works fine with ffmpeg -s 640x360 -an -f null

(gdb) r bug-1583.cut.mpg -vf scale=640:360 -nosound
Starting program: mplayer bug-1583.cut.mpg -vf scale=640:360 -nosound
[Thread debugging using libthread_db enabled]
MPlayer SVN-r32969-4.5.2 (C) 2000-2011 MPlayer Team
MMX supported but disabled
MMX2 supported but disabled
SSE supported but disabled

Playing bug-1583.cut.mpg.
TS file format detected.
VIDEO MPEG2(pid=49) NO AUDIO! NO SUBS (yet)! PROGRAM N. 1
VIDEO: MPEG2 1920x1080 (aspect 3) 29.970 fps 15100.0 kbps (1887.5 kbyte/s)
Load subtitles in ./
Opening video filter: [scale w=640 h=360]
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffmpeg2] vfm: ffmpeg (FFmpeg MPEG-2)
==========================================================================
Audio: no sound
Starting playback...
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
[swscaler @ 0x87f38e0]BICUBIC scaler, from yuv420p to yuv420p using C
VO: [vdpau] 640x360 => 640x360 Planar YV12
Detaching after fork from child process 10128.
V:39651.4 31/ 31 116% 6% 0.0% 0 0
Program received signal SIGSEGV, Segmentation fault.
0x08582c33 in hScale_C (dst=0x8e75330, dstW=640, src=0xf4816020 <Address 0xf4816020 out of bounds>,

srcW=1920, xInc=196608, filter=0x8df0f50, filterPos=0x8df0a20, filterSize=11)
at swscale_template.c:2206

2206 val += ((int)src[srcPos + j])*filter[filterSize*i + j];
(gdb) bt
#0 0x08582c33 in hScale_C (dst=0x8e75330, dstW=640, src=0xf4816020 <Address 0xf4816020 out of bounds>,

srcW=1920, xInc=196608, filter=0x8df0f50, filterPos=0x8df0a20, filterSize=11)
at swscale_template.c:2206

#1 0x085859e3 in hyscale_C (isAlpha=0, pal=0x8de985c, formatConvBuffer=0x8dea080 "", hLumFilterSize=11,

hLumFilterPos=0x8df0a20, hLumFilter=0x8df0f50, xInc=196608, srcW=1920, src=<value optimized out>,
dstWidth=640, dst=0x8e75330, c=0x8de97f0) at swscale_template.c:2396

#2 swScale_C (isAlpha=0, pal=0x8de985c, formatConvBuffer=0x8dea080 "", hLumFilterSize=11,

hLumFilterPos=0x8df0a20, hLumFilter=0x8df0f50, xInc=196608, srcW=1920, src=<value optimized out>,
dstWidth=640, dst=0x8e75330, c=0x8de97f0) at swscale_template.c:2691

#3 0x085890b0 in sws_scale (c=0x8de97f0, src=0xffffa8d0, srcStride=0xffffa920, srcSliceY=464,

srcSliceH=16, dst=0x8ecc99c, dstStride=0x8ecc9ac) at swscale.c:1999

#4 0x081399e1 in scale (sws1=<value optimized out>, sws2=0x0, src=<value optimized out>,

src_stride=0xffffa920, y=464, h=16, dst=0x8ecc99c, dst_stride=0x8ecc9ac, interlaced=0)
at libmpcodecs/vf_scale.c:402

#5 0x081e2def in draw_slice (s=0x8dcf150, src=0x8de05b0, offset=0xffffa990, y=464, type=3,

height=<value optimized out>) at libmpcodecs/vd_ffmpeg.c:527

#6 0x084191e5 in ff_draw_horiz_band (s=0x8dd58b0, y=464, h=16) at mpegvideo.c:2133
#7 0x083d853e in mpeg_decode_slice (s1=0x8dd58b0, mb_y=4002, buf=0xffffabfc, buf_size=53383)

at mpeg12.c:1771

#8 0x083da859 in decode_chunks (avctx=0x8dcf150, picture=0x8dcf050, data_size=0xffffad7c,

buf=0xf57c1008 "", buf_size=72658) at mpeg12.c:2441

#9 0x083dd385 in mpeg_decode_frame (avctx=0x8dcf150, data=0x8dcf050, data_size=0xffffad7c,

avpkt=0xffffad28) at mpeg12.c:2243

#10 0x084a869e in avcodec_decode_video2 (avctx=0x8dcf150, picture=0x8dcf050, got_picture_ptr=0xffffad7c,

avpkt=0xffffad28) at utils.c:705

#11 0x081e21f1 in decode (sh=0x8db82c8, data=0xf57c1008, len=72658, flags=0)

at libmpcodecs/vd_ffmpeg.c:838

#12 0x08114f5f in decode_video (sh_video=0x8db82c8, start=0xf57c1008 "", in_size=72658, drop_frame=0,

pts=39651.421875, full_frame=0xffffae5c) at libmpcodecs/dec_video.c:392

#13 0x0809ab08 in update_video (blit_frame=0xffffcf54) at mplayer.c:2400
#14 0x0809fa2c in main (argc=5, argv=0xffffd014) at mplayer.c:3715
(gdb) disass $pc-32 $pc+32
Dump of assembler code from 0x8582c13 to 0x8582c53:
0x08582c13 <hScale_C+51>: mov 0x20(%ebp),%edx
0x08582c16 <hScale_C+54>: xor %esi,%esi
0x08582c18 <hScale_C+56>: mov 0x24(%ebp),%edi
0x08582c1b <hScale_C+59>: movzwl (%edx,%eax,2),%ebx
0x08582c1f <hScale_C+63>: test %edi,%edi
0x08582c21 <hScale_C+65>: jle 0x8582c56 <hScale_C+118>
0x08582c23 <hScale_C+67>: movswl %bx,%ebx
0x08582c26 <hScale_C+70>: mov -0x14(%ebp),%ecx
0x08582c29 <hScale_C+73>: add 0x10(%ebp),%ebx
0x08582c2c <hScale_C+76>: xor %esi,%esi
0x08582c2e <hScale_C+78>: xor %edx,%edx
0x08582c30 <hScale_C+80>: movswl (%ecx),%eax
0x08582c33 <hScale_C+83>: movzbl (%ebx),%edi
0x08582c36 <hScale_C+86>: inc %edx
0x08582c37 <hScale_C+87>: imul %edi,%eax
0x08582c3a <hScale_C+90>: inc %ebx
0x08582c3b <hScale_C+91>: add %eax,%esi
0x08582c3d <hScale_C+93>: add $0x2,%ecx
0x08582c40 <hScale_C+96>: cmp 0x24(%ebp),%edx
0x08582c43 <hScale_C+99>: jne 0x8582c30 <hScale_C+80>
0x08582c45 <hScale_C+101>: sar $0x7,%esi
0x08582c48 <hScale_C+104>: mov $0x7fff,%eax
0x08582c4d <hScale_C+109>: cmp $0x7fff,%esi
End of assembler dump.
(gdb) info registers
eax 0x1555 5461
ecx 0x8df0f50 148836176
edx 0x0 0
ebx 0xf4816020 -192847840
esp 0xffffa620 0xffffa620
ebp 0xffffa638 0xffffa638
esi 0x0 0
edi 0xb 11
eip 0x8582c33 0x8582c33 <hScale_C+83>
eflags 0x10246 [ PF ZF IF RF ]
cs 0x23 35
ss 0x2b 43
ds 0x2b 43
es 0x2b 43
fs 0x0 0
gs 0x63 99

[cehoyos]

1MB sample for bug 1583