An Integer overflow in Mplayer-1.4-8’s libmpcodecs/ad_hwmpa.c

[Taolaw]

Summary of the bug:An Integer overflow in Mplayer-1.4-8’s libmpcodecs/ad_hwmpa.c
How to reproduce:

In line 136 of ad_hwmpa.c, When the memset function is called , since the third argument of this function is a unsigned int , the argument passed in is a signed int , which causes Integer overflow when tot2 < tot .

```
	memset(&buf[tot], 0, tot2-tot);
	return tot2;
```

gdb-peda$ r -vo null -ao null Integer-overflow 
Starting program: /root/tmp/crash/picture/mplayer -vo null -ao null Integer-overflow
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
MPlayer 1.4-8 (C) 2000-2019 MPlayer Team

Playing Integer-overflow.
libavformat version 58.27.102 (internal)
libavformat file format detected.
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]big_values too big
[mp2 @ 0x555556465000]Error while decoding MPEG audio frame.
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]big_values too big
[mp2 @ 0x555556465000]Error while decoding MPEG audio frame.
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]big_values too big
[mp2 @ 0x555556465000]Error while decoding MPEG audio frame.
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mpeg @ 0x555556449d60]decoding for stream 0 failed
[mpeg @ 0x555556449d60]start time for stream 0 is not set in estimate_timings_from_pts
[mpeg @ 0x555556449d60]Could not find codec parameters for stream 0 (Audio: mp2, stereo, s16p, 40 kb/s): unspecified sample rate
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[lavf] stream 0: audio (mp2), -aid 0
Load subtitles in ./
==========================================================================
Requested audio codec family [mpg123] (afm=mpg123) not available.
Enable it at compilation.
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
libavcodec version 58.51.100 (internal)
[mp2float @ 0x555556465000]Header missing
[mp2float @ 0x555556465000]Header missing
[mp2float @ 0x555556465000]Header missing
[mp2float @ 0x555556465000]Header missing
[mp2float @ 0x555556465000]Header missing
[mp2float @ 0x555556465000]Header missing
Unknown/missing audio format -> no sound
ADecoder init failed :(
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
[mp2 @ 0x555556465000]big_values too big
[mp2 @ 0x555556465000]Error while decoding MPEG audio frame.
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]big_values too big
[mp2 @ 0x555556465000]Error while decoding MPEG audio frame.
[mp2 @ 0x555556465000]Header missing
[mp2 @ 0x555556465000]Header missing
Unknown/missing audio format -> no sound
ADecoder init failed :(
Requested audio codec family [mad] (afm=libmad) not available.
Enable it at compilation.
Opening audio decoder: [hwmpa] MPEG audio pass-through (fake decoder)
AUDIO: 32000 Hz, 2 ch, mpeg2, 40.0 kbit/3.91% (ratio: 5000->128000)
Selected audio codec: [hwmpa] afm: hwmpa (MPEG audio pass-through for hardware MPEG decoders)
==========================================================================
AO: [null] 32000Hz 2ch mpeg2 (1 bytes per sample)
Video: no video
Starting playback...
A:   0.8 (00.7) of 2.5 (02.5) ??,?% 
Program received signal SIGSEGV, Segmentation fault.
[----------------------------------registers-----------------------------------]
RAX: 0x0 
RBX: 0x420 
RCX: 0xffffffffffdfe160 
RDX: 0x555556eed280 --> 0x0 
RSI: 0x0 
RDI: 0x5555570ef000 
RBP: 0x300 
RSP: 0x7fffffffcdf8 --> 0x5555556f8066 (<decode_audio+198>:	add    rsp,0x48)
RIP: 0x7ffff78b5b0d (<__memset_avx2_erms+13>:	rep stos BYTE PTR es:[rdi],al)
R8 : 0xffffffffffffffe0 
R9 : 0x420 
R10: 0x7fffffffce38 --> 0xb000000001 
R11: 0x555556d43980 --> 0x15d0d0312fba2a70 
R12: 0x7fffffffce3c --> 0xb0 
R13: 0x555556d51a00 --> 0x555556d36cb0 --> 0x0 
R14: 0x555556eed280 --> 0x0 
R15: 0x7fffffffce34 --> 0x100000180
EFLAGS: 0x10282 (carry parity adjust zero SIGN trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0x7ffff78b5b03 <__memset_avx2_erms+3>:	mov    rcx,rdx
   0x7ffff78b5b06 <__memset_avx2_erms+6>:	movzx  eax,sil
   0x7ffff78b5b0a <__memset_avx2_erms+10>:	mov    rdx,rdi
=> 0x7ffff78b5b0d <__memset_avx2_erms+13>:	rep stos BYTE PTR es:[rdi],al
   0x7ffff78b5b0f <__memset_avx2_erms+15>:	mov    rax,rdx
   0x7ffff78b5b12 <__memset_avx2_erms+18>:	ret    
   0x7ffff78b5b13:	data16 nop WORD PTR cs:[rax+rax*1+0x0]
   0x7ffff78b5b1e:	xchg   ax,ax
[------------------------------------stack-------------------------------------]
0000| 0x7fffffffcdf8 --> 0x5555556f8066 (<decode_audio+198>:	add    rsp,0x48)
0008| 0x7fffffffce00 --> 0x0 
0016| 0x7fffffffce08 --> 0x7fff00000420 
0024| 0x7fffffffce10 --> 0x7fff00000531 
0032| 0x7fffffffce18 --> 0x555556eed280 --> 0x0 
0040| 0x7fffffffce20 --> 0xff000000000000ff 
0048| 0x7fffffffce28 --> 0x100000420 
0056| 0x7fffffffce30 --> 0x18000001f40 
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Stopped reason: SIGSEGV
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
151	../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: 

gdb-peda$ bt
#0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
#1  0x00005555556f8066 in decode_audio (sh=<optimized out>, buf=<optimized out>, 
    minlen=<optimized out>, maxlen=<optimized out>) at libmpcodecs/ad_hwmpa.c:136
#2  0x00005555556f9229 in filter_n_bytes (len=0xa00, sh=0x555556d51a00)
    at libmpcodecs/dec_audio.c:409
#3  mp_decode_audio (sh_audio=0x555556d51a00, minlen=0x400) at libmpcodecs/dec_audio.c:493
#4  0x000055555569e843 in fill_audio_out_buffers () at mplayer.c:2168
#5  main (argc=<optimized out>, argc@entry=0x6, argv=<optimized out>, argv@entry=0x7fffffffe088)
    at mplayer.c:3781
#6  0x00007ffff777d09b in __libc_start_main (main=0x55555569c580 <main>, argc=0x6, 
    argv=0x7fffffffe088, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7fffffffe078) at ../csu/libc-start.c:308
#7  0x00005555556a0c3a in _start () at mplayer.c:2242

Patches should be submitted to the mplayer-dev-eng mailing list and not this bug tracker.

[Taolaw]

crash test case

[reimar]

Fixed in r38224.