An integer overflow is occurred on the argument of calloc in the function lschunks_intrak () of libmpdemux/demux_mov.c

[ylzs]

Version: SVN-r38374-13.0.1

Build command: ../configure --disable-ffmpeg_a && make (compiling with asan)

Summary of the bug: An Integer overflow is found in fucnction lschunks_intrak () which affects mplayer and mencoder. The attached file can reproduce this issue (ASAN-recompilation is needed).

How to reproduce:

1.Command: ./mplayer testcase

2.Result:

MPlayer SVN-r38374-13.0.1 (C) 2000-2022 MPlayer Team

Playing.
libavformat version 58.29.100 (external)
libavformat file format detected.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f1604003600]error reading header
LAVF_header: av_open_input_stream() failed
Quicktime/MOV file format detected.
=================================================================
==10708==ERROR: AddressSanitizer: calloc parameters overflow: count * size (-16777215 * 8) cannot be represented in type size_t (thread T0)
    #0 0x563fa4698362 in __interceptor_calloc (/home/jlx/good_mplayer/asan_mplayer/mplayer+0x343362)
    #1 0x563fa4a61287 in lschunks_intrak /home/jlx/good_mplayer/mplayer/libmpdemux/demux_mov.c:1774:22
    #2 0x563fa4a61287 in lschunks /home/jlx/good_mplayer/mplayer/libmpdemux/demux_mov.c:1305:8
    #3 0x563fa4a61576 in lschunks_intrak /home/jlx/good_mplayer/mplayer/libmpdemux/demux_mov.c
    #4 0x563fa4a61576 in lschunks /home/jlx/good_mplayer/mplayer/libmpdemux/demux_mov.c:1305:8
    #5 0x563fa4a61576 in lschunks_intrak /home/jlx/good_mplayer/mplayer/libmpdemux/demux_mov.c
    #6 0x563fa4a61576 in lschunks /home/jlx/good_mplayer/mplayer/libmpdemux/demux_mov.c:1305:8
    #7 0x563fa4a61576 in lschunks_intrak /home/jlx/good_mplayer/mplayer/libmpdemux/demux_mov.c
    #8 0x563fa4a61576 in lschunks /home/jlx/good_mplayer/mplayer/libmpdemux/demux_mov.c:1305:8
    #9 0x563fa4a62068 in lschunks /home/jlx/good_mplayer/mplayer/libmpdemux/demux_mov.c:1332:6
    #10 0x563fa4a5a88c in mov_read_header /home/jlx/good_mplayer/mplayer/libmpdemux/demux_mov.c:1961:5

==10708==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: calloc-overflow (/home/jlx/good_mplayer/asan_mplayer/mplayer+0x343362) in __interceptor_calloc
==10708==ABORTING

[reimar]

I do not see how this is an MPlayer bug. An integer overflow not handled correctly by libc would be a libc bug (one that has been fixed many many years ago in glibc and has a CVE, so any libc having this issue has not cared about security for a good decade).
We have no intention of working around libc security issues, these are up to the libc maintainers to fix.

[ylzs]

I'm sorry I didn't realize that this is a libc bug in my report. This is the first time I've submitted a bug Reporte to a project

[reimar]

No worries, the ASAN error is really badly written.
The only real hint that the issue is not really a security/integer overflow issues is the part where it says that allocator_may_return_null=1 disables it.
I double-checked that we handle allocation failure properly in this case:

trak->durmap = calloc(len, sizeof(mov_durmap_t));
trak->durmap_size = trak->durmap ? len : 0;

So I'll close this one.