vdpau decoders don't work anymore in mplayer trunk with ffmpeg master

[schalk]

Summary of the bug: vdpau decoders don't work anymore in mplayer trunk with ffmpeg trunk, if video dimensions aren't a multiple of 8(?). That was never a problem before.

How to reproduce:

% mplayer -vo vdpau -vc ffh264vdpau xyz.mp4
MPlayer SVN-r37946-6.3.1 (C) 2000-2017 MPlayer Team

Playing xyz.mp4.
libavformat version 57.76.100 (internal)
libavformat file format detected.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x56585360]Protocol name not provided, cannot determine if input is local or a network protocol, buffers and access patterns cannot be configured optimally without knowing the protocol
[lavf] stream 0: video (h264), -vid 0
[lavf] stream 1: audio (aac), -aid 0, -alang und
VIDEO:  [H264]  852x480  24bpp  23.976 fps  1079.5 kbps (131.8 kbyte/s)
==========================================================================
Forced video codec: ffh264vdpau
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
libavcodec version 57.103.100 (internal)
Selected video codec: [ffh264vdpau] vfm: ffmpeg (FFmpeg H.264 (VDPAU))
==========================================================================
Clip info:
 major_brand: isom
 minor_version: 512
 compatible_brands: isomiso2avc1mp41
 encoder: Lavf56.4.101
Load subtitles in .
==========================================================================
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
AUDIO: 48000 Hz, 2 ch, floatle, 96.0 kbit/3.12% (ratio: 12000->384000)
Selected audio codec: [ffaac] afm: ffmpeg (FFmpeg AAC (MPEG-2/MPEG-4 Audio))
==========================================================================
[AO OSS] audio_setup: Can't open audio device /dev/dsp: No such file or directory
AO: [alsa] 48000Hz 2ch floatle (4 bytes per sample)
Starting playback...
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
VO: [vdpau] 852x480 => 854x480 H.264 VDPAU acceleration 
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
VO: [vdpau] 852x480 => 854x480 H.264 VDPAU acceleration 
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)
Dropping frame with size not matching configured size (852x480 vs 840x480 vs 852x480)

[beastd]

I can reproduce the described problem.

[beastd]

I think the problem is triggered by FFmpeg commit 07596e4 which changes how cropping is handled in FFmpeg's h264 decoder.

If I revert that FFmpeg commit, I can play my test video again.

[reimar]

All these hacks resulted in an implicit API change for the get_buffer2 callback.
Previously you could just always set AVFrame width/height = avcontext width/height adjusted by avcodec_align_dimensions2.
Now for hwaccel it is necessary to instead set it to coded_width/coded_height, because otherwise the apply_cropping function in decode.c results in the cropping being applied TWICE, leading to this error.

[reimar]

Worked around by r37952.

[beastd]

Re-tested here and works for me too.