mplayer / mencoder fail to link when compiling with -flto

[Mayer Jocelyn]

Summary of the bug:
How to reproduce:
» Make sure '-flto' is set in build flags
» try to build mplayer with --enable-libass
» mplayer & mencoder link would fail with multiple undefined reference to sse_int32_map_factor' and sse_int32_80h' in libmpcodecs/vf_ass.c

This has been reproduced with (old) 1.3.0 version and newer 1.4 version.
As far as I can see, no fix have been committed for this in trunk.

A workaround is available in gentoo linux bugtracking system:
​https://bugs.gentoo.org/650458 with a patch available, which follow for easier potential fix.

This patch applied make the link succesful.

[Mayer Jocelyn]

Patch submitted to gentoo bugtracking system in order to fix this link issue

[reimar]

The analysis of the issue in the gentoo bug is at least incomplete. The symbols are marked attribute((used)) so gcc has been explicitly told it has to preserve them and very much knows it has to keep them.

[reimar]

I would consider this a compiler bug: It has been told the symbol is needed yet it does not keep it available for all code in the relevant compilation unit (as a minimum).
This is how FFmpeg works around it:
if enabled lto; then

test "$cc_type" != "$ld_type" && die "LTO requires same compiler and linker"
check_cflags -flto
check_ldflags -flto $cpuflags
disable inline_asm_direct_symbol_refs

fi

We don't support disabling direct symbol refs for inline asm yet.
Also vf_ass is not the only file ending up with link issues by far (vf_fspp does as well), but it might be the only one enabled when you link against system FFmpeg libraries.

[reimar]

-flto-partition=none works at the cost of longer compile time.