getting mplayer to work with -fstack-protector

[basic@…]

Not sure which component I should use for this, so I'll try core. What I'm
trying to do here is to get mplayer working when compiled with
-fstack-protector. Currently mplayer (on gcc 3.3.4 gentoo linux x86) compiles
with -fstack-protector as long as one has -O or -O2 in CFLAGS. There are a few
issues I've encountered:

1) loading win32 dlls do not work (patch for this coming up).
2) (with my patch to fix 1) quicktime win32 audio cause a segfault (I've a
workaround, though I don't like it)

I've yet to test other formats. Does mplayer have a repository of test videos
for regression testing?

[basic@…]

ldt_keeper.patch

[basic@…]

add "eax" to the list of clobbered registers. Without this I get a stack
smashing error when loading any win32 dll decoders.

[basic@…]

(In reply to comment #0)

2) (with my patch to fix 1) quicktime win32 audio cause a segfault (I've a
workaround, though I don't like it)

More on the quicktime audio decoder issue, it seems to crash at loader/win32.c
line 796. I've tried checking all the values there, the only var that causes a
crash is "ml". My workaround is to just disable -fstack-protector on win32.c .

[basic@…]

no-stack-protector patch

[basic@…]

this patch adds my workaround for problem 2. Not sure if it is the right thing
to do though.

[reimar]

The problem with windows dlls is not fixable due to the way that windows handles
the stack, see the comment at loader/pe_image.c, line 888.

[basic@…]

dmo wmv patch

[basic@…]

add "edx" to list of clobbered registers to make wmv files not trigger crash
without attachment 79

[basic@…]

this makes me wonder how does wine load dlls (wine does work with
-fstack-protector, I think, except for the preloader, but that's due to it not
linking into libc at all)...

[reimar]

I have to admit I am not 100% percent sure that comment is correct, but I think
it is. And another point is that I actually don't really know what
-fstack-protector does...

[basic@…]

(In reply to comment #7)

I have to admit I am not 100% percent sure that comment is correct, but I think
it is.

Well I've been playing with the -O flags and found that -O2 I get the quicktime
problem but -O3 I do not. And since the default -O flag for mplayer is -O4 , I'd
say we can recommend people using -O3 or -O4 if they find problems ;). Anyways
I'll test and see if -O1 would work as well.

And another point is that I actually don't really know what
-fstack-protector does...

-fstack-protector causes the program to crash when a stack overflow happens,
this is helpful as it makes it easier to catch such bugs. Besides realplayer,
wmp and quicktime are there any other formats that uses win32 dlls in mplayer?

[basic@…]

with -O3 this is not neeeded. I'm testing -O1, right now, will also try to see
which flag (turned on/off by -O3) is the one that make it work.

[basic@…]

ok tested -O1, didn't work. Tried -O1 with -O3 for win32.c, that worked! And
tried -frename-registers on win32.c, that works! So it seems that
-fstack-protector + -frename-registers is needed for win32.c on x86 gcc 3.3.4
otherwise it seems to trigger some kind of bug (gcc bug?). I'll be testing mp3
next. Would love some suggestions on what to test.

[basic@…]

I think I'm about done. The 2 patches should be all there is to it (barring any
issues that I've not tested for). What do I need to do to get this checked in?

[reimar]

(In reply to comment #11)

I think I'm about done. The 2 patches should be all there is to it (barring any
issues that I've not tested for). What do I need to do to get this checked in?

I'll send the patches to the -dev-eng mailing list (a bit modified, I put the
clobber list in a separate line to minimize the diff). If I get no objections I
will apply them. So the answer is "probably nothing".

[reimar]

In CVS