Opened 14 years ago

Closed 14 years ago

Last modified 14 years ago

#169 closed enhancement (fixed)

getting mplayer to work with -fstack-protector

Reported by: basic@… Owned by: alex@…
Priority: normal Component: core
Version: HEAD Severity: minor
Keywords: Cc:
Blocked By: Blocking:
Reproduced by developer: Analyzed by developer:

Description

Not sure which component I should use for this, so I'll try core. What I'm
trying to do here is to get mplayer working when compiled with
-fstack-protector. Currently mplayer (on gcc 3.3.4 gentoo linux x86) compiles
with -fstack-protector as long as one has -O or -O2 in CFLAGS. There are a few
issues I've encountered:

1) loading win32 dlls do not work (patch for this coming up).
2) (with my patch to fix 1) quicktime win32 audio cause a segfault (I've a
workaround, though I don't like it)

I've yet to test other formats. Does mplayer have a repository of test videos
for regression testing?

Attachments (3)

ldt_keeper.patch (533 bytes) - added by basic@… 14 years ago.
ldt_keeper.patch
no-stack-protector.patch (1.3 KB) - added by basic@… 14 years ago.
no-stack-protector patch
dmo-wmv.patch (466 bytes) - added by basic@… 14 years ago.
dmo wmv patch

Download all attachments as: .zip

Change History (16)

Changed 14 years ago by basic@…

ldt_keeper.patch

comment:1 Changed 14 years ago by basic@…

add "eax" to the list of clobbered registers. Without this I get a stack
smashing error when loading any win32 dll decoders.

comment:2 Changed 14 years ago by basic@…

(In reply to comment #0)

2) (with my patch to fix 1) quicktime win32 audio cause a segfault (I've a
workaround, though I don't like it)

More on the quicktime audio decoder issue, it seems to crash at loader/win32.c
line 796. I've tried checking all the values there, the only var that causes a
crash is "ml". My workaround is to just disable -fstack-protector on win32.c .

Changed 14 years ago by basic@…

no-stack-protector patch

comment:3 Changed 14 years ago by basic@…

this patch adds my workaround for problem 2. Not sure if it is the right thing
to do though.

comment:4 Changed 14 years ago by reimar

The problem with windows dlls is not fixable due to the way that windows handles
the stack, see the comment at loader/pe_image.c, line 888.

Changed 14 years ago by basic@…

dmo wmv patch

comment:5 Changed 14 years ago by basic@…

add "edx" to list of clobbered registers to make wmv files not trigger crash
without attachment 79

comment:6 Changed 14 years ago by basic@…

this makes me wonder how does wine load dlls (wine does work with
-fstack-protector, I think, except for the preloader, but that's due to it not
linking into libc at all)...

comment:7 Changed 14 years ago by reimar

I have to admit I am not 100% percent sure that comment is correct, but I think
it is. And another point is that I actually don't really know what
-fstack-protector does...

comment:8 Changed 14 years ago by basic@…

(In reply to comment #7)

I have to admit I am not 100% percent sure that comment is correct, but I think
it is.

Well I've been playing with the -O flags and found that -O2 I get the quicktime
problem but -O3 I do not. And since the default -O flag for mplayer is -O4 , I'd
say we can recommend people using -O3 or -O4 if they find problems ;). Anyways
I'll test and see if -O1 would work as well.

And another point is that I actually don't really know what
-fstack-protector does...

-fstack-protector causes the program to crash when a stack overflow happens,
this is helpful as it makes it easier to catch such bugs. Besides realplayer,
wmp and quicktime are there any other formats that uses win32 dlls in mplayer?

comment:9 Changed 14 years ago by basic@…

  • attachments.isobsolete changed from 0 to 1

with -O3 this is not neeeded. I'm testing -O1, right now, will also try to see
which flag (turned on/off by -O3) is the one that make it work.

comment:10 Changed 14 years ago by basic@…

ok tested -O1, didn't work. Tried -O1 with -O3 for win32.c, that worked! And
tried -frename-registers on win32.c, that works! So it seems that
-fstack-protector + -frename-registers is needed for win32.c on x86 gcc 3.3.4
otherwise it seems to trigger some kind of bug (gcc bug?). I'll be testing mp3
next. Would love some suggestions on what to test.

comment:11 Changed 14 years ago by basic@…

I think I'm about done. The 2 patches should be all there is to it (barring any
issues that I've not tested for). What do I need to do to get this checked in?

comment:12 Changed 14 years ago by reimar

(In reply to comment #11)

I think I'm about done. The 2 patches should be all there is to it (barring any
issues that I've not tested for). What do I need to do to get this checked in?

I'll send the patches to the -dev-eng mailing list (a bit modified, I put the
clobber list in a separate line to minimize the diff). If I get no objections I
will apply them. So the answer is "probably nothing".

comment:13 Changed 14 years ago by reimar

  • Resolution set to fixed
  • Status changed from new to closed

In CVS

Note: See TracTickets for help on using tickets.