libgccjit 5.2 on Mac OSX Yosemite

[David Malcolm]

On Mon, 2015-08-10 at 17:23 -0400, David Malcolm wrote:
> On Mon, 2015-08-10 at 21:36 +0100, Dibyendu Majumdar wrote:
> > Hi
> > 
> > I tried building gcc 5.2 on Mac OSX but could not get the linking to
> > work. Then fortunately found that homebrew has a formula for deploy
> > gcc 5.2 with jit enabled.
> 
> 
> Aha.  Thanks.  This would seem to be:
>  https://github.com/Homebrew/homebrew/blob/master/Library/Formula/gcc.rb
> 
> which I see slightly patches
>   gcc/jit/Make-lang.in
> to avoid an assumption that we're building using the GNU linker.
> 
> 
> 
> > All installed and then built Ravi successfully. But when I try to run
> > Ravi any attempt to invoke the JIT fails with:
> > 
> > Lua 5.3.1  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> > 
> > > function x() return end
> > 
> > > ravi.compile(x)
> > 
> > Undefined symbols for architecture x86_64:
> > 
> >   "_luaD_poscall", referenced from:
> > 
> >       _ravif1 in cczhNq92.o
> > 
> >   "_luaF_close", referenced from:
> > 
> >       _ravif1 in cczhNq92.o
> > 
> > ld: symbol(s) not found for architecture x86_64
> > 
> > collect2: error: ld returned 1 exit status
> > 
> > libgccjit.so: error: error invoking gcc driver: exit_status: 256 err: 0
> > 
> > libgccjit.so: error: whilst attempting to run a driver named:
> > x86_64-apple-darwin14.4.0-gcc-5.2.0
> > 
> > libgccjit.so: error: PATH was:
> > /Users/username/github/ravi/build:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/git/bin:/Library/TeX/texbin
> > 
> > aborting due to JIT error: error invoking gcc driver: exit_status: 256 err: 0
> > 
> > Abort trap: 6
> > 
> > Any clues?
> > 
> > I had set PATH and LD_LIBRARY_PATH.
> 
> Presumably "luaD_poscall" and "luaF_close" are symbols that you're
> exporting from ravi for use by the JIT-generated code, via:
> https://github.com/dibyendumajumdar/ravi/blob/master/src/ravi_gcctypes.c#L802
> 
>   t->luaD_poscallT = gcc_jit_context_new_function(
>       ravi->context, NULL, GCC_JIT_FUNCTION_IMPORTED, t->C_intT,
> "luaD_poscall",
>       3, params, 0);
> 
> I'm not quite sure exactly where the leading underscore came from.
> 
> This appears to be a linker error from the static link that creates the
> jit-generated .so file.  If I'm reading things right, it's complaining
> that the code is using luaD_poscall, but it doesn't have a definition
> for it.
> 
> My guess is that "ld" on OS X has slightly different behavior from the
> GNU "ld" I've been developing against, and that it defaults to a
> stricter linkage model where it wants to see those symbols upfront, when
> we want to defer them until the later dynamic link phase (when we dlopen
> the generated .so file).  This is a guess though.

It looks like we might need to pass an "-undefined" option to the OS X
linker;

http://www.manpages.info/macosx/ld.1.html says:

(begin quote)

-undefined treatment
Specifies how undefined symbols are to be treated. treatment can be:
error, warning, or suppress. Which cause the treatment of undefined
symbols as either, errors, warnings, or suppresses the checking of
undefined symbols. The default is to treat undefined symbols as errors.
When the environment variable MACOSX_DEPLOYMENT_TARGET is set to 10.3
then -undefined dynamic_lookup can also be used to allow any undefined
symbols to be looked up dynamically at runtime. Use of a binary built
with this flag requires a system with a dynamic linker from Mac OS X
10.3 or later. The flag -undefined define_a_way can also be used to
cause the static linker to create a private definition for all undefined
symbols. This flag should only be used if it is known that the undefined
symbols are not referenced as any use of them may cause a crash.


(end of quote)

Note that it defaults to treating undefined symbols as errors, which
looks like what you're seeing.

Hence I *think* we need to pass:

  -undefined dynamic_lookup

to the linker.

Hence I think we need something like this when invoking the driver:

 -Wl,-undefined,dynamic_lookup

so that it passes the option to the linker.


> What does "ld --version" print?
> 
> It may be easiest to try to work from a minimal reproducer for this.
> To that end, what happens if you try building and running
>   gcc/testsuite/jit.dg/test-calling-external-function.c
> from the jit testsuite, using the homebrew build?


Please can you try enabling logging using:
https://gcc.gnu.org/onlinedocs/jit/topics/contexts.html#gcc_jit_context_set_logfile

to capture the driver invocation line (have a look for invoke_driver in
the log).

You'll also need to set the bool option
GCC_JIT_BOOL_OPTION_KEEP_INTERMEDIATES to stop it deleting everything.

With that, you can try to re-run the driver invocation directly from the
command-line, to see if you can reproduce the problem there, and if the
extra option above
  -Wl,-undefined,dynamic_lookup
fixes it.

Thanks
Dave