Dynamic loading - solutions and more problems

[Matt Welsh <mdw at cs dot berkeley dot edu>]

Hi all,

I have some good news and some bad news.

First, the good news:

I managed to get dynamic loading from precompiled classes (as .so files) 
to work with gij again. This is with the first "alternate" version listed
on download.html from the GCJ webpage (gcc-2.95.2 with Bryce's patches,
along with libgcj from CVS from March 02, 2000 00:00:00, on Linux x86).

Here's how it works. (Tom, you might want to update the FAQ if this isn't
all in there):

1) Compile your .class or .java files to .so with:
      gcj -shared -o Foo.so Foo.java
   or
      gcj -shared -o Foo.so Foo.class

   DO NOT USE "-c" with "-shared" !

   You can also compile entire packages to a .so file. The class
   "foo.bar.someclass" will be sought for in the files:
      foo-bar-someclass.so
      foo-bar.so
      foo.so
   in that order.

2) Add the directories where any .so files are located to either
   LD_LIBRARY_PATH or LTDL_LIBRARY_PATH. (The latter is used internally
   by libgcj and is convenient if you don't want to pollute LD_LIBRARY_PATH
   for some reason).

3) You can then use:
     gij Foo
   
It doesn't look like you can compile your application to a binary and have it
dynamically load classes from .so files; you have to specify the .so files on
the 'gcj' command line or the linker will complain. When you do this I'm
pretty sure that those .so files will be loaded when the application is
started, rather than on the fly. I'm wondering if there's any way around this.

One warning: if you have a file called "Foo" (rather than "Foo.so") on your 
LD_LIBRARY_PATH/LTDL_LIBRARY_PATH, it will override the "Foo.so". This
is probably a bug in libltdl. The problem is that you might have a binary
(or some other file) called "Foo" which is unrelated to "Foo.so", however,
"Foo" is loaded first and (if it's a binary, say) causes gij to segfault
when it tries to treat it as a shared object file.

Ideally, to avoid this namespace collision we would have a new extension
for compiled class files (like .gcjso) and gij would only search for those
files. 

All right, now for the bad news:

Okay - so all of the above works fine. However, I am running into a bug which
seems to have to do with one precompiled .so trying to dynamically load
another.

The following program dynamically loads the class "ToLoad", creates an
instance of it, and calls the method "doit()" on that instance.

public class MainProgram {

  public static void main(String args[]) {
    try {
      Class ToLoadCls = Class.forName("ToLoad");
      ToLoad toload;
      toload = (ToLoad)ToLoadCls.newInstance();
      toload.doit();
    } catch (Exception e) {
      e.printStackTrace();
    } 
  }
}

Now, this works fine if I compile MainProgram to a .class file and let
gij interpret the bytecodes. It also works fine on other JVMs (of course).

However, if I compile MainProgram.java to MainProgram.so as above, 
I get the following error (the lines with 'MDW' are printf's I added
to libgcj to aid debugging):

mm33:~/src/ninja/test/mdw/autoload/load2% gij MainProgram
MDW: Class.loadClass() trying to find: MainProgram
MDW: Class.loadClass() calling findSystemClass for: MainProgram
MDW: Runtime::loadLibraryInternal() loading MainProgram
MDW: Runtime::loadLibraryInternal() got handle 0x0
MDW: Class.loadClass() calling findClass for: MainProgram
java.lang.ClassNotFoundException: MainProgram
   at 0x400e74cb: java::lang::Throwable::Throwable(java::lang::String *) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x400daff4: java::lang::Exception::Exception(java::lang::String *) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x400da364: java::lang::ClassNotFoundException::ClassNotFoundException(java::lang::String *) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x400ff2e6: java::net::URLClassLoader::findClass(java::lang::String *) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x400d9781: java::lang::ClassLoader::loadClass(java::lang::String *, bool) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x40135896: _Jv_FindClass(_Jv_Utf8Const *, java::lang::ClassLoader *) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x40132a52: java::lang::Class::forName(java::lang::String *) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x4013667a: java::lang::FirstThread::run(void) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x400e4742: java::lang::Thread::run_(void) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x4013c1da: java::lang::Thread::run__(java::lang::Object *) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x4014cdc2: _Jv_ThreadSetPriority(_Jv_Thread_t *, int) (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcj.so.1)
   at 0x401f3d16: GC_start_routine (/home/cs/mdw/disks/mm33/egcs-jaguar/install/gcc-2.95.2-new-jaguar/lib/libgcjgc.so.1)
   at 0x4020ace9: pthread_detach (/lib/libpthread.so.0)
   at 0x402ee45a: __clone (/lib/libc.so.6)


Now, if I comment out the lines:

      //toload = (ToLoad)ToLoadCls.newInstance();
      //toload.doit();

gij manages to load the .so just fine (although, of course, it can't
create an instance of the class it's loading). 

The broken MainProgram.so has the following extra undefined symbols:

_CL_6ToLoad
_Jv_CheckCast
newInstance__Q34java4lang5Class

My guess is that it's the _CL_6ToLoad which is causing the problem.

It looks like lt_dlopenext() returns NULL if MainProgram.so has 
undefined symbols which it can't resolve. Well, since I'm referencing
a class (ToLoad) which I load dynamically, it's not surprising that
this would be the case. However, it's a bug, since this case *should* work.

It's kind of a chicken-and-egg problem, since we need to load MainProgram.so
in order to execute it to load ToLoad.so. But, you need to load ToLoad.so
to let MainProgram.so load. Yikes!

Is there anything we can do about this?

Thanks much!
Matt Welsh, UC Berkeley