Page faults and libgcj.so startup time

[Bryce McKinlay <bryce at albatross dot co dot nz>]

Dynamic binaries built against libgcj.so exhibit a pagefault count (as
measured with /usr/bin/time) which is roughly proportionate to the
number of classes in the class library. The startup overhead of such a
binary is roughly 100ms on a 650Mhz P3. This is still substantially
better than the JDK's startup cost but it is still annoying (to
me) that every time new classes are added, the boot time gets higher.

My (or Tom's) first theory for the cause of this was the class
registration global constructors that are dispersed thoughout the
library, which would be causing a lot of pages to be brought in as
they run. Ian Lance Taylor suggested that putting such functions in a
named section which causes the linker to bring them all together in
one physical location in the archive. This turns out to be easy to do
(patch attached), and works nicely. Putting the constructors as well
as the class objects and class utf8 consts (which are all touched by
the class registration) into their own sections gives us (other parts
of output omitted):

$ objdump -h ~/gcc3/lib/libgcj.so

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
 16 .gcj_clreg    00008238  0027b410  0027b410  0027b410  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 19 .gcj.utf8     0005028c  0028cce8  0028cce8  0028cce8  2**1
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 21 .gcj_cldata   0001229c  00318f60  00318f60  00317f60  2**5
                  CONTENTS, ALLOC, LOAD, DATA

So this ~33K of registration functions, 328K of Utf8 constants, and
74K of java.lang.Class objects ought to be all the data that is
required to register libgcj's classes, and most of the other ~5MB of
code in libgcj should be able to go untouched?

Unfortunatly, it does not work. All this has a negligable impact on
the pagefault count.

My next theory came from messing around with LD_DEBUG (try setting
LD_DEBUG=help). It looks like *every* symbol in the library is being
resolved by the linker immediatly at startup time, which would require
most of the pages in the lib to be read by the linker. Most of the
startup cost is probibly in the linker's resolution code itself and
not due to the page loading, except perhaps when the library is not
already cached in memory.

So question 1 is: what ever happened to lazy symbol resolution? I
thought that the linker was supposed to be resolving symbols as code
is executed? Why isn't this working?

One possible solution lies in the -Bsymbolic linker option. This
causes global symbols which exist within libgcj to be resolved at link
time rather than at run time, so the dynamic linker has less work to
do. I experimented with this a bit and got some weird crashes during
initialization of my main class, but it got far enough to dump out a
stack trace and demonstrate that it was starting up around 4X faster
than before.

Question 2 is: What are the disadvantages of "-Bsymbolic"? (the ld
manual indicates that its an ELF-only thing for one). Assuming it can
be made to work, are there any reasons why we wouldn't want to use it?

regards

  [ bryce ]

--- /home/bryce/cvs/gcc/gcc/java/class.c	Thu Mar 22 18:33:08 2001
+++ class.c	Thu Mar 29 18:41:54 2001
@@ -813,6 +813,7 @@
   int name_hash;
   tree ref = IDENTIFIER_UTF8_REF (name);
   tree decl;
+  const char *section = ".gcj.utf8";
   if (ref != NULL_TREE)
     return ref;
 
@@ -860,6 +861,7 @@
   TREE_THIS_VOLATILE (decl) = 0;
   DECL_INITIAL (decl) = cinit;
   TREE_CHAIN (decl) = utf8_decl_list;
+  DECL_SECTION_NAME (decl) = build_string (strlen(section), section);
   layout_decl (decl, 0);
   pushdecl (decl);
   rest_of_decl_compilation (decl, (char*) 0, global_bindings_p (), 0);
@@ -1264,6 +1266,7 @@
   tree interfaces = null_pointer_node;
   int interface_len = 0;
   tree type_decl = TYPE_NAME (type);
+  const char *cldata = ".gcj_cldata";
 
   this_class_addr = build_class_ref (type);
   decl = TREE_OPERAND (this_class_addr, 0);
@@ -1459,6 +1462,7 @@
   FINISH_RECORD_CONSTRUCTOR (cons);
 
   DECL_INITIAL (decl) = cons;
+  DECL_SECTION_NAME (decl) = build_string (strlen(cldata), cldata);
   rest_of_decl_compilation (decl, (char*) 0, 1, 0);
 }
 
@@ -1879,6 +1883,7 @@
   tree init_type = build_function_type (void_type_node, end_params_node);
   tree init_decl;
   tree t;
+  const char *section = ".gcj_clreg";
 
   init_decl = build_decl (FUNCTION_DECL, init_name, init_type);
   SET_DECL_ASSEMBLER_NAME (init_decl, init_name);
@@ -1887,6 +1892,7 @@
   DECL_RESULT (init_decl) = build_decl(RESULT_DECL, NULL_TREE, void_type_node);
   /*  DECL_EXTERNAL (init_decl) = 1;*/
   TREE_PUBLIC (init_decl) = 1;
+  DECL_SECTION_NAME (init_decl) = build_string (strlen(section), section);
   pushlevel (0);
   make_decl_rtl (init_decl, NULL);
   init_function_start (init_decl, input_filename, 0);