[Ralf.Wildenhues@gmx.de: Re: [rth@redhat.com: libjava build times]]

[Joe Buck <Joe dot Buck at synopsys dot COM>]

The libtool folks seem to be making some progress in attacking our
problems.  I had forwarded them Richard's data on the libtool
performance problems.

When they have something ready to try, I hope someone on our end
who knows about this stuff (Alexandre?  Java folks?) can try it out.

I'd send a URL to the thread but the lists.gnu.org site seems to
be down; if it ever comes back there is

http://lists.gnu.org/archive/html/bug-libtool/

----- Forwarded message from Ralf Wildenhues <Ralf.Wildenhues@gmx.de> -----

Date: Thu, 5 May 2005 12:11:29 +0200
From: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
To: Joe Buck <Joe.Buck@synopsys.COM>, bug-libtool@gnu.org
Cc: libtool-patches@gnu.org
Subject: Re: [rth@redhat.com: libjava build times]
Mail-Followup-To: Joe Buck <Joe.Buck@synopsys.COM>,
	bug-libtool@gnu.org, libtool-patches@gnu.org
In-Reply-To: <20050504194021.GA14132@iam.uni-bonn.de>
Organization: Department of Numerical Simulation, University of Bonn
X-pstn-levels:     (S:99.90000/99.90000 R:95.9108 P:95.9108 M:97.0232 C:98.7678 )
X-pstn-settings: 2 (0.5000:0.5000) s gt3 gt2 gt1 r p m c 
X-pstn-addresses: from <Ralf.Wildenhues@gmx.de> [27/2] 
X-Bogosity: No

Hi Joe, others,

* Ralf Wildenhues wrote on Wed, May 04, 2005 at 09:40:21PM CEST:
> * Ralf Wildenhues wrote on Wed, May 04, 2005 at 06:30:57PM CEST:
> > * Joe Buck wrote on Mon, May 02, 2005 at 08:01:49PM CEST:
> > >
> > > We really need something done about this problem, as it interferes
> > > with our ability to efficiently develop GCC.

Thank you for providing this example!  It shows the bottlenecks neatly.

I will tackle 'link mode' first, as I have worked on that previously and
know roughly what to do.  libtool-cache integration comes afterwards
(and probably in form of a small compiled program).

> > How many objects does libjava contain?  Rather 100 or 1000?  Do you
> > need relinking because of command line length?
> 
> I am compiling libjava ATM to see what happens.

OK, step 1: Executing the GCC provided libtool script for
libgcj0_convenience.la (warm cache) takes

  294.35user 45.47system 6:13.42elapsed 91%CPU (0avgtext+0avgdata 0maxresident)k
  0inputs+0outputs (0major+6043907minor)pagefaults 0swaps

on my laptop (Pentium M 1.8), with --dry-run, so without doing any real work.

One significant bugger is the quadratic renaming loop, and all the
stuff that I already fixed in Libtool HEAD.  The attached rough patch
kills the loop and brings Libtool HEAD down to

  170.22user 10.41system 3:17.38elapsed 91%CPU (0avgtext+0avgdata 0maxresident)k
  0inputs+0outputs (0major+1483544minor)pagefaults 0swaps

building the piecewise archiving cmdline better gets me

  133.80user 4.28system 2:36.78elapsed 88%CPU (0avgtext+0avgdata 0maxresident)k
  0inputs+0outputs (4major+747642minor)pagefaults 0swaps

I'm pretty sure I can get it quite a bit faster even.  The patches need
cleanup so that they use allowed file names and work properly in corner
cases as well, but those don't scale with the number of objects, so they
matter less.

If you are interested in further work: followup patches and cleaned up
versions of these will be posted to the libtool-patches list only.

Regards,
Ralf

	* config/ltmain.m4sh (func_mode_link): Kill quadratic loop
	while renaming objects.
	Reported by Joe Buck <Joe.Buck@synopsys.COM>.

Index: config/ltmain.m4sh
===================================================================
RCS file: /cvsroot/libtool/libtool/config/ltmain.m4sh,v
retrieving revision 1.66
diff -u -r1.66 ltmain.m4sh
--- config/ltmain.m4sh	27 Apr 2005 11:30:38 -0000	1.66
+++ config/ltmain.m4sh	5 May 2005 09:42:10 -0000
@@ -6298,43 +6298,31 @@
 	# static archive out of a convenience library, or when linking
 	# the entirety of a libtool archive into another (currently
 	# not supported by libtool).
-	if (for obj in $oldobjs
-	    do
-	      func_basename "$obj"
-	      $ECHO "$func_basename_result"
-	    done | sort | sort -uc >/dev/null 2>&1); then
-	  :
-	else
+	gentop="$output_objdir/${outputname}x"
+	generated="$generated $gentop"
+	func_mkdir_p "$gentop"
+	objs_bnames="_objs_basenames"
+	objs_nonuniq="_objs_nonuniq"
+	$ECHO "$oldobjs" | $SP2NL | $SED -n -e "$basename" -e '/./p' >"$objs_bnames"
+	sort "$objs_bnames" | uniq -d > "$objs_nonuniq"
+	if test -s "$objs_nonuniq"
+	then
 	  $ECHO "copying selected object files to avoid basename conflicts..."
-	  gentop="$output_objdir/${outputname}x"
-	  generated="$generated $gentop"
-	  func_mkdir_p "$gentop"
 	  save_oldobjs=$oldobjs
-	  oldobjs=
-	  counter=1
-	  for obj in $save_oldobjs
+	  : >_replace_script
+	  # make sure we use unique names
+	  counter=`$SED -n '/^lt-[[0-9]][[0-9]]*/ { s/^lt-//; s/[[^0-9]].*//; p; }' _objs_basenames \
+		       | sort -n | $SED -n '$p'`
+	  $ECHO "$oldobjs" | $SP2NL | $SED -n -e '/./p' >_objs
+	  paste "$objs_bnames" _objs | sort >_objs_plus_bnames
+	  join "$objs_nonuniq" _objs_plus_bnames | while read objbase obj
 	  do
-	    func_basename "$obj"
-	    objbase="$func_basename_result"
-	    case " $oldobjs " in
-	    " ") oldobjs=$obj ;;
-	    *[[\ /]]"$objbase "*)
-	      while :; do
-		# Make sure we don't pick an alternate name that also
-		# overlaps.
-		newobj=lt$counter-$objbase
-		counter=`expr $counter + 1`
-		case " $oldobjs " in
-		*[[\ /]]"$newobj "*) ;;
-		*) if test ! -f "$gentop/$newobj"; then break; fi ;;
-		esac
-	      done
-	      func_show_eval "ln $obj $gentop/$newobj || cp $obj $gentop/$newobj"
-	      oldobjs="$oldobjs $gentop/$newobj"
-	      ;;
-	    *) oldobjs="$oldobjs $obj" ;;
-	    esac
+	    counter=`expr $counter + 1`
+	    newobj=lt$counter-$objbase
+	    func_show_eval "ln $obj $gentop/$newobj || cp $obj $gentop/$newobj"
+	    $ECHO "s#^$obj\$#$gentop/$newobj#" >>_replace_script
 	  done
+	  oldobjs=`$SED -f _replace_script _objs | $NL2SP`
 	fi
 	eval cmds=\"$old_archive_cmds\"
 


----- End forwarded message -----