GCC 3.3 compile speed regression - AN ANSWER

[Linus Torvalds]

On Wed, 12 Feb 2003, Jan Hubicka wrote:
> Yes, that may be the case.  My experimence is that number of parts of
> kernel are hand optimized to get assembly out of GCC right.  They are
> tuned for whatever version of GCC author has, so it is more dificult to
> beat it by new GCC.

That's true for some code cases, but usually this ends up being the case 
only for things like "schedule()" etc that are _really_ core to the kernel 
and show up on a lot of things.

The more general and longer code-paths usually don't see that kind of 
optimization.

>	  Thats why I would be interested in seeing specific
> examples as this commonly uncovers stupid mistakes new GCC does.

I didn't do the timings myself, and most of the benchmarking was done on 
"real loads" (that just happen to be very heavy for the kernel), so they 
aren't trivial microbenchmarks that are easy to pinpoint.

The ICC comparisons were done with lmbench, which makes it slightly easier
to compare since you at least get the general _area_ where the improvement
happened. But even then the problem is that the improvements tend to be
much more visible in the "longer" code-paths that haven't been
micro-optimized to the same degree.

> > That said, at least some comparisons with the Intel compiler on an older 
> > kernel imply that the Intel compiler improves things like UDP and TCP 
> > latency by 10-25% on 2.4.18 (in fact, the UDP path was apparently 
> > magically improved by up to 30%, which is damn impressive):
> 
> I remember seeing some comparisons to ICC.  They were to profile driven
> optimization so not 100% fair.  Anyway it would be nice to know what
> makes them faster.

The list archive entry I pointed to does have profile-driven cases too, 
but there are totally fair comparisons there (ie plain "icc -O2"). The 
most interesting case is

	*Local* Communication latencies in microseconds - smaller is better
	-------------------------------------------------------------------
	Host                 OS 2p/0K  Pipe AF     UDP  RPC/   TCP  RPC/ TCP
	                        ctxsw       UNIX         UDP         TCP conn
	--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
	p70        Linux 2.4.18 0.770 4.111 7.30  13.1  32.8  18.9  44.7 69.1
	p70        Linux 2.4.18 0.700 4.405 7.51  13.2  32.8  19.2  45.6 70.1
	p70        Linux 2.4.18 0.730 4.392 7.37  13.5  32.8  19.3  44.7 70.8
	gcc32      Linux 2.4.18 0.780 4.224 7.33  15.7  35.3  21.6  47.5 72.9
	gcc32      Linux 2.4.18 0.870 4.327 7.56  15.7  35.6  21.8  47.1 73.0
	gcc32      Linux 2.4.18 0.900 4.377 7.55  15.6  35.5  21.9  48.0 72.8

note how UDP latency (which is basically the same as performance - it's
the speed path through the networking stack) improved by roughly 15%. The
TCP numbers also improved by 10%+ (the RPC cases also improved, but since
those cases have much more of a user-level footprint, the kernel compiler
obviously won't impact them as much).

10% is quite a bit. 15% is a lot.

The same post also shows comparisons with a P4 and using gcc-3.0, there 
gcc seems to be doing relatively better, and the ICC improvements are less 
noticeable. That may be a P4 issue, ie the P4 trace scheduling might be 
less sensitive to instruction scheduling issues or something like that. 

It may also be that on a P4 the real costs end up being things like 
synchronization primitives etc that are not something the compiler can do 
anything about..

> My common experience is that such a large speedups often comes from very
> simple changes (better memcpy implementation, inlining of one particular
> function or so).  Not sure whether it is the case of kernel code that is
> well hand optimized, but usually it is not that dificult to make GCC to
> do similar trick.

On the whole, I'm pretty happy about gcc, don't get me wrong. People on 
the gcc lists only see me when I complain and moan, so you guys probably 
think I hate your guts and would rather see you all burn in hell. It ain't 
so, and I'm sorry for ragging on you.

Gcc tends to generate "good enough" code as far as I'm concerned, and 
usually if we find some case where we really care, we can tweak it to make 
it easier to optimize.

And yes, this is probably why I at least personally think that performance
improvements in gcc itself are more important than new optimization
schemes. Even if ICC is 10-15% "better" on some load, the fact is, you're
within 10% of a vendor compiler in code quality on the _worst_ example I
could find, so I bet you can find cases where you're at parity or beat
them. So yes, 10% is a "big"  improvement, but at the same time I wouldn't
actually ask you to worry about it.

Quite the reverse, in fact. I think the "10% worse" on one thing is
totally negligible since it doesn't even seem to be across the board.  
But the 4-6x slower compiles _aren't_, which is why the compile speed 
thing comes up.

			Linus