Re: Compilation time (was Re: GCC 3.3)

[Phil Edwards <phil at jaj dot com>]

Somewhat re-ording your suggestions.


On Tue, Apr 29, 2003 at 05:30:22PM -0700, Matt Austern wrote:
>  1. Compile an unchanging piece of source code.

Anything in particular?  I don't much care what, but its build time
multiplied by 5 needs to not be hellishly long.  (The machine is used for
other things; and of course is can't be doing any of those other things
while these tests are running.)

I thought about building SPEC2000, since that seems to be fast ('prox
5 minutes), but I don't have enough spare time in the day to also /run/
SPEC2000 ('prox 1 to 2 hours), which is sort of the whole point of SPEC.


> Terrific idea! How are you doing the timing?  The reason I ask is that 
> I've found you need to be pretty careful to get reproducible results.
[...]
>  4. Compile the test case five times, and throw away the first result.  
> Rationale: at least on some OSs, the result the first time you do the 
> compilation is too noisy.  It's dominated by uninteresting details of 
> what the caches happened to look like before you started the test.  
> ("Five" is slightly arbitrary, but you should at least have three 
> results that you're not throwing away.)
>  5. You've now timed the test case four times.  Report both the mean 
> and the standard deviation.  Rationale: reporting the standard 
> deviation lets people known how reproducible the results really are.

I have some framework scripts that do almost exactly all this.  Adapting them
from running simulations to building some software package should be simple.
Although I'm considering just throwing them out.


Phil

-- 
If ye love wealth greater than liberty, the tranquility of servitude greater
than the animating contest for freedom, go home and leave us in peace.  We seek
not your counsel, nor your arms.  Crouch down and lick the hand that feeds you;
and may posterity forget that ye were our countrymen.            - Samuel Adams