Compilation time (was Re: GCC 3.3)

[Timothy J. Wood]

   Maybe this is obvious, but I'd also add:

6.  Don't use 'configure', 'make' or any other complex glue code in the 
timing tests.  You want to test the compiler time, not the whole build 
process.  This will help in detecting small variations in compiler 
speed by making the percentage of time spent in the compiler larger.  
If you want to use some project for testing that builds using using a  
bunch of glue, maybe do it once via the glue and capture the emitted 
commands, building a trivial shell script out of them.

-tim


On Tuesday, April 29, 2003, at 05:30  PM, Matt Austern wrote:

> On Tuesday, April 29, 2003, at 12:30  PM, Phil Edwards wrote:
>
>> I've added timing to the autocrasher.  Look for "TIME::" followed by 
>> real
>> (wall clock), user, and system seconds in the logs.  It'll start 
>> appearing
>> in the next 31 minutes to 7 days, depending on which build you're 
>> examining.
>
> Terrific idea! How are you doing the timing?  The reason I ask is that 
> I've found you need to be pretty careful to get reproducible results.
>
> I've got some specific suggestions, some of which you've probably 
> thought of already:
>
>  1. Compile an unchanging piece of source code.  Compiler bootstrap is 
> a poor test, for example, because you're never timing the same thing 
> twice.
>  2. Choose a test case that takes at least a minute or so to compile.  
> (Not that there has to be any single translation unit that takes that 
> long, mind you: a test case that consists of a number of smallish 
> translation units is fine.)  Rationale: too short and you find that 
> your results are dominated by noise.
>  3. Just to state the obvious, even though I know you know this 
> already: run on an unloaded system.
>  4. Compile the test case five times, and throw away the first result. 
>  Rationale: at least on some OSs, the result the first time you do the 
> compilation is too noisy.  It's dominated by uninteresting details of 
> what the caches happened to look like before you started the test.  
> ("Five" is slightly arbitrary, but you should at least have three 
> results that you're not throwing away.)
>  5. You've now timed the test case four times.  Report both the mean 
> and the standard deviation.  Rationale: reporting the standard 
> deviation lets people known how reproducible the results really are.
>
> I think that if you follow these suggestions you'll be able to catch 
> sub-1% regressions on a daily granularity, which would be really 
> valuable.
>
> 			--Matt