C compile time

[Diego Novillo]

On Thu, 2003-06-19 at 15:21, Dara Hazeghi wrote:

> 1) At this point, the SPEC testers are keeping track
> of runtime. We already have compile regression testers
> for the testsuite which automatically report
> regressions to the list. Should the same be done for
> compile-time, ie a compile time increase of more than
> 1% prompts a message to the list.
> 

SPEC already uses a geometric mean when doing more than 1 run.  The
problem with SPEC is that it uses wall-clock timers.  Variations in the
1-4% range aren't uncommon (even if the compiler didn't change).

If the machine running SPEC happens to be doing something else at the
time, the results will be affected.  Running SPEC in single-user mode is
not practical for an automated tester, so all you can do is -STOP
everything else and disallow logins.

You really need to examine trends in performance and do the detective
work.  In theory, the SPEC testers are collecting everything necessary,
down to the daily diffs.  But the process is time consuming, just the
same.


Diego.