Re: Performance regression testing?

[Mike Stump <mrs at apple dot com>]

On Nov 29, 2005, at 12:27 PM, Hans-Peter Nilsson wrote:
> > Ah, ok, good.  I'd eject the ,previous to the filename, and reorder
> > slightly, but, certainly that is trivial to do.
>
> Um, (typo?) not a filename, but a line from the file with raw
> results, alternatively the baseline input.  The "previous"
> identifies the baseline.  It's supposed to be sed-able as in
> e.g. "sed -e 's/,previous /,4.2-branch /' < csibe-results >
> /path/to/baseline/4.2-baseline".
>
> What field order looks better to you?  I'm agnostic, except I'd
> like to keep one and the same field delimiter except for the
> result, and it's *slightly* easier to keep it as "," (as in the
> original csibe output).

4.1-sparc-r104567/my-perf-suite.sum
mainline-sparc-r102355/my-perf-suite.sum

name_of_analysis_prog 4.1-sparc-r104567/my-perf-suite.sum mainline- 
sparc-r102355/my-perf-suite.sum

exactly like dejagnu.  Here, there is no previous, there is only the  
first argument to the analysis program, which, by definition names  
the `previous' run.

For a cron job, sure, one can have a sym link from previous to the  
last generated result, then one can do:

run_perf >$RESULT
ln -s $RESULT current
name_of_analysis_prog previous current
mv current previous.

As an non-release engineer, I'm personally more interested in just  
the regressions on a fast timeframe basis against the last run.  A  
release engineer would also want to track against last released  
compiler, notice though, that's just another run of the analysis  
package with a different first argument.

Conflating a single file with a notion of previous is wrong.  Better  
to have them in a separate file, then, one can choose any previous  
that makes sense.