Re: Automated regression testing

[Geoff Keating <geoffk at geoffk dot org>]

> Date: Sun, 18 Mar 2001 23:38:01 -0500 (EST)
> From: John Wehle <john@feith.com>
> Cc: geoffk@cygnus.com

>   2) Is there interest in automatic nag messages?  I'm assuming that
>      I should just modify the scripts that Geoff uses for this (in
>      which case I need a pointer on where to get them).

I'm unenthusiastic about setting up more testers if that means that
instead of getting a single mail message it means people get two or
more.  If this happens, I suspect people will just start filtering
them out; if a single commit results in seven messages, _I_ will start
filtering the mail.

I'm currently working on a way to integrate multiple testers, but then
the problem is that you have to have uniformly fast machines or
the delay between making the patch and getting the message is too
long.  My aim is that if someone commits a patch and then leaves for
lunch, they should get a complaint about it in time to fix it before
going home.

Finally, note that the whole point of the regression tester is that it
does its very best to only send mail to the person who actually caused
the regression.  If you only test once a day, you're testing tens of
patches at a time, and so you really need a human to narrow it down.

> Issues:
> 
>   1) Currently only the powerpc target builds with any regularity though
>      I do have a patch in the queue for arm-elf and I'm looking at the
>      sh-elf issues.

Of course, before any useful regression testing can be done you need
to have a target that builds, otherwise there can't possibly be any
regressions ;-).

-- 
- Geoffrey Keating <geoffk@geoffk.org>