Testcases in Bugzilla

[Janis Johnson]

On Tue, May 13, 2003 at 01:48:26PM +0200, Volker Reichelt wrote:
> Hi again,
> 
> we still haven't reached a conclusion about the testcases in Bugzilla.

No, but we're making progress.

> Bugzilla offers us the possibility to mark an attachment as testcase
> and a possibility to download them as pointed out in
> 
> http://gcc.gnu.org/ml/gcc/2003-05/msg01212.html
> 
> But how can we actually test the testcases?
> The dejagnu testing harness doesn't allow you to specify a branch
> or something like that where the test is supposed to pass/fail AFAIK.
> And without such a feature the whole testing becomes a nightmare.

The testing harness is actually quite flexible and lets you add whatever
new functionality you need in the .exp files used by the tests.

> There are some resolutions, but they all have their drawbacks:
> 
> a) Create a testcase for each branch. (Way too much work IMHO.)

No, although in some cases it might be necessary to have different
test cases.

> b) Mark all tests as XPASS and supply lists for each branch with the failures.
>    (Too complicated in practice IMHO. And what about other architectures?)

We will want to be able to handle tests that are only expected to fail
on some targets, as well as on some branches.

> c) Add a couple of lines at the top of each testcase that specify the
>    compiler versions where we should XFAIL.

Yes.

>    Run them through a little preprocessor script that removes these lines
>    and adds an XFAIL where appropriate. Hand the result over to dejagnu.

The code in the .exp file could handle this directly.

> d) We can rely on my testing script which is just a small shell script
>    which doesn't use dejagnu and therefore has only limited capabilities,
>    but works for a lot of testcases and can deal with different compiler
>    versions. (See http://gcc.gnu.org/ml/gcc/2002-12/msg01250.html )
>    In that case I'd post a revamped version.

I'd prefer to use the existing testing tools.

> e) We could modify the script in d) to accomplish c).
> 
> f) Extend dejagnu with versioning.

Use DejaGNU functionality to support versioning.

> g) ...?
> 
> 
> I think we should reach a conclusion *before* we start adding the testcases
> to bugzilla so that we don't have to modify the testcases over and over again.

Yes, that's a good point.  I can spend some time on this now.

> Any suggestions?
> 
> Regards,
> Volker
> 
> PS: To avoid name clashes between testcases, we should agree on a naming
> convention (e.g. PR12345A.c).

Or PR12345-1.c, in line with the existing convention for multiple tests
of the same bug/fix.

We'll need some kind of restriction on execute tests to assure people
running these tests that they aren't going to do anything antisocial.

Janis