Testcases in Bugzilla

[Volker Reichelt <reichelt at igpm dot rwth-aachen dot de>]

Hi again,

we still haven't reached a conclusion about the testcases in Bugzilla.

Bugzilla offers us the possibility to mark an attachment as testcase
and a possibility to download them as pointed out in

http://gcc.gnu.org/ml/gcc/2003-05/msg01212.html

But how can we actually test the testcases?
The dejagnu testing harness doesn't allow you to specify a branch
or something like that where the test is supposed to pass/fail AFAIK.
And without such a feature the whole testing becomes a nightmare.

There are some resolutions, but they all have their drawbacks:

a) Create a testcase for each branch. (Way too much work IMHO.)

b) Mark all tests as XPASS and supply lists for each branch with the failures.
   (Too complicated in practice IMHO. And what about other architectures?)

c) Add a couple of lines at the top of each testcase that specify the
   compiler versions where we should XFAIL.
   Run them through a little preprocessor script that removes these lines
   and adds an XFAIL where appropriate. Hand the result over to dejagnu.

d) We can rely on my testing script which is just a small shell script
   which doesn't use dejagnu and therefore has only limited capabilities,
   but works for a lot of testcases and can deal with different compiler
   versions. (See http://gcc.gnu.org/ml/gcc/2002-12/msg01250.html )
   In that case I'd post a revamped version.

e) We could modify the script in d) to accomplish c).

f) Extend dejagnu with versioning.

g) ...?


I think we should reach a conclusion *before* we start adding the testcases
to bugzilla so that we don't have to modify the testcases over and over again.

Any suggestions?

Regards,
Volker

PS: To avoid name clashes between testcases, we should agree on a naming
convention (e.g. PR12345A.c).