Re: GCC testsuite philosophy?

["Joseph S. Myers" <jsm28 at cam dot ac dot uk>]

On Tue, 20 Mar 2001, Stan Shebs wrote:

> In particular, I want to add compile and noncompile tests for ObjC,
> a la gcc.c-torture.  For both of these, I want to be able to match
> on warnings and errors.

This sounds more like gcc.dg than gcc.c-torture.

> For noncompile tests, I was able to make an objc-dg.exp and get
> reasonable results, except that it runs each test at different
> optimization levels, which seems kind of pointless, since the
> compiler errors out during parsing.  So is that really how tests for
> compilation errors are supposed to work?

gcc.dg/noncompile does this since GCC tries to continue after errors, and
errors should not lead to compiler crashes later (but often have).  
"confused by earlier errors" simply puts a nicer face for the user on
these lower-priority bugs, but at least for C (maybe not for C++ yet) it's
still a bug that should be fixed if an ICE occurs after an error.

> For compile tests, I wanted to get a warning and match on it to
> make sure it's the right message, but it seems that the c-torture
> expect script always passes -w, which kind of defeats the purpose. :-)
> This leaves me wondering if I'm missing something obvious, or do
> I have to use objc-dg.exp here also?

Probably you should use objc-dg.exp.  I reckon that the c-torture tests
are only really for cases where (a) the test should compile/execute, and
diagnostics aren't of interest and (b) no special options are required
beyond the standard torture ones.  This covers both regressions and
explicit optimisation tests (if this doesn't get optimised away, we get a
link failure / abort).

> Also, I need to come up with a way to test some of the objc/
> tests one way when using the NeXT-derived runtime in OS X, and
> a different way when using the libobjc&GNUstep runtime.  Presumably
> I can arrange this similarly to how multiple optimization options
> are handled; site.exp hackery seems too heavy-duty.
> 
> Finally, are the arguments to dg-warning and dg-error documented
> anywhere?

These things need cleaning up by someone deeply familiar with dejagnu.
I'd like a better way of doing tests with multiple options than what
gcc.dg/format/format.exp currently does.  I'd also like dg-hard-error /
dg-warning-only added, to require that a diagnostic be of a particular
kind - asserts that something is an error are a pain when both a dg-bogus
(to check it's not a warning) and a dg-error are needed (see the C
standards tests).

-- 
Joseph S. Myers
jsm28@cam.ac.uk