Re: Testsuite reorganization, documentation, cleanup?

[Stan Shebs <shebs at apple dot com>]

Nathanael Nerode wrote:

> Looking at the testsuite, it's rather confusing.  Some directories are
> documented in READMEs; many aren't.  There are multiple organizational
> systems.  There are multiple drivers, many of which seem very similar;
> one suspects redundancy.
Since most of the expect scripts run tests on *.c and suchlike (since
99% of the tests don't need special scripts), much of the organization
is forced by the need to have all source files in a directory treated
alike.

There have been rounds of testsuite cleanup in the past, but in some
cases the rationale wasn't written down, and it's not always as obvious
5-10 years later.  For instance, the theory of dates for a test name
was that that the test didn't necessarily have a "theme" or specific
feature that was being tested, so using the date is a simple way to
generate a canonical name.

>  Beneath that, a subdivision by correctness, code quality, or compiler
>  behavior quality.
Code quality tests are pretty rare, and I don't think we have enough
of them to know how to organize sensibly yet.  I've been thinking
about how to do this kind of test off and on, no doubt there will be
much tcl code involved. :-)

>  Beneath that, subdivision by category where appropriate (abi,
>  templates, etc.)
>
>  Beneath that, one directory for each variety (compile, execute, fail to
>  compile, produce warning 'blah')
I don't like this level of subdivision very much.  It's needed for
torture tests because they don't have any info internal to the file,
but in general I think test instructions instructions for a file should
be in the file, not implicit in some detail of its pathname.

> 2. What do the various existing undocumented directories represent?
>
> For instance, why are there 'gcc.c-torture', 'gcc.dg',
> and 'gcc.misc-tests'?  What are the tests in g++.dg in the subdirectories
> 'bprob','debug','gcov','opt',and 'tls', and why aren't they mentioned in
> g++.dg/README ?  
Documentation standards for testsuite have been pretty lax.  More
general description would be good, and if at some point, the doc has
to say "this sounds stupid, but has to be this way for hysterical
reasons", that's your clue that a reorg is desirable.

Stan