Re: Regression(?) on the 3.2 branch: 23_containers/vector_capacity.ccexecution test

[Hans-Peter Nilsson <hp at bitrange dot com>]

(You seem to have removed gcc@.  I think this is on-topic for
gcc@, so I added it back.)

On Tue, 10 Dec 2002, Benjamin Kosnik wrote:
> >From '2002-11-24 05:00 UTC' to '2002-12-09 04:00 UTC' appeared:
> >FAIL: 23_containers/vector_capacity.cc execution test
> >on the 3.2 branch for all newlib targets (mmix-knuth-mmixware,
>
> So, roughly this is what is failing:

I don't know.  Did you check or did you make an assumption based
on the added test?

> [suggestion for reduced test]
> You might see if that, with the cout bits, is more illuminating.

I might try that.  Hey, why not add it as a separate test?
Then a person wouldn't have to go in "manually" and check now...
;-)

For the wider issue about the organization of the libstdc++-v3
test-suite, in summary I don't understand any of your objections
below; the objections don't seem to target what I suggest.

> >It doesn't help bug-tracking, that the way the libstdc++-v3
> >project adds to the testsuite by *modifying* existing named
> >test-files.
>
> [snip]
>
> It's my opinion that a testsuite designed for bug tracking is less
> useful than a testsuite designed to test conformance. Reasonable people
> may differ, and have. The libstdc++ testsuite is organized around
> conformance: some of the other gcc projects, including gcc/g++, have
> organized around bug tracking and regression testing.

The libstdc++-v3 test-suite can certainly serve both purposes,
with the add-new-test-as-separate-file suggestion.

Besides, what's better conformance-test-wise with knowing that
*some* test in vector_capacity failed, rather than (say) *the*
test vector_capacity/vector_resize/03.cc failed?

> >Can the libstdc++-v3 project please consider adding *files* when
> >adding to the test-suite?  For example, the added test could
> >have been testsuite/23_containers/vector_capacity/03.cc or
> >testsuite/23_containers/vector_capacity-03.cc.
>
> Well, the organization can certainly be improved.
>
> I'm not a fan of adding files like "03.cc" to a testsuite, as I don't
> believe that it's helpful in the long run,

Helpful for what?  Perhaps you actually refer to bug-tracking?  ;-)

> even if it does contain only
> one bug.

I suggested adding tests named (e.g.) vector_capacity/03.cc, not
just 03.cc at some top-level.  Besides, it was just one of the
two naming scheme suggestions.  (Pick another one.)

I don't see how, from any helpfullability (sp?) perspective, my
as-separate-file suggestion differs from hiding a test named
test03() in another file.  AFAICS the latter is certainly worse
from all testing aspects, except it's *slightly* simpler to add
code than to add a file.

> If you've ever worked with large, commercial testsuites that
> implement this kind of organization, you will quickly see how unhelpful
> this becomes as the tests multiply and you attempt to figure out what,
> at a glance, is wrong. At least with the current organization, you,
> without much effort, could tell the failure was within the general range
> of vector capacity, as indicated in chapter 23 of the standard.
>
> In general, I'd like to keep this kind of structure. As the tests become
> more and more involved, I'm suggesting that tests like
> 23_containers/vector_capacity get broken down

No, please don't reorg the test-suite just for naming.  That'd
be a real catastrophy bug-tracking-wise.

> into things like
>
> 23_container/vector_size
> 23_container/vector_capacity
> 23_container/vector_reserve
> 23_container/vector_resize

You can start adding tests using this new scheme or another of
your choice, and that should just work.

> Your thought to make sub directories was interesting, but seems
> difficult given the constraints of CVS, and its known issues dealing
> with directories and renames. Thanks for suggesting it, however.

No need or desire to rename anything; that'd be just wrong for
bug-tracking perspective.  Hence no need to punt the issue,
referring to known CVS issues.

[*shrug*] I've done what I could do, I guess.  [Throws up
hands.]  I'll see about the specific bug, though.

brgds, H-P