Re: "ways of accomplishing ... the basic goal"

[Tom Lord <lord at emf dot net>]

       > The problem after merges is mainly _unknown_ regressions -
       > regressions not shown up in the testsuite.  Many of these
       > could perhaps be found by bulk builds of real world software
       > - such as BSD packages/ports collections and GNU/Linux
       > distributions - but doing such builds requires vastly more
       > resources than a bootstrap/test of GCC.

       Note that this way the new C++ parser never could have gone in:
       I am quite sure it has broken a most significant amount of C++
       packages (be it in the FreeBSD ports collection or Debian
       packages)...

       > And then you want some human filtering for the cases where
       > the problem was in the code that broke, not in GCC.)

       ...and checking all these breakages (most of which would not
       have been the fault of the new C++ parser) would have been
       hardly feasible.


Hold on there.  Maybe (probably) I'm not reading you right.

Let's suppose we wanted to make a massive test suite for a new C++
parser using, say, the Debian source base.

So, first we identify those packages that use C++.

Then we pick a stable version of GCC to use as a baseline.

Then try building everything.  Just discard any packages that don't
build.

There's the test suite.  Anything that doesn't build with the new
parser is, until proven otherwise, a regression.

Now, there is _something_ lost here, at least initially.  If you don't
do that test through automation, but by getting users to build their
packages on their own, then you have a small army of users who are
turning the regressions into hopefully high quality bug reports --
i.e., reducing a failed build to a tiny example that illustrates the
problem.

But is that really necessarilly lost?  After all, who is doing that?
Even now, it's people who have chosen to use non-released versions of
GCC in order to help with testing.  So if that test _is_ automated,
those people can still play -- and they're better off because you can
point to the failed builds and ask people to focus their test
compilations on those, specifically.

And then there's all the volunteer testers compiling non-public source
with pre-release source.  They can still play too -- you're just
pointing them at a different branch or snapshot saying "Hey, want
faster C++ compilations?  Test this on your sources and tell us what
goes wrong...."

And yes -- the new parser wouldn't go in the trunk so early.  It
wouldn't have to.  A lot more testing would take place off-trunk.

One refinement that this suggests to the proposal:  perhaps acceptance
tests should vary according to the content of the proposed change.

-t