Re: branch bootstrap breakage in new DECL_RTL

[kenner at vlsi1 dot ultra dot nyu dot edu (Richard Kenner)]

    True, but to the extent that a machine can do things it frees up
    humans to do more productive work.  A machine also doesn't get bored
    or impatient doing rote meticulous tasks & testing that developers
    often neglect.

Just to be clear: I'm not arguing against adding more testing, which is
a good thing, but on becoming too *reliant* on it and using it to set very
high standards.

My concern is what standards we set for the top-of-trunk.  We have to balance
two things: we want to encourage development (which may tend to break things)
but also want to keep the tree in as stable a state as possible.

Clearly, development can't proceed unless the tree is working the majority of
the time.  When we fork off a release, we set a higher standard, but is it
reasonable to set that similar standard for the mainline (if we do, why fork)?

Over the last two weeks we've been talking about a number of patches that
broke either the release branch or mainline.  But each has been due to a
*different* reason.  I had an ia32 patch that was fine, but got misapplied to
the branch.  Testing the patch as it was meant to be applied (the usual
procedure) didn't find this, though testing on commit would have.  RTH's
Sparc64 patch might or might not have been caught with testing: even with
that patch in, sometimes the bootstrap still worked and we don't know exactly
what combination caused the problem (which has now finally been fixed).  Note
also that this only occurred with checking disabled and it would normally be
more useful to do the testing with checking enabled, so this probably
wouldn't have been caught anyway.

Then there was the patch I did to enable a validity check on BLOCK notes and
nodes.  That uncovered serious problems in BLOCK handling in the Java front
end (of course, the *purpose* of enabling checks is precisely to smoke out
such problems).  Automated testing would have only shown this problem if Java
had been enabled (it was disabled by default at the time).  Mark's DECL_RTL
change missed a few cases.  Those caused warnings when compiled with GCC and
errors when compiled with vendor compilers.  Either starting the bootstrap
with a vendor compiler or considering warnings as errors would have caught
that, but we made the opposite choices when testing it.  The configure
breakage occurred only when GNU grep wasn't used, but I recall some problem
that occured only when GAS *was* used.

Note that *each* of these breakages were due to a different reason and each
would dictate a *different* additional test.  To be complete, testing would
have to be done with and without checking and with and without GNU tools,
among other choices, and this is across a lot of targets.  That's clearly
infeasable.

As I said, that's not to mean we shouldn't try to do better, but if we don't
accept that things *will* break from time to time, all we're going to do is
get ourselves frustrated.

There's also another issue: one of the major motivators of having the open
tree and development process was to encourage people to do with on the
compiler.  Obviously, we have to exert quality control on their work, but
there's a balance there too: if we make it *too* hard for them to contribute,
they won't.

RTH's patch is a good example of this.  He knows the compiler well, so this
doesn't apply to him, but that sort of patch is one that could well have been
done by somebody very new to GCC who had just learned the parameter location
code in sparc.c and had found that bug.  This person might have been very
proud of themselves and want to contribute that fix to the GCC project.

If our response to that fix is "your fix looks completely correct, but
unfortunately, it just so happens that it changes the addressees of things
and produces a subtle mismatch in the bootstrap.  So you'll have to debug
that before we'll accept your work", the person is very likely to get
completely discouraged and go away.  Sure, they can ask for help, but are
they likely to get it in practice?  I don't think so.

So the outcome here would have been that that patch (which fixes a real bug,
reported by a real user, by the way) would not have gone in, the latent bug
would have stayed latent for longer, and we'd have lost a potential
contributor.  If this happens enough times, we run the real risk of having a
reputation of being hard to work with and we get the contributors we need.

So that's also a balance.

What standard *should* we aim for?  I feel strongly that "It's never broken
anywhere anytime" is simply unachievable and if we aim for that standard,
it'll be a very high burden all around and be unattainable anyway.

If we define a "major breakage" as meaning that the compiler won't bootstrap
on a major target or build on a secondary target or in a common (but not
default) configuration, and a "minor breakage" as meaning a test failure on a
common configuration or a bootstrap or build problem on an obscure
configuration, I think a reasonable standard to aim for is one major breakage
every 10 days or so and a minor breakage every 3-5 days.  Right after a
release, I'd expect more breakage than when we're ready to fork for a release.