Re: Check-In Requirements

[Loren James Rittle <rittle at latour dot rsch dot comm dot mot dot com>]

In article <20000601225013J.mitchell@codesourcery.com>,
Mark Mitchell <mark@codesourcery.com> writes:

>   I'm a little worried, though.  If someone makes an erroneous change,
> runs the tests (and some of them fail), but then uses the
> test-reporting script, that will be confusing.  You're suggesting, of
> course, that people only use the script when everything is ready to
> check-in, but I'm afraid that people might accidentally send in bogus
> reports.

OK, I think I understand the concern you raise.  Whenever I post
results to gcc-testsuite, I always include a statement of any patches
--- mine or other's pending --- that I have applied to my source tree.
I do this by having the summary tool send the output to me instead of
the gcc-testsuite list directly.  I then forward the mail with any
additional comments as needed to generally describe how the source
tree differs from the mainline (of course, only at the time the
bootstrap was started).  As I wrote up the proposed wording, I didn't
mention the importance of annotation since it slipped my mind.

Maybe I am too timid, but unless it falls under the obvious bug-fix
rule, I wouldn't dream of explicitly asking for one of my patches to
be applied to the gcc tree unless I can point to such a gcc-testsuite
posting.  Granted, I am a very infrequent gcc patcher (always limited
to simple portability and bug fixes for legal reasons), but I have
been following this model since before the 2.95 release cycle.  I am
not the only one that follows that model.  It might be true, in
general, that the further someone is from the core gcc developer
group, the more likely they are to post to gcc-testsuite when they
have pending patches... ;-)

This is not a rhetorical question: should we only be sending results
to gcc-testsuite generated from pristine (w.r.t. the mainline and/or
tagged releases) source trees?

>   Also, I was just trying to write down our current practices.  I
> don't think your suggestion is part of normal current practice; it's
> an additional step.

OK, I thought it was suppose to be normal practice to report test
suite results.  But, in light of your comments, I admit that I might
be a little overeager in my encouragement of others to send them in.

>   I can see that it proves that you ran the tests -- but I think we
> all trust each other.  I would certainly believe someone if they said
> they ran the tests, checked the results, and found no differences
> before a check-in.

>   Could you elaborate on what benefits you expect this to have?  

First of all, maybe I should be clearer too (I didn't capture this in
my proposed wording before): I don't expect people submitting small,
visually-inspectable patches to have to send in complete regression
tests for each incremental change (I too trust them to use good
judgment on exactly how and when they will run the full test suite and
review its results, if they tell me they do it).  But I would hope
that people changing large amounts of functionality in sweeping
patches and/or fundamental configuration changes would be held to my
earlier posted proposal.

I see two benefits:

1. People can't always answer e-mail about their recent changes.
Sometimes, just knowing how the person that made a recent major change
configures their tree verses yourself can help you debug a problem you
are now seeing in your environment.  This might allow the tree to
stabilize faster when it breaks, but I can offer no proof of that
assertion.  There are some people that make very important changes to
gcc that *never* post their results to gcc-testsuite (I won't name
names other than to commend one particular prolific contributor that
does regularly post to gcc-testsuite: Alexandre Oliva).  I *don't*
believe that others fail to post because they are lazy or not running
the test suite, but rather that they don't know that they should be or
how easy it is to do with the contrib script.

2. "Trust but verify (especially when technically easy to do for all involved)"

OK, the rules are that one is suppose to fully bootstrap and
regression test any changes that affect the entire project, right?

The time needed to run contrib/test_summary and forward with
annotations to gcc-testsuite is very minor compared to all the effort
that led to that point.  If people can't be bothered to do that last
minor step, then yes, I might wonder whether they really ran the
testsuite with their change.  I know that you have asked all patch
submissions to gcc-patches to contain a statement of where it was
bootstrapped and tested, but given that some people are still confused
as to the meaning of a proper full gcc bootstrap and what running the
test suite means, I think requesting a posting to gcc-testsuite will
help.  If they were strongly encouraged to post results, then they
could get more feedback and, hopefully, improve the way they
contribute to gcc.

A concrete example will help (a recent issue, but details removed to
protect the guilty ;-):

If someone told me that they bootstrapped the *entire* gcc tree just
before they checked in the recent patch to make --enable-XXX\=yes the
default (it was a small patch size-wise, but in terms of scope it was
a fairly major change), I would *not* believe them unless they could
explain how the errors found by myself and fnf didn't affect their
build environment (the only explanation that comes to mind is that a
certain non-standard library is part of libc on a certain operating
system ;-).  The tree is still broken a day later.  I suspect that the
person didn't know they were suppose to bootstrap and regression test
such a change.

Thanks you for allowing me to address your concerns.  As mainly a user
of gcc, I know I don't have a place to demand anything.

Perhaps you and the gcc steering committee would accept some wording
that only strongly encourages the step of submitting annodated results
to gcc-testsuite after generating make-check results for extensive
patches (in either size or scope)?

Regards,
Loren