on patch queues (reply to Joseph S. Myers)

[Tom Lord]

Here is a revised and more detailed proposal for patch "queue"
automation (though it's a pool, not a queue).

================================================================


* Joseph S. Myers: Patch Queues Good / 2-Phase Process Bad

	We could do with patch queues (preferably a simple system
	based on a bot reading gcc-patches) to reduce the rate of
	patches going unreviewed, but this is orthogonal to version
	control changes, and I think a formal system of merge windows
	such as you describe is unlikely to be properly effective for
	entirely volunteer developers in different timezones who may
	have other commitments for a few weeks preventing them from
	doing the merge, etc..  The problem is more one of
	contributors getting tired of keeping doing merges to update
	their patch while review goes slowly or legal paperwork needs
	to be done.


I understand what you are saying about contributors.

Version control tools are still relevant to automation: they can still
make life easier for maintainers, by automating some work,
distributing some work, and helping to plan the best order for work.

Here's a revised automation proposal that I think more closely matches
your requirements.


1) As you say: a simple bot that grabs patches from the gcc-patches
   list.

   Is it reasonable to require (or, more likely, to simply _favor_)
   patches with two properties?:

	A) The contributor identifies a recent mainline revision (or
	   specific release) against which the patch is claimed to
	   apply cleanly.  (And, see my other reply: this
	   identification can be automated by the `mkpatch' tool.)

	B) The contributor prepares the patch with a stand-alone
	   tool, `mkpatch', which is independent of any particular
	   revision control system, but which prepares whole-tree
	   patches and formats them in a flat-file, email-friendly
	   manner for which a formal specification has been prepared.

   I will assume, for now, that at least favoring patches with those
   properties is acceptable, and that the burden they impose on
   maintainers is negligible.


2) The bot owns it's own repository.  For security and bandwidth
   reasons, ideally a second server can be scrounged for this
   repository -- so it does not reside on the machine hosting the
   mainline.  At the very least, it is probably a second repository,
   owned by a distinct user id.


3) Upon receipt of a qualifying patch (one with the two properties
   mentioned above), the bot forms (in its own repository) a local
   branch of mainline from the indicated base revision.  It applies
   the patch and commits the result.

   (Upon receipt of a non-qualifying patch, the bot simply archives
   the text of the patch itself.  A maintainer can work with that
   directly, or convert it to a qualifying patch and re-submit it in
   that form.  Thus, contributors still using `diff -r' are not 
   harmed by this new automation.)


4) The bot produces a web page, documenting the status of pending 
   patches.  Cron-jobs on the bot machine periodically update that
   status.

   The cron job generates a list of pending patches by scanning
   archive and log message contents.

   The cron jobs will periodically try various forms of merging
   each patch with the head revision on mainline --  there are
   several merge techniques that might work, and the bot can try
   more than one of them.   Thus, the status report for each patch
   can identify fairly early when mainline divergence has introduced
   conflicts with the patch, and can give maintainers a hint about how
   to merge a given patch.   This will give maintainers an early
   indication of how much non-trivial merging work is implied by
   the pending patch pool, and sometimes speed them up at performing
   merges.


5) _Some_ contributors may be willing to keep their (or other
   people's) patches up-to-date while paperwork and other issues are
   being processed.  We can make it easier for them to do that.

   In particular: any such contributor can form, in their own local
   repository, branches from the branch for a particular patch.
   They can work on problematic updates locally, and then mail
   the additional changes.

   The bot can capture these additional changes and (according to
   policy) apply them either to the original patch branch, or to
   a new branch of that patch branch.

   So while contributors aren't required to hit any particular
   window, they will still have the opportunity to eagerly keep
   their patches up-to-date.

   This also may help with testing.   A popular, pending, qualifying
   patch is available for checkout and testing.  Thus, independently
   of any maintainer action, contributors can obtain early testing
   from 3rd parties and update their contributions accordingly, all
   coordinated via the project's services.
   

6) As in the 2-phase set-up, a cron-job can examine the patch pool
   and figure out which patches are (textually) orthogonal and which
   conflict -- thus producing (for the web page) a list of conflicts
   that will arise as patches are merged.  Maintainers can use this
   information to allocate their time more effectively (an orthogonal
   patch can go quickly;  conflicting patches may require more time or
   a careful choice of which one goes first).


7) The patch-pool repository can be used to help with vetting.
   In particular, review and testing activities can be separated from
   merge activities in a natural way:  a reviewer or tester can commit
   to individual patch branches, noting their review outcome in the 
   log message in an RFC822-style header.  This information can be 
   collected and incorporated into the patch pool status page.

   To the extent that servers are available for automated testing,
   they can be applied to patch branches in addition to mainline.


8) Please take a look at my answer to Stan Shebs ("new batch of
   replies (C)").  Briefly, I suggested improvements to the way
   that conflicts are recorded in a tree such that it would be
   practical to commit and otherwise work on archived revisions
   that contain conflicts.   With this feature, maintainers could
   work on merging particular patches incrementally, committing 
   the intermediate states, and perhaps getting help from other
   maintainers of the same patch or feedback from the contributor.


In summary, distributed revision control helps manage the patch pool
by:

	1) Creating a safe and powerful mechanism for archiving
	   pending patches.

	2) Automating some of the trial-and-error work that can help
	   discover the best merging technique for a particular patch,
	   and the best order in which to merge patches.

	3) Encouraging and facilitating (but not requiring)
	   contributors to make controlled improvements to their
	   pending patches while they are still pending.

	4) Creating a safe and powerful mechanism for archiving
	   review and testing results.

	5) (If we enhance "commit with conflicts" functionality):
	   facilitating fine-grained, asynchronous cooperation among
	   maintainers when merging patches with many conflicts.

Distribution, orthogonal implementation of whole-tree patch
manipulation, rich facilities for merging, a convenient log message
format, optionally -- the ability to make controlled commits with
conflicts: these features of revision control all come into play.



================================================================