Commit messages and the move to git

[Richard Earnshaw (lists)]

On 09/11/2019 06:01, Eric S. Raymond wrote:
> Richard Earnshaw (lists) <Richard.Earnshaw@arm.com>:
>> Which makes me wonder if, given a commit log of the form:
>>
>>
>> 2019-10-30  Richard Biener  <rguenther@suse.de>
>>
>>          PR tree-optimization/92275
>>          * tree-vect-loop-manip.c (slpeel_update_phi_nodes_for_loops):
>>          Copy all loop-closed PHIs.
>>
>>          * gcc.dg/torture/pr92275.c: New testcase.
>>
>> Where the first line is a ChangeLog style date and author, we could spot the
>> PR line below that and hoist it up as a more useful summary (perhaps by
>> copying it rather than moving it).
>>
>> It wouldn't fix all commits, but even just doing this for those that have
>> PRs would be a help.
> 
> Speaking from lots of experience with converting old repositories that
> exhibited similar comment conventions, I would be nervous about trying
> to do this entirely mechanically.  I think the risk of mangling text
> that is not fornatted as you expect - and not noticing that until the
> friction cost of fixing it has escalated - is rather high.
> 
> On the other hand, reposurgeon allows a semi-neechanized attack on
> the problem that I think would work well, because I've done similar
> things in ither coversions.
> 
> There's a pair of commands that allow you to (a) extract comments from
> a range of commits into a message list that looks like an RFC822
> mailbox file, (b) modify those comments, and (c) weave the the message
> list reliably back into the repository.
> 
> If it were me doing this job, I'd write a reposurgeon command that
> extracts all the comments containing PR strings into a message box
> Then I'd write an Emacs macro that moves to the next nessage and
> hoists its PR line.
> 
> Then I'd walk through the comments applying the macro and keeping an eye on
> them for cases where what the macro doesn't do quite the right thing and
> using undo and hand-editing to recover.  Human eyes are very good at
> spotting anomalies in an expected flow of textm and once you've gotten
> into the rhythm of a task like this is is easily possible to filter
> approximately a message per second. In round numbers, providing
> the anomaly rate isn't high, that's upwards of 3000 messages per hour.
> 
> The point is that for this kind of task a hnman being who undertands
> what he's reading is likely to have a lower rate of mangling errors than
> a program that doesn't.
> 

The hook into reposurgeon is even better... it allows a more intelligent 
hook to be produced, for example the attached.

In a reposurgeon lift script you can write something like

~gcc2 msgout >gcc.commitlog
shell ./fixbugmessages
msgin <gcc.fixedprs

I wrote this script for two reasons
    1) To learn some python (finally I had a good reason to go and do 
this :)
    2) To try to improve some of our legacy commit messages, especially 
where they appear in git as just the name of the author (information 
that is already readily available in other components.

It works by scanning for commits that match a traditional ChangeLog 
style author line and then, if the commit mentions a PR, looking that
up in the Bugzilla database to extract the original component and the 
summary line from the PR.  It then produces a single line summary based 
on the PR and the summary associated with it.  So for example, a commit 
such as this one:

2019-10-30  Richard Biener  <rguenther@suse.de>

	PR tree-optimization/92275
	* tree-vect-loop-manip.c (slpeel_update_phi_nodes_for_loops):
	Copy all loop-closed PHIs.

	* gcc.dg/torture/pr92275.c: New testcase.


Would be matched and the line

    PR tree-optimization/92275 ICE: error: definition in block 11 does 
not dominate use in block 15 since r277566

added as a summary

Since there are a number of errors that over the years have crept into 
the commit logs, there are also features to try to match (where they are 
specified) the bug component against that listed in the commit log. 
That matching is fuzzy, since it too can be inaccurate: many bugs, for 
example, are classified as target in the PR, but subsequently get a 
commit component that reflects where the fix was applied.  So there are 
some filters to simplify reporting to just those categories that look 
obviously suspicious (like a fortran bug with a C component reported). 
There are also facilities to mark some commits that have been manually 
verified as clean so that they are no-longer reported.

The script generates new summary lines for nearly 43000 commits that 
would mostly have just shown the author line.  Of those about 560 are 
flagged as potentially suspicious and will need some manual checking.

R.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bugdb.py
Type: text/x-python
Size: 18253 bytes
Desc: not available
URL: <https://gcc.gnu.org/pipermail/gcc/attachments/20191118/b303d23b/attachment.py>
-------------- next part --------------
#!/bin/sh

# Note the largest bugid is currently less than 100000, so we strip
# off any above that as they must clearly be bogus

egrep "\bPR( ?#?[0-9]+|[ 	]+[^/ 	]+/[0-9]+)([[:punct:]]|[ 	]|$)"\
      gcc.commitlog | \
    sed -E "s:\bPR( [a-zA-Z][-+a-zA-Z0-9]*/| ?#?)([0-9][0-9]*).*:x\2:" | \
    sed -E "s/.*x([0-9][0-9]*)/\1/" | \
    sort -n | \
    uniq | \
    sed -e "/.......*/d" > bugids.list

./bugdb.py