GCC GIT mirror

There is a read-only git mirror of the gcc svn repository available at:

git://gcc.gnu.org/git/gcc.git
https://gcc.gnu.org/git/gcc.git
git+ssh://gcc.gnu.org/git/gcc.git

It can be browsed at gcc.gnu.org/git and is mirrored at repo.or.cz and GitHub.

Getting started - Read-Only

To work with git a local clone of the public repository is required. This is close to a svn checkout, but a clone copies the repository with the complete history. Therefore commands like "git log", "git status", "git diff", "git blame" can be executed without an internet connection.

First, create the clone. With a normal git repository 'git clone' grabs all branches, but because of the SVN mirror we need to do something more elaborate:

mkdir gcc; cd gcc
git init
# By default pulls all heads (trunk, release branches, git-only branches, a few other selected SVN branches)
git remote add origin git://gcc.gnu.org/git/gcc.git
# Also include all the other SVN branches (adds about 0.2GB)
git config --add remote.origin.fetch 'refs/remotes/*:refs/remotes/svn/*'
git fetch
git checkout -b trunk svn/trunk

Now the repository is available in the directory "gcc". Try some commands

> cd gcc
~/gcc> git status
# On branch trunk
nothing to commit (working directory clean)

To fetch changes from the official svn-mirror repository into your local repository

git pull --rebase

To check changes into your local repository

git commit -a

To extract patches for submission

# Show all changes on this branch that are not on the upstream branch, from earliest to latest
git log -p --reverse @{u}..

On branches

Our clone contains all the gcc branches.

List the available branches:

~/gcc> git branch -a
* trunk
  remotes/origin/HEAD -> origin/trunk
...
  remotes/origin/gcc-4_9-branch
  remotes/origin/jason/comdat-debug
...
  remotes/svn/gcc-4_9-branch
  ...

To work with one of these branches we need a local branch we can edit.

~/gcc> git checkout -b gcc49 svn/gcc-4_9-branch
Branch gcc49 set up to track remote ref refs/remotes/gcc-4_9-branch.
Switched to a new branch 'gcc49'

This means "create a new local branch gcc49 which tracks the remote branch svn/gcc-4_9-branch". Then you can switch between branches with

~/gcc> git checkout trunk
~/gcc> git checkout gcc49

Note that many SVN branches will also be in remotes/origin, but it's better to refer to the version in remotes/svn so that your branch will work with git-svn.

Also note that subdirectory branches like redhat/gcc-4_4-branch don't work this easily; see below for how to deal with those branches.

Advanced Usage

The above works fine for browsing the repository and simple hacking, but has various issues. Here's what seems to me to be the optimal way of using git-svn with the GCC repository. Please email (jason) or grosser@ if you have any questions/issues with these instructions.

Commit upstream (git-svn)

The repository we create above is read only. To commit changes upstream, patches still have to be committed into the svn repository. However, there is a way to commit changes directly out of your local git repository: git-svn

Commit on trunk

We start with the read only repository created above.

Add git-svn support to be able to commit

# Use this exact SVN url for gcc.git and git-svn to work together,
# except that "username" should be replaced with your username on gcc.gnu.org
git svn init -s --prefix=svn/ svn+ssh://username@gcc.gnu.org/svn/gcc

Integrate the latest upstream changes in the local repository

# Update index, fetch changes and rebase trunk onto the SVN trunk
# Generally doing 'git fetch' first is faster
git fetch
git svn rebase

See the changes we are going to commit

git log -p @{u}..
# OR
# git diff @{u}

Commit our local changes into the official gcc subversion repository

git svn dcommit

"git svn dcommit" commits automatically to the svn branch that the local branch you are working on was created from.

Working with branches

Subdirectory branches

If you want a branch that lives in a subdirectory of branches, such as redhat/gcc-4_4-branch, things are a bit more complicated due to a bug/shortcoming in git svn: it thinks the redhat directory is a branch by itself, rather than a directory containing multiple branches. There are multiple ways to address this. One way is to ignore the git mirror and use git svn fetch to fetch the branch from SVN directly:

~/gcc> git config --add svn-remote.svn.fetch branches/redhat/gcc-4_4-branch:refs/remotes/subdir/redhat/gcc-4_4-branch
~/gcc> git svn fetch

But this can take hours for the initial fetch. Alternately, you can checkout the whole redhat directory:

~/gcc> git branch redhat svn/redhat

With the second option, a checkout has all the redhat branches in subdirectories, rather than being able to check out a single branch at toplevel. If you only want one, you can use the git 1.7 sparse checkout function to limit what gets checked out:

~/gcc> git config core.sparseCheckout true
~/gcc> echo "gcc-4_4-branch/" > .git/info/sparse-checkout
~/gcc> git checkout redhat

Now only the files that match the patterns in .git/info/sparse-checkout will actually be present in the working tree. Remember to remove .git/info/sparse-checkout if you want to switch back to a normal branch. Or just do this in a separate working directory, as described below.

A third option works like the first without the slow initial fetch, but requires some git magic to rewrite the branch:

# Fork a local branch from the redhat directory branch
git branch --no-track r44tmp svn/redhat
# Rewrite redhat-gcc-4_4-branch to look like it was imported from the subdirectory
git filter-branch --subdirectory-filter gcc-4_4-branch --msg-filter 'sed -e "s,branches/redhat@,branches/redhat/gcc-4_4-branch@,"' r44tmp
# Remove the backup of r44tmp
rm -r .git/refs/original
# Set up git-svn to track the subdirectory properly
git config --add svn-remote.svn.fetch branches/redhat/gcc-4_4-branch:refs/remotes/subdir/redhat/gcc-4_4-branch
# Move the rewritten branch to where git-svn now expects to find it
mkdir -p .git/refs/remotes/subdir/redhat
mv .git/refs/heads/r44tmp .git/refs/remotes/subdir/redhat/gcc-4_4-branch
# And now we can check out a local copy to work in.
git checkout -b rh44 subdir/redhat/gcc-4_4-branch

A branch created this way won't be updated by fetch from the git mirror unless you repeat this process, but git svn rebase will update it just fine.

Working on multiple branches at once

To make a separate working directory for hacking on another branch at the same time, you can just create a new working tree. First make sure the branch is in your git repository, as above. Then make a local tracking branch, if you haven't already done that with checkout -b:

~/gcc> git branch foo-branch svn/foo-branch

Then use the git-new-workdir script from the git sources to create a working directory for that branch:

git clone git://git.kernel.org/pub/scm/git/git.git
./git/contrib/workdir/git-new-workdir ~/gcc gcc-foo foo-branch
cd gcc-foo

Now you have a directory gcc-foo which contains a checkout of your foo-branch. This directory uses the same git repository as your main directory, it just has a different branch checked out.

Note that using git cherry-pick to copy changes between SVN branches brings along svn metadata, but git-svn is clever enough not to be confused by this.

Merging SVN branches

To merge trunk into a development branch:

$ git checkout my_branch
$ git svn rebase
$ git merge --squash origin/trunk
[ ... Fix merge conflicts ... ]
$ git commit -m'Merge from blah blah'

NOTE: A normal (non-squashed) git merge works very badly with rebase, including git svn rebase; the rebase replaces the merge commit with a single copied commit for each of the commits brought in by the merge. If you then use git svn dcommit, git-svn will commit them separately, causing a lot of SVN traffic. Additionally, if the commits have references to bugzilla PRs, they will be used to spam bugzilla, which will eventually force you to send a message along the lines of https://gcc.gnu.org/ml/gcc/2011-02/msg00047.html. See also https://gcc.gnu.org/ml/gcc/2011-02/msg00096.html and other messages.

You have been warned. Use --squash!

Recent versions of git have support for svn:mergeinfo, but it sounds like this is still fragile, so using --squash seems like a more reliable solution. Setting branch.<name>.rebase to preserve might be useful.

Once that's done, your branch should look like this:

$ git status
# On branch my_branch
# Your branch is ahead of 'origin/my_branch' by 1 commit.
#
nothing to commit (working directory clean)

$ git svn dcommit --dry-run
Committing to svn+ssh://gcc.gnu.org/svn/gcc/branches/pph ...
diff-tree e63f... e63f...

$ git log -1
commit e63f...
Author: My Name  <my@address>
Date:   Thu Feb 3 12:55:55 2011 -0500

   Merge from blah blah

Test the merge and then commit with

$ git svn dcommit

Git-only branches

To allow others to view your changes online you can push your local branches to the Git mirror. To set up your git repository for this, do

$ git config remote.origin.pushurl ssh://gcc.gnu.org/git/gcc.git
$ git config remote.origin.push mybranch:myname/mybranch

You are now set up to push your local branch "mybranch" to "myname/mybranch" on gcc.gnu.org. Whenever you want to do this, just

$ git push

If you want to push another local branch, you can

$ git push origin otherbranch:myname/otherbranch

Please keep personal branches in a subdirectory based on your gcc.gnu.org username. Non-subdirectory branches can be used for collaborative development.

The git repository is configured to deny non-personal branch deletion requests; if you really need to delete a non-personal branch, you can email jason about it.

Usage hints for git

git-merge-changelog

Before long you'll get frustrated with git's handling of ChangeLog merges, which is just as bad as SVN's. But there's a fix for that!

git clone git://git.savannah.gnu.org/gnulib.git
cd gnulib
# alternately, install gnulib from a package manager with something like
# 'sudo apt-get install gnulib' and just run /usr/bin/gnulib-tool
./gnulib-tool --create-testdir --dir=/tmp/testdir123 git-merge-changelog
cd /tmp/testdir123
./configure
make
make install
git config --global merge.merge-changelog.name "GNU-style ChangeLog merge driver"
git config --global merge.merge-changelog.driver "/usr/local/bin/git-merge-changelog %O %A %B"
echo "ChangeLog   merge=merge-changelog" >> ~/gcc-git/.git/info/attributes

NOTE: git-merge-changelog used to be extremely slow for cherry-picking changes from trunk to release branches, but this was fixed with a change on 2009-07-02.

Update bash function

Here's my generic "update this source tree" bash function.

up() { if [ -d CVS ]; then cvs -q update "$@";
       elif [ -d .svn ]; then svn up "$@";
       elif gitd=`git rev-parse --git-dir 2>/dev/null`; then
           if [ -f $gitd/objects/info/alternates ]; then
               for d in `cat $gitd/objects/info/alternates`; do
                   (cd $d; git fetch) || return 1
               done
           fi
           git fetch || return 1
           if p=$(git rev-parse '@{u}' 2>/dev/null); then
               # If we have a defined upstream branch, use that
               us=$p;
           else if git svn info >/dev/null 2>&1; then
               us=`git svn log --show-commit --max-count=1 --oneline|cut -d' ' -f3`
           else
               echo "no upstream defined for current branch, use 'git branch -u' to set it" >2
               return 1
           fi
           if test -z "$(git log --merges $us..)"; then
               git rebase $us
           else
               # Don't rebase a merge
               git merge $us
           fi
       fi; }

rebase -i: Squashing multiple local commits into a single SVN commit

git rebase -i is a very useful tool for organizing local changes for svn dcommit.

If you have been making changes to your local tree, addressing review feedback, making minor fixes, etc. your local tree will have many commits. So, when you finally push the patch to SVN with 'git svn dcommit', you will be pushing all the local commits separately, which may not be what you want; a good rule of thumb is to have one commit for each patch sent to gcc-patches. To commit all your changes as a single SVN transaction, you need to squash all your local patches together. To do this:

git fetch
git svn fetch -p
git rebase -i @{u}

The -i switch causes git rebase to ask you which patches you want to pick or squash. Reorganize the lines so that all the related changes are grouped together, and within a group of changes that you want to combine, replace 'pick' with 'squash' or 's' (or 'fixup' or 'f' if you don't need to adjust the commit message) on all lines except the first one. Then save the file and let 'git rebase' finish.

After this, the change(s) in your local git branch should look more like the patch(es) you want to send to gcc-patches, which you can then extract separately with git log -p for emailing to gcc-patches; see the 'git lgp' alias in the next section. Now, when you run 'git svn dcommit', you will push these patches to SVN rather than your local edit history.

While you are working, if you already know that you want your new commit to be squashed with a previous commit, you can use commit --squash or --fixup to specify that; rebase -i --autosquash will then reorganize the patches automatically.

To apply some but not all of your current commits to SVN, use git rebase -i to move the ones you want to apply to the top of the list, select "edit" for the last one you want to apply, and do git svn dcommit before git rebase --continue.

Removing accidentally committed hunks from a commit

If you look at your local commit log and notice that there's a hunk in one of your commits that doesn't belong with it, it's reasonably easy to edit it out. If the commit is not HEAD, you can use rebase -i 'edit' to go to that commit for editing. Once the commit in question is at HEAD, I do

# Move HEAD to the previous commit, leaving the commit I'm editing in the index.
git reset --soft HEAD^
# Go through all the hunks interactively to choose what to remove from the index
git reset -p
# Re-commit everything that's left in the index, using the same commit log as before
git commit -C ORIG_HEAD

At this point the commit is edited and the hunks you removed are only in the working tree. Now you can give them their own commit or discard them.

Useful aliases

Here are some aliases I'm finding useful (to add to your ~/.gitconfig). Note that these only work with the advanced setup due to differences in the branch.$BRANCH.merge config.

[alias]
        st = status
        ci = commit
        br = branch
        co = checkout
        cp = cherry-pick
        sr = svn rebase
        sci = svn dcommit
        dir = rev-parse --git-dir
        # The current branch.
        cbr = "!f(){ expr $( ( git symbolic-ref -q HEAD || cat $(git dir)/rebase-merge/head-name ) 2>/dev/null ) : 'refs/heads/\\(.*\\)'; }; f"
        # The branch being tracked by the current branch.
        track = "!f() { if p=$(git rev-parse --symbolic-full-name '@{u}' 2>/dev/null); then echo origin/${p##*/}; else git svn info|sed -n 's,^URL.*gcc/\\(branches/\\)\\?\\(.*\\),origin/\\2,p'; fi; }; f"
        # Show all the local commits on this branch.
        lg = "!git log -p `git track`.."
        # Write all the local commits to ~/patch, filtering out modifications to ChangeLog files
        lgp = "!git log -p `git track`.. | filterdiff -x '*/ChangeLog' | sed -e '/^diff.*ChangeLog/{N;d}' > ~/patch"
        # Show all the local changes on this branch as one big diff.
        df = "!git diff $(git merge-base $(git track) HEAD)"
        dfc = diff --cached
        # Reorganize the local commits on this branch.
        rb = "!git rebase -i `git track`"
        rc = rebase --continue
        # 'git rmerge mybranch' to reintegrate a temporary branch onto the top of the current branch
        rmerge = "!f(){ cur=`git cbr`; git rebase $cur $1; git rebase $1 $cur; }; f"
        # Squash everything since the specified commit (or HEAD^) into that commit
        sq = "!f(){ git reset --soft ${@:-HEAD^} && git commit --amend -C HEAD; }; f"
        # Pop everything since the specified commit, but leave it in the index
        rs = "!f(){ git reset --soft ${@:-HEAD^}; }; f"
        # Choose patch hunks to remove from the index (e.g. after doing git rs); after removing some hunks I then do a commit -C ORIG_HEAD
        rp = reset -p
        # Show the commit corresponding to an SVN revision
        ssh = "!f(){ git show $(git svn find-rev r$1); }; f"

Pushing to a testing box

I tend to do development on my laptop, and then send patches off to a hefty compile server to do full regression testing. Before git, I did this by rsyncing diffs, but git simplifies the process. First, add to your gcc-git/.git/config:

[remote "testbox"]
        url = testbox.foo.bar:gcc-git

Then to push your current changes to the testbox, just "git push testbox +trunk". Note that you'll then need to "git co -f" or "git reset --hard" on the testbox to update the working copy from the git repository. This could be another alias:

        tup = "!f(){ cur=`git cbr`; git push $1 +$cur && ssh $1 \"cd src/$cur; git co -f\"; }; f"

So then you would write "git tup testbox" (assuming that "ssh testbox" also works). The alias could then proceed to run tests directly, but that is left as an exercise for the reader (as my setup involves a bunch of scripts).

Emacs

GNU Emacs 23 has decent git support in the vc package. The main attraction is C-x v g (vc-annotate), a handy interface to git blame that lets you step back to the revision before the line you're looking at with the 'a' key. But since it doesn't run asynchronously yet, I use "git gui blame" instead.

I also find that keeping the modeline updated via vc-state is unacceptably slow because it wants to call "git status", which can take more than 30s to load the entire repository into memory. To work around this, I added this to my .emacs:

;; Break vc-git-state because it slows down editing GCC source too much.
(defun vc-git-state-heuristic (file)
  "Just claim we're up to date."
  'up-to-date)

For editing git-related temporary files in emacs, I set my EDITOR environment variable to "emacsclient -t" and added the following to my .emacs:

(defun maybe-truncate ()
  (interactive)
  (cond ((string-match "/git-rebase-todo\\>" buffer-file-name)
         (toggle-truncate-lines))))
(add-hook 'find-file-hook 'maybe-truncate)

Old Information

Below here is information that no longer describes my use of git, but that other people might find interesting.

Making Patches

I currently make patches for gcc-patches using my 'git lgp' alias, above.

However, git always produces unified diffs, while the GCC project policy is to prefer context diffs. I don't bother with this, but if you really want to make a context diff, you can create a wrapper script somewhere:

cat > ~/bin/git_diff_wrapper <<EOF
exec diff -p -L "$1" "$2" "$5" | cat
EOF
chmod +x ~/bin/git_diff_wrapper

and when you want to make a context diff for submitting a patch, do

GIT_EXTERNAL_DIFF=git_diff_wrapper git diff trunk

Further Reading

Git crash course for SVN users

git-svn tutorial

Using git without feeling stupid (part 1), by Paolo Bonzini

Using git without feeling stupid (part 2), by Paolo Bonzini

http://git-scm.com/documentation/

None: GitMirror (last edited 2014-09-10 14:17:26 by JasonMerrill)