• ForgeExperiment

Experiment: Forge-based workflow for Sourceware projects

Background

The software projects hosted on Sourceware have been around for many years, with some pre-dating the creation of the world-wide web. Although their development flows have evolved over the years (Emacs revision files, RCS, CVS, Subversion, GIT) they are still largely based on an email driven workflow. But email has its problems:

• It cannot capture patch series meta-data accurately (it has to be reconstructed from message-Id chains)
• Corporate email policies are often not friendly to submitting patches to free software projects
• HTML has largely become the defacto format for email content, which is not appropriate for sending diffs in text files
• It's never been fully 8-bit clean, leading to potential issues with line encodings
• Spam is an ever-present problem. While spam filters eliminate the bulk of spam, they are not perfect and may occasionally filter valid patches
• Security extensions to email, such as DKIM and SPF can lead to delivery problems.
• New developers are often more familiar and productive with forge-based workflows than email-based ones. Most are not even aware that an email workflow is possible (Github claims to have 100M users). To them, the workflow is a barrier to overcome rather than an effortless tool.

It's time to experiment with potential alternatives to identify whether we should adopt something else.

This experiment does not imply a commitment to move to a new workflow, we need first to establish whether it is possible and what advantages it would bring.

Current state of the forge experiment

• The Sourceware forge forgejo is available for experimenting.

  • It is based on Forgejo, which is licensed under the GPL.

• The instance contains organizations that mirror existing projects hosted on sourceware.
• Read only mirrors of git repositories are available in their respective organizations
  • Forks of these mirrors are available for developers and maintainers to experiment with (these end in -TEST)

• The instance's configuration is documented and made available in the forge repository.

  • An experimental ansible-based version can be used to quickly recreate the environment on a VM or your infrastructure

• The forge only allows people with sourceware.org or gcc.gnu.org email addresses to sign up.

  • The forge only allows forking an existing project from within the server. It is not a general server for hosting software projects
• There is a bot account (forge-bot) which implements a workflow that assists forge users with posting patches on the mailing list. It is currently working on:

  • the forge repository, sending emails to the forge mailing list.

  • the gcc-TEST repository, sending emails to the test-list mailing list.

• The code behind the bot is published in the batrachomyomachia repository on the forge itself. Please contribute!

If you are a maintainer of a project and wish to enable the forge bot for your repository, please get in touch with the overseers over IRC.

Please take note:

• The following resources are experimental, they may disappear at any time, either temporarily or permanently. Given that this is an experiment, it could fail for various reasons. At that point we could decide to just throw everything away. But we wouldn't do that randomly and people are willing to let the experiment run for at least a year before deciding it does or doesn't work. And we would of course give people the chance to migrate their work they want to preserve somewhere else (forgejo has good import/export to various other forges).

• As this is an experiment, expect the server to be unstable at times.
• The sourceware admins are being helpful, but they are only human, are fairly new to forgejo themselves and have other responsibilities as well.

The proposed workflow and how to make use of the forge

(This is a basic description, you can find more details and options in the sections below)

One time set up

1. Create an account using your sourceware.org or gcc.gnu.org email address.

2. Please do not try to use a language other than English as our spam filters will silently drop emails generated by the forge.

3. Add one or more public ssh keys in your user settings and make sure your ssh uses them when accessing the forge

4. Fork the repository you want to work on (being mindful not to fork the repositories with -mirror in their name!)

5. cd into your local clone of the repository and add a git remote that points to your fork. For example: git remote add forge ssh://git@forge.sourceware.org/XXXX/gcc-TEST.git (change XXXX to your username)

6. The first time you propose a change, you will have to ask someone to /allow you to perform the next steps. This can be anyone that has been allowed before. Once allowed, you will be able to allow others. This is a light weight process to stop spammers. Please only allow people writing bona fide pull requests. (example)

See also the Setting up your fork section below.

Posting a change

1. Create a branch for the change you're making and add your commits

2. Use git push forge to push your branch to your fork.

3. Open your fork on the forge.
4. Under the tab where the number of commits is, you'll find the dropdown list of branches. Open your branch and then click on the button next to the branches dropdown. This will start the New Pull Request workflow.
5. Add your cover letter as description of the pull request.
6. The forge-bot account will post a welcome comment providing up to date instructions.

7. When you are happy with the state of the pull request, add a comment with only the word /preview. The forge bot will record the current state of yor branch and post a patch series to your email for you to check.

8. If the patch series looks ok, add a comment with only the word /submit. The forge bot will send your patch to the right mailing list and post a comment with a link to the cover letter in the relevant inbox.

Post a new version of a patch series

1. Rebase and change your commits on your local clone

2. Use git push --force forge to push your branch to your fork.

3. Update the pull request description with the changes you've made.

4. Repeat the /preview or /submit commands. The forge bot will send the email on your behalf.

Reviewing changes on the forge

1. You need to have an account in order to comment on people's pull requests. Create one using your Sourceware email address.

2. The default view of a pull request is the Conversation. You can enter free flow comments but this does not necessarily start a review.
3. Head to the Files changed tab. This will show you a list of files changed. Hovering over the changes, a cross appears on the edge. Clicking on the cross will allow you to write comment on a specific line of code. The comment can be standalone or part of a wider review spanning multiple comments.
4. When you're done with a specific file, you can use the Viewed tick box to hide it.
5. When you've completed reviewing single files, click on the Finish review button. This will give you an opportunity to write a wider comment on the entire change. You have the option of Approving, commenting or requesting changes are made.

Monitoring and merging changes

1. There is currently no wide agreement on whether review on the forge is enough to push patches. Monitor the mailing lists for replies and act accordingly.
2. Merging your patches can be done as usual by pushing to the upstream repository, if you have write privileges and obtained approval per project policy.

A bullet point list of community requests

Most of these bullet points stem from Joseph Myers' excellent write up and subsequent discussion.

Policy/Social/Design

1. There is no need for each sourceware project or gcc component to make the same decision about moving to such software and workflow.
2. Hosting must meet appropriate security requirements.

3. Software must achieve a passing grade on the GNU Ethical Repository Criteria.

4. The software would need to be entirely free software.
5. It's important that the forge software is receptive to the addition of features.
6. We should be able to disable bug tracking and wiki and use existing ones for the time being.
7. Provide a more familiar workflow that avoids many of the problems with email.
8. No change in policies regarding who can approve a change for inclusion.
9. No change in policy for obvious changes. (If someone can approve, they can self-approve, pending CI not failing)

10. Simplicity of any solution is a plus. (If what you end up doing is too complicated, it may end up severely impacting the gfortran project (and possibly others)).

11. Open question: The community should decide whether to require opening PRs for backport commits.
12. Open question: should there be a bot that rebases and merges once CI works? Or do we use the LLVM method of CI running and reverting changes that cause issues.
13. (Richard E.) Some accessibility to those lacking javascript should be afforded.
14. The application of a forge based workflow should not adversely impact the work of developers that operate on the middle-end and, consequently, on multiple back ends at the same time.

Mechanics

1. The forge should have API support to allow people to build tools supporting their own workflow.
   • Following their own logic, not the forge's, for what changes are of interest.
2. The forge provides appropriate interfaces to allow integration of external customisations.
3. The forge must provide structured data about proposed changes and their status (to help all people involved to track information and find patches needing review).
4. All existing pre-commit checks from hooks should be kept in some form, to maintain existing invariants on both tree and commit contents.
5. Existing cron jobs that read from or commit to repositories should use normal remote git access, not filesystem access to the repository, including using APIs to create and self-merge PRs when pushing to the repository is involved.
6. No change in the technical limits on who can cause a change to enter mainline.
   • (Claudio) Random Contributor must be able to log into forge, create a fork of the repository, add patches on top and then create a PR. This means that access to the forge should be less strictly controlled than it is now. Some spammer protection is in order. Will anubis be enough?
7. All people with commit access must be able to use a button in a web interface to cause to PR to be merged.
   • Though policy might limit when they should do so
8. Support for reviewing the proposed commit message, not just the diffs.
9. Patch series and dependencies between patches are important.
   • Series can sometimes have hundreds of patches; any solution for patch series and dependencies needs to scale that far.
10. All revisions of the changes for a PR that was rebased need to remain permanently available.
11. Replication for reliably ensuring key data remains available, even if the software is no longer maintained.
12. API allows maintaining local copies of all PR comments etc. in a plain text form that can be searched with grep.
13. Good plain text outward email notifications for all new merge requests.
14. Good plain text outward email notifications for all comments thereon
15. Good plain text outward email notifications for significant actions (changing metadata, approvals / merges, etc.)
16. Such notifications would go to the existing mailing lists.
17. The choice of mailing lists for a patch can depend on which parts of the tree it changes
18. Some very routine changes, such as reports of clean CI results, might be omitted by default from notifications sent to the mailing lists.
19. "good" includes quoting appropriate diff hunks being commented on.
20. The quality of the plain text part in email matters.
21. Diffs themselves should be included in new-PR emails (and those for changes to a PR) unless very large.
22. It should be possible for people to push directly to mainline as well.
23. Direct commits to branches other than mainline (and maybe release branches) should continue to be OK long term (although it should also be possible to make a PR to such a branch if desired).
24. Merges to mainline should be able to go through a CI system that makes sure there are no regressions for at least one configuration before merging changes.

Non issues

1. The forge will not prevent the accumulation of unreviewed patches
2. Trying to take incoming email (unstructured) and turn it into more structured comments on particular parts of a PR in the database.
3. CI might take long time and cause some inconvenience for someone who only edits something like MAINTAINER lists.

Maintainer requests captured during the Cauldron in Porto 2025

Richard Earnshaw: I want a system where: I see when a patch series has been reposted and know when issues have been resolved, without having to manually track all emails. Where I can keep track of everything that has to do with a review together (Searching for something on my corporate email client sucks)

J. Wakely: Currently the forge is only usable by people who already have write access, and those people already know how the email workflow works and have no great motivation to switch. It’s one-way, it can't push back to the real repo

Richard Biener: I worry that introducing the forge will split the community and make working on maintainership harder

Kyrylo Tkachov: Ability to connect AI agents that provide automated reviews (Structure in data is useful)

Alexandre Oliva: I really enjoy the email workflow. I would like it to work well with emacs. I care about not having to enable javascript in my browser. I worry about proprietary authentication frameworks in javascript

Andreas Hüttel: I want a system that helps me test on multiple architectures

Nick Clifton: I want to bring in new reviewers and contributors and to simplify the process

David Malcolm: I want CI along with the patches. And less wall of text from the bot.

Tamar Cristina: I write patches spanning multiple areas and simple PR workflows can make pushing them painful.

Andrew Pinski: Can we not take 5 years to do this?

Known issues, potential enhancements

List of known issues

There are a number of known issues and limitations. Some of these may need attention from the Forgejo developers.

Branch synchronisation

One issue is that the trunk and master branches in the gcc-TEST repo are distinct branches. In the official gcc.gnu.org repo they are the same branch (trunk is a symbolic-ref pointing to master). This means that making changes to master in your fork of gcc-TEST will not update trunk, and vice versa.

A significant usability problem today is that unlike the read-only gcc-mirror repo, the gcc-TEST repository is not automatically updated from the official repo on gcc.gnu.org. In order to keep it up to date somebody needs to run these commands periodically (usually JonathanWakely when he remembers to do it):

git -C $GCCSRC fetch -q origin
git -C $GCCSRC push  -q git@forge.sourceware.org:gcc/gcc-TEST.git origin/master:trunk
git -C $GCCSRC push  -q git@forge.sourceware.org:gcc/gcc-TEST.git origin/master:master

The commands above assume that $GCCSRC refers to your local clone which has origin referring to https://gcc.gnu.org/git/gcc.git and they require a working SSH key that allows you to push to the forge (making it difficult to automate via cron). It's necessary to push to both trunk and master because on the forge those are distinct branches.

The batrachomyomachia automation can periodically sync specific refs. This is done by specifying upstream_url,mirror_schedule and mirror_refs_mappings repository configurations.

    upstream_url: "https://gcc.gnu.org/git/gcc.git"
    mirror_schedule: "*/30 * * * *"
    mirror_refs_mappings:
      - "refs/heads/master:refs/heads/master"
      - "refs/heads/master:refs/heads/trunk"

Forge emails are not adequate for mailing lists

Forgejo has support for sending emails when certain events occur but:

• users have little control over what they can subscribe to
• the formatting is not quite how some would like it to be.

The batrachomyomachia automation was written to generate customizable emails when certain events occur, such as when a pull request author writes a specific comment. This allows the community to quickly improve it based on their needs and eventually get changes into forgejo proper.

Possible Enhancements

For development branches (e.g. the algol68 one) it could be useful to have issues on the forge, since it's not appropriate to use BZ since the content is not yet in trunk.

Tips & Tricks

Details on setting up your fork and what to clone

Do not fork the -mirror repos, because you won't be able to create any Pull Requests from your new fork. The -mirror repos don't accept pull requests, they only sync from the main git repositories and don't allow any other changes. If you made a fork of a mirror repository, you must delete your fork before you're able to clone the proper repository.

Make your own fork of the gcc-TEST repo by clicking the "Fork" button near the top right of the page. You might want to name your fork just "gcc" instead of "gcc-TEST" (the chosen name doesn't affect forge functionality). Once you've created that fork, the URL for authenticated access over SSH will be shown just above the list of files, on the right edge. You can use that URL to create a remote in your local Git tree, so that you can push to your fork, e.g. git remote add forge git@forge.sourceware.org:<user>/gcc.git (replacing <user> with your forge username).

When you create a pull request, first push your work to your fork, e.g. after setting up the remote as above:

$ git status
On branch refactor-stuff
Your branch is ahead of 'origin/master' by 14 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean
$ git push forge refactor-stuff:refactor-stuff
Enumerating objects: 71, done.
Counting objects: 100% (71/71), done.
Delta compression using up to 12 threads
Compressing objects: 100% (37/37), done.
Writing objects: 100% (41/41), 12.35 KiB | 4.12 MiB/s, done.
Total 41 (delta 35), reused 8 (delta 4), pack-reused 0 (from 0)
remote: Checking connectivity: 41, done.
remote: 
remote: Create a new pull request for 'redi:refactor-stuff':
remote:   https://forge.sourceware.org/gcc/gcc-TEST/compare/trunk...redi:refactor-stuff
remote: 
To forge.sourceware.org:redi/gcc.git
 * [new branch]              refactor-stuff -> refactor-stuff

Notice that the remote server tells you the URL to use to create a new PR! If you want to do it manually, you can go to your fork in your browser and there should be a notice at the top saying something like "You pushed on branch refactor-stuff two minutes ago" and a button to create a new pull request from that branch, which takes you to the same URL that was printed when you pushed the branch.

Using AGit to quickly create a Pull Request

Forgejo currently supports the AGit workflow described on this page. There's some documentation on how to do this.

The gist is: - You don't have to create a fork - You only have to clone the repository on the forge and create a local branch. - You push changes using a specific refspec that will be familiar to Gerrit users

git push upstream HEAD:refs/for/main -o topic="reuse" \
  -o title="Complete licensing information" \
  -o description="This change adds licensing information according to the recommendations in version 3.3 of the REUSE specification. See https://reuse.software/ for more info."

One major limitation (as of Oct 2025) is that it's not possible to pass a multiline option to git options.

Linking your repo to Bugzilla

It is possible to link your repository to our bugzilla instance. Once you have created your fork, go to the Settings->Units->Issues, then enable issues and select External tracker. For GCC, the settings are

This will match patterns of the form  component/issue-number  and create a link to the external bugzilla entry.

Avoid using "#1234" in commit messages

In a Git commit message, or in an online comment on the forge, Forgejo treats any string like #1234 as a reference to issue 1234 or pull request 1234 in the current repo, wherever the commit or comment appears. (This auto-linking behaviour is consistent with GitHub and GitLab). This means that a Git commit message that uses something like "PR #1234" to refer to a GCC Bugzilla bug will get linked to a (possibly non-existent) issue in the Forgejo repo instead. Using "PR 1234" or "PR middle-end/1234" would be preferable, and hopefully one day we'll be able to make references like that auto-link to the correct place, in Bugzilla.

Similarly, using #1234 to refer to an issue in some other Git repo (such as the Rust-GCC/gccrs repo) will auto-link to a completely different issue if that commit is late applied to the main gcc repo (see https://github.com/Rust-GCC/gccrs/issues/3213 for more on that). Using Rust-GCC/gccrs#3213 will avoid the #3213 being turned into an auto-link to an unrelated issue in the current repo.

ChangeLog formatting in pull request bodies

When creating a pull request from a git commit with a commit message in the expected format for GCC commits, the ChangeLog part of the commit msg will be interpreted as a Markdown code block, due to the indentation (in Markdown, indenting text by 4 spaces, or a tab, creates an HTML <pre> block). The indentation will then be lost when rendered as HTML, because it's treated as creating the code block, not as literally part of the commit msg. To avoid this, you can use the other form of <pre> code block to surround the ChangeLog text, i.e. triple backticks before and after the ChangeLog block. This should only be in the Pull Request text though, not the actual Git commit. e.g.

• gcc: Summary of commit
  
  Main body of commit message with detailed explanation.
  
  ```
  gcc/ChangeLog:
  
    * blah.c: Blah di blah.
  ```

Easily applying patches from a pull request

Contemporary forges, including forgejo, support appending .patch or .diff to a pull request or commit URL to get several git am-able patches (or a raw diff for .diff).

$ wget -q https://forge.sourceware.org/gcc/gcc-TEST/pulls/88.patch -O - | git am