GCC Automated Testing System

The GCC automated continuous testing system is, at present, a single machine which waits for a checkin to the GCC Subversion repository; checks out a URL of that tree; checks that it builds (for the native case, checks that it bootstraps); and runs the testsuite, checking whether any test which used to pass now fails. If any of the checks fails, it determines from the ChangeLog entries which checkins were made and sends mail to their authors.

The tester tests a powerpc-darwin8.5 native.

The following information is available:

  • current status of the tester also available as a continuously updated JavaScript version;
  • logfile containing the results of each test;
  • log output of the last build for native;
  • .log and .sum files from the last time the tree successfully built for native;
  • The list of regressions (tests that used to pass and now fail) from the last time the tree successfully built for native;
  • the last mail the tester sent;
  • list of tests that are known to pass, for native.

    • All the mail from the tester is also sent to the gcc-regression mailing list. The tester also uses the test_summary script to send mail of its successful runs to the gcc-testresults mailing list.

    If idle, the tester automatically tries to update its dejagnu and other tools daily, if the GCC tree is stable:

  • current status;
  • logfile;
  • log output of the last build;

    • Occasionally people ask "Why did I get sent mail? I didn't break anything!" These are the reasons why you might get sent mail by the tester:

  • Your patch actually did break something. Yes, I know your patch couldn't possibly have caused the behaviour shown, but stranger things have happened.
  • Your patch was tested at the same time as another patch which caused the problem. The tester doesn't have the CPU power to test each patch individually, this would need a farm of about eight machines each at least as fast as the one machine doing the testing now. One feature that would be nice, but isn't implemented yet, is to have the tester look more closely at the failure and try to eliminate patches that couldn't possibly have caused it (for instance, a change to the mips backend can't cause a regression test failure on x86 native).
  • The tester sent you mail to warn you that it couldn't fully test your patch, because the tree didn't build. This is not supposed to happen very often; when I was designing the tester, I didn't expect the tree to be unbuildable for weeks or months at a time. I believe the best way to reduce this mail is to try to ensure that the tree doesn't get broken, not by changing the tester. I'd rather not switch this off, because that would only make the underlying problem worse.
  • The tester sent you mail to warn you that it couldn't fully test your patch, because there were pre-existing regressions, and it's possible that the regressions were hiding a bug in your patch (for instance, if your patch is trying to do something with exception handling, and all the exception handling tests were failing before your patch went in, then your patch might have a bug but there's no way to tell). It'll only send you one of these until the list of regressions changes. This isn't supposed to happen often either.
    •
    Geoffrey Keating geoffk@apple.com