Differences between revisions 3 and 4
Revision 3 as of 2020-02-28 19:28:14
Size: 7514
Editor: JeffreyALaw
Comment: Note that Jenkins is not currently mailing testresults to the lists or nag mail when things break
Revision 4 as of 2020-03-07 00:04:32
Size: 6816
Editor: JeffreyALaw
Comment: Various minor updates to Jenkins state
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
The Jenkins master node is running on AWS. Most builders are also running within AWS. The GCC compile farm is providing a few native targets and Red Hat is providing access to an s390x instance for building that platform. Prior to moving the Jenkins master into AWS, it was also using Red Hat OpenShift resources for a variety of builds -- I expect once some technical glitches are resolved we will be able to reconnect those OpenShift resources and use them again. Red Hat has graciously provided OpenShift resources which are used for a handful of the static builders. Amazon has also graciously provided $5000 in AWS credits via their OpenSource AWS credits program. The AWS credits are used for the Jenkins master and additional builders when needed. The GCC compile farm provides a few native targets and Red Hat is also providing access to an s390 slice.
Line 7: Line 7:
All the AWS resources are paid for via Amazon's Open Source AWS credits program. The system was moved into AWS in early February and we're burning through our AWS credits at a rate which should just about exhaust the credits at the end of a year, which is exactly what I budgeted for when applying to the program. As I bring the Red Hat OpenShift resources back online I hope we will have some cost headroom to increase the testing coverage and support developer driven patch testing to some extent. I also expect to reach out to other cloud providers to try and get additional resources for similar purposes.
Line 15: Line 14:

I have tried testing gdb, but the results have been terribly flakey. Clearly this is something that would be good to improve so that we can consistently test the latest GCC & GDB together.
Line 55: Line 56:
I recently tried to turn on GDB testing so that we could test how the most recent GDB and GCC worked together. That turned out to be horribly flakey. This clearly needs to be addressed and re-enabled.

The GCC project has a Jenkins instance for continuous testing of GCC.

At this time the Jenkins instance is read-only. You are welcome to look at build status, download test results, build/test logs, etc. At some point we hope to allow GCC developers to schedule jobs, modify the build/testing scripts, submit patches for testing, etc. In the immediate term, reach out to law@redhat.com if you're looking to contribute to the Jenkins system.

Red Hat has graciously provided OpenShift resources which are used for a handful of the static builders. Amazon has also graciously provided $5000 in AWS credits via their OpenSource AWS credits program. The AWS credits are used for the Jenkins master and additional builders when needed. The GCC compile farm provides a few native targets and Red Hat is also providing access to an s390 slice.

Native Tests

Native testing is accomplished by using ssh to connect to an appropriate builder and running a build script. The build script will update the sources, apply patches from a github repo, then proceed to build binutils/gdb, gcc, glibc and the linux kernel. It will then run the binutils and gcc testsuites. If there are any regressions, the build is marked as failing.

Native testing applies to aarch64-linux-gnu, i686-linux-gnu, powerpc64-linux-gnu, powerpc64le-linux-gnu, s390x-linux-gnu and x86_64-linux-gnu. We have in the past also done native testing on sparc64-linux-gnu, but the builder in the compile farm has an outdated version of Java and no longer works.

We would certainly welcome adding additional targets for native builds. There are mechanisms which would allow individuals to attach unique builder resources to Jenkins and use those for native builds. So for example, FreeBSD, OSX, AIX, etc.

I have tried testing gdb, but the results have been terribly flakey. Clearly this is something that would be good to improve so that we can consistently test the latest GCC & GDB together.

Chroot native testing

For some targets such as 32 bit ARM, we have a mini root filesystem and on appropriate hardware we can chroot into that filesystem and run a native build. For example, we can build arm-linux-gnueabi on aarch64-linux-gnu in a chroot. Similarly for powerpc-linux-gnu on powerpc64-linux-gnu or s390-linux-gnu on s390x-linux-gnu. This requires chroot access on the builder and as a result is currently disabled for powerpc-linux-gnu and s390-linux-gnu.

Otherwise this looks largely like native testing. We may at some point convert i686-linux-gnu to this model as it may be considerably faster than the i686 machine in the compile farm.

Chroot qemu emulated testing

Given a suitable root filesystem and qemu support we can also bootstrap other targets. This is slow, but potentially faster than native hardware and does not require Java to work on the target. These targets have been somewhat more flakey, particularly for glibc and kernel builds which are disabled on some of the targets. These also just run a subset of the GCC testsuite (check-gcc in particular).

We currently use this testing for aarch64_be, alpha, armeb, armeb-hardfloat, hppa, m68k, mipsisa32r2, riscv64, sh4, and sh4eb linux-gnu targets.

Fixing the issues with kernel and glibc builds should be a priority (this was working until a few weeks ago). Expanding testing to include the full testsuite should be considered, but be aware the full GCC testsuite can cause these builds to take several days. Also note debugging failures in this environment with gdb is exceedingly painful.

Cross testing of linux-gnu targets

These are fairly standard cross builds. We start by building a cross binutils and cross compiler. Those are in turn used to build glibc and the kernel. These are not currently running any testsuites, but could. Testing could be particularly interesting for targets with some limited qemu support such as nios or microblaze.

We use this style for csky, microblaze, mips64, mips64el, nios2, s390, sh3 and sh3eb -linux-gnu targets.

Cross testing *-elf with newlib

These are the fairly standard *-elf targets which have newlib support. The scripts will build cross-binutils, cross-gcc, then build newlib with those cross tools. The GCC testsuite is run in one of two modes. Most targets are set up to use a dummy simulator that always returns success. Clearly that's not testing the correctness of code generation, but it has still proven valuable. A relatively small set of targets are testing with the old Cygnus simulators (v850, h8300).

Adding testing with simulators for other targets would be helpful. Additionally most targets do not do any multilib testing. If maintainers reach out with a set of the most important multilibs to test, I'm happy to update the scripts to test those multilibs.

These targets are not building any C++ runtimes or running any C++ tests. That should be fixed.

Cross testing of targets without newlib support

A few targets do not have newlib support. For those we just build the cross tools and build libgcc. This provides a very limited amount of testing that the compiler doesn't immediately ICE and generates at least some code that is accepted by the cross binutils.

Local Patches

I ran this system on my laptop for about 18 months prior to moving it into AWS. During that time it became clear that some provision for patching the source components was needed. This could be to work around temporary bugs in the various components, disable flakey tests, increase stability of the simulator, provide "baseboards" descriptions for embedded targets that didn't have them, etc. The patches are kept at: github:/JeffreyALaw/patches

Chroots

You'll note that some systems use chroots for testing. Those chroots are also kept on github. github:/JeffreyALaw/chroots

Testing Notes

The single most annoying thing that tends to happen is we have a test that used to pass and for various reasons we have to xfail it. The compare_test script will flag those as failures. That in turn means I have to generate a fresh set of baseline test results. It's really annoying. I'll likely further twiddle the compare_test script so that it ignores XFAIL tests.

There are also some tests that are flakey. They'll pass one day and fail the next for no apparent reason. Those tests are currently removed from the source repositories.

The system is not currently sending out mail to gcc-testresults, but can be easily configured to do so. It's on my TODO list :-)

The system is not currently sending notifications of failed builds. This is something that can be easily enabled as well. I did this in early instances of the tester, but didn't when I revamped things about a year ago.

None: Jenkins (last edited 2020-03-07 00:04:32 by JeffreyALaw)