Hi, We have noticed what appear to be random results in go and libgo tests on arm and aarch64. Since we don't plan to actually work on go, I post here the list of such tests to share information in case others have noticed similar things. On arm (AArch32): libgo:database/sql libgo:os libgo:os/signal libgo:context On AArch64: libgo:os/signal libgo:runtime/pprof libgo:sync We have also observed similar failures on x86_32 and x86_64, but much less frequently. These tests are running natively.
What do you mean by "random results?" Can you post some output? Thanks.
Well, we have a nightly Jenkins job that builds GCC (from scratch) and runs 'make check' on ARM, AArch64, and x86 hosts, and from time to time we see regression reports on the mentioned tests. And the next run generally says those tests now pass. For instance: Test Run By tcwg-buildslave on Sat Feb 3 08:20:17 UTC 2018 Native configuration is armv8l-unknown-linux-gnueabihf === libgo tests === [....] PASS: os/exec --- FAIL: TestAtomicStop (5.61s) testing.go:560: iteration 0: exit status "signal: Interrupt": output: testing.go:560: iteration 1: exit status "signal: Interrupt": output: testing.go:560: iteration 3: exit status "signal: Interrupt": output: testing.go:560: iteration 5: exit status "signal: Interrupt": output: testing.go:560: iteration 7: exit status "signal: Interrupt": output: lost signal on tries: 2 testing.go:560: iteration 7: lost signal testing.go:560: iteration 9: exit status "signal: Interrupt": output: FAIL FAIL: os/signal PASS PASS: os/user For libgo:context, there might be a real compiler bug though: go1: internal compiler error: Segmentation fault 0x5ae6ab crash_signal /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/toplev.c:325 0x1ea8e9 Block::determine_types() /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6062 0x1ea90d Block::determine_types() /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6071 0x1ea90d Block::determine_types() /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6071 0x1ea90d Block::determine_types() /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6071 0x1ea90d Block::determine_types() /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6071 0x1ea90d Block::determine_types() /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6071 0x1ecd41 Gogo::determine_types() /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:5148 0x1e81ad go_parse_input_files(char const**, unsigned int, bool, bool) /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/go.cc:133 0x1e5115 go_langhook_parse_file /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/go-lang.c:329 Please submit a full bug report, with preprocessed source if appropriate. Please include the complete backtrace with any bug report. See <https://gcc.gnu.org/bugs/> for instructions. FAIL: context but the next build is still running, I don't know yet if it passes.
> FAIL: context > but the next build is still running, I don't know yet if it passes. Next build completed, and: PASS: context Did something change wrt this test since yesterday?
FWIW, I have seen these on i686 and x86-64. They are purely random.
Today on ARM: --- FAIL: TestExtraFiles (1.19s) testing.go:560: TestExtraFiles: Something already leaked - closed fd 63 testing.go:560: Run: exit status 1; stdout "leaked parent file. fd = 58; want 57\n", stderr "" FAIL FAIL: os/exec
Author: ian Date: Mon Feb 12 18:50:16 2018 New Revision: 257599 URL: https://gcc.gnu.org/viewcvs?rev=257599&root=gcc&view=rev Log: PR go/84215 runtime, sync/atomic: use write barrier for atomic pointer functions This copies atomic_pointer.go from 1.10rc2. It was omitted during the transition of the runtime from C to Go, and I forgot about it. This may help with https://gcc.gnu.org/PR84215. Reviewed-on: https://go-review.googlesource.com/93197 Added: trunk/libgo/go/runtime/atomic_pointer.go Modified: trunk/gcc/go/gofrontend/MERGE trunk/libgo/go/runtime/stubs.go trunk/libgo/go/sync/atomic/atomic.c
I just committed a patch that may affect some of these failures. Please let me know if you notice any change. Thanks.
Still seeing at least os/signal having random pass/fail.
On i686 with many cores, r257758 gave: ... goroutine 24690 [GC worker (idle)]: runtime.mcall /export/gnu/import/git/sources/gcc/libgo/runtime/proc.c:342 runtime.gopark /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/proc.go:321 runtime.gcBgMarkWorker /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/mgc.go:1776 runtime.kickoff /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/proc.go:1161 created by runtime.gcBgMarkStartWorkers /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/mgc.go:1722 +92 goroutine 23558 [running]: goroutine running on other thread; stack unavailable created by testing.T.Run /export/gnu/import/git/sources/gcc/libgo/go/testing/testing.go:824 +740 goroutine 24113 [chan send]: runtime.mcall /export/gnu/import/git/sources/gcc/libgo/runtime/proc.c:342 runtime.gopark /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/proc.go:321 runtime.goparkunlock /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/proc.go:327 runtime.chansend /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/chan.go:249 runtime.chansend1 /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/chan.go:135 testing.tRunner..func1 /export/gnu/import/git/sources/gcc/libgo/go/testing/testing.go:728 runtime.deferreturn /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/panic.go:205 testing.tRunner /export/gnu/import/git/sources/gcc/libgo/go/testing/testing.go:783 runtime.kickoff /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/proc.go:1161 created by testing.T.Run /export/gnu/import/git/sources/gcc/libgo/go/testing/testing.go:824 +740 eax 0x0 ebx 0x2 ecx 0xb93e1afc edx 0x0 edi 0x0 esi 0x8 ebp 0xb93e1afc esp 0xb93e1ae0 eip 0xf7f42db9 eflags 0x282 cs 0x23 fs 0x0 gs 0x63 FAIL: runtime goroutine 4122 [IO wait]: internal_poll.runtime_pollWait /export/gnu/import/git/sources/gcc/libgo/go/runtime/netpoll.go:176 internal_poll.pollDesc.wait /export/gnu/import/git/sources/gcc/libgo/go/internal/poll/fd_poll_runtime.go:85 internal_poll.pollDesc.waitRead /export/gnu/import/git/sources/gcc/libgo/go/internal/poll/fd_poll_runtime.go:90 internal_poll.FD.Accept /export/gnu/import/git/sources/gcc/libgo/go/internal/poll/fd_unix.go:372 net.netFD.accept /export/gnu/import/git/sources/gcc/libgo/go/net/fd_unix.go:238 net.TCPListener.accept /export/gnu/import/git/sources/gcc/libgo/go/net/tcpsock_posix.go:136 net.TCPListener.Accept /export/gnu/import/git/sources/gcc/libgo/go/net/tcpsock.go:259 net_http.Server.Serve /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest249618/test/server.go:2770 httptest.func1 /export/gnu/import/git/sources/gcc/libgo/go/net/http/httptest/server.go:280 created by net_http_httptest.Server.goServe /export/gnu/import/git/sources/gcc/libgo/go/net/http/httptest/server.go:278 +198 goroutine 4535 [semacquire]: syscall.Accept4 /export/gnu/import/git/sources/gcc/libgo/go/syscall/libcall_linux.go:176 poll.accept /export/gnu/import/git/sources/gcc/libgo/go/internal/poll/sock_cloexec.go:17 internal_poll.FD.Accept /export/gnu/import/git/sources/gcc/libgo/go/internal/poll/fd_unix.go:365 net.netFD.accept /export/gnu/import/git/sources/gcc/libgo/go/net/fd_unix.go:238 net.TCPListener.accept /export/gnu/import/git/sources/gcc/libgo/go/net/tcpsock_posix.go:136 net.TCPListener.Accept /export/gnu/import/git/sources/gcc/libgo/go/net/tcpsock.go:259 crypto_tls.listener.Accept /export/gnu/import/git/sources/gcc/libgo/go/crypto/tls/tls.go:52 net_http.Server.Serve /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest249618/test/server.go:2770 httptest.func1 /export/gnu/import/git/sources/gcc/libgo/go/net/http/httptest/server.go:280 created by net_http_httptest.Server.goServe /export/gnu/import/git/sources/gcc/libgo/go/net/http/httptest/server.go:278 +198 eax 0x0 ebx 0x2 ecx 0xce0f1fbc edx 0x0 edi 0x0 esi 0x8 ebp 0xce0f1fbc esp 0xce0f1fa0 eip 0xf7f3edb9 eflags 0x286 cs 0x23 fs 0x0 gs 0x63 FAIL: net/http /tmp/cc2A9bsi.s: Assembler messages: /tmp/cc2A9bsi.s:7954: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:7965: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:7976: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:7977: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:7978: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:7979: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:7993: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8004: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8015: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8016: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8017: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8018: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8032: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8043: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8054: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8055: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8056: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8057: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8071: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8082: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8093: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8094: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8095: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8096: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8110: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8121: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8132: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8133: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8134: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8135: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8149: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8160: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8171: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8172: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8173: Error: leb128 operand is an undefined symbol: .LVU443 /tmp/cc2A9bsi.s:8174: Error: leb128 operand is an undefined symbol: .LVU443 FAIL: golang_org/x/crypto/chacha20poly1305/internal/chacha20 FAIL: golang_org/x/crypto/curve25519 FAIL: golang_org/x/net/lex/httplex checksyms: found unexpected symbol "__libc_start_main@@GLIBC_2.0" FAIL: checksyms spawn [open ...]^M fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=2 addr=4140130304 pc=4152026464] runtime stack: doscanstack1 /export/gnu/import/git/sources/gcc/libgo/runtime/stack.c:77 runtime_mstart /export/gnu/import/git/sources/gcc/libgo/runtime/proc.c:527 goroutine 33 [GC worker (idle)]: created by runtime.gcStart /export/gnu/import/git/sources/gcc/libgo/go/runtime/mgc.go:1299 +412 goroutine 1 [GC assist marking (scan)]: panic /export/gnu/import/git/sources/gcc/libgo/go/runtime/panic.go:462 runtime_panicstring /export/gnu/import/git/sources/gcc/libgo/runtime/panic.c:38 main.main..func1 ./index0-out.go:140 main.test ./index0-out.go:112 main.main ./index0-out.go:140 FAIL: ./index0-out.go execution, -O0 -g -fno-var-tracking-assignments
Yesterday I noticed: * on arm: FAIL: libgo:database/sql * on aarch64: FAIL: libgo:runtime/pprof
Please include all the output from each failure, if possible. Thanks.
It's not very informative. On arm: PASS: crypto/x509 timed out in gotest /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/libgo/testsuite/gotest: line 624: 15642 Killed ./a.out -test.short -test.timeout=${timeout}s "$@" FAIL: database/sql PASS PASS: database/sql/driver On aarch64: PASS: runtime/internal/sys /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_64-build/snapshots/gcc.git~master/libgo/testsuite/gotest: line 624: 28388 Segmentation fault (core dumped) ./a.out -test.short -test.timeout=${timeout}s "$@" FAIL: runtime/pprof PASS PASS: runtime/pprof/internal/profile I included the PASS lines only to show the actual context around the FAIL lines.