seen with trunk 20240101, building the golang-1.21 package with -flto (with testsuite running during the build), many architectures: ok cmd/internal/src 0.113s ok cmd/internal/test2json 0.422s --- FAIL: TestCGOLTO (0.25s) cgo_test.go:32: /home/packages/tmp/golang-1.21-1.21.5/bin/go env CC: "gcc" cgo_test.go:32: /home/packages/tmp/golang-1.21-1.21.5/bin/go env CGO_CFLAGS: "-O2 -g" --- FAIL: TestCGOLTO/0 (9.48s) cgo_test.go:123: CGO_CFLAGS="-O2 -g -flto" /home/packages/tmp/golang-1.21-1.21.5/bin/go build cgo_test.go:125: # cgolto /home/packages/tmp/golang-1.21-1.21.5/pkg/tool/linux_amd64/link: running gcc failed: exit status 1 lto1: error: qsort comparator non-negative on sorted output: 64 during IPA pass: static-var lto1: internal compiler error: qsort checking failed 0x18e4cee qsort_chk_error ../../src/gcc/vec.cc:204 0x6518ea qsort_chk(void*, unsigned long, unsigned long, int (*)(void const*, void co nst*, void*), void*) ../../src/gcc/vec.cc:246 0x1c50705 gcc_sort_r(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) ../../src/gcc/sort.cc:296 0x203e8c1 vec<symtab_node*, va_heap, vl_embed>::sort(int (*)(void const*, void const*, void*), void*) ../../src/gcc/vec.h:1232 0x203e8c1 vec<symtab_node*, va_heap, vl_ptr>::sort(int (*)(void const*, void const*, void*), void*) ../../src/gcc/vec.h:2252 0x203e8c1 lto_output() ../../src/gcc/lto-streamer-out.cc:2799 0x203e453 write_lto ../../src/gcc/passes.cc:2779 0x8b2643 ipa_write_optimization_summaries(lto_symtab_encoder_d*) ../../src/gcc/passes.cc:2969 0x2039557 stream_out ../../src/gcc/lto/lto.cc:177 0x2039557 stream_out_partitions_1 ../../src/gcc/lto/lto.cc:229 0x1fcd178 stream_out_partitions ../../src/gcc/lto/lto.cc:244 0x1fcd178 lto_wpa_write_files ../../src/gcc/lto/lto.cc:417 0x1fcd178 do_whole_program_analysis ../../src/gcc/lto/lto.cc:592 0x1fcd178 lto_main() ../../src/gcc/lto/lto.cc:666 Please submit a full bug report, with preprocessed source (by using -freport-bug). Please include the complete backtrace with any bug report. See <file:///usr/share/doc/gcc-14/README.Bugs> for instructions. lto-wrapper: fatal error: gcc returned 1 exit status compilation terminated. /usr/bin/ld: error: lto-wrapper failed collect2: error: ld returned 1 exit status cgo_test.go:128: go build failed: exit status 1 cgo_test.go:146: failed FAIL FAIL cmd/link 10.473s ok cmd/link/internal/benchmark 0.020s
The comparison function in question: ``` static int cmp_symbol_files (const void *pn1, const void *pn2, void *id_map_) { const symtab_node *n1 = *(const symtab_node * const *)pn1; const symtab_node *n2 = *(const symtab_node * const *)pn2; hash_map<lto_file_decl_data *, int> *id_map = (hash_map<lto_file_decl_data *, int> *)id_map_; int file_order1 = n1->lto_file_data ? n1->lto_file_data->order : -1; int file_order2 = n2->lto_file_data ? n2->lto_file_data->order : -1; /* Order files same way as they appeared in the command line to reduce seeking while copying sections. */ if (file_order1 != file_order2) return file_order1 - file_order2; /* Order within static library. */ if (n1->lto_file_data && n1->lto_file_data->id != n2->lto_file_data->id) return *id_map->get (n1->lto_file_data) - *id_map->get (n2->lto_file_data); /* And finaly order by the definition order. */ return n1->order - n2->order; } ``` I wonder if the map changes ...
I hit this a little while ago (https://bugs.gentoo.org/912152) but I gave up after I couldn't get Go to preserve the files I needed.
13 is affected too at least, I'm pretty sure, but I can't check right now. Just needs checking.
it's really odd, I don't see how it could fail. The only suspicious bit is int file_order1 = n1->lto_file_data ? n1->lto_file_data->order : -1; int file_order2 = n2->lto_file_data ? n2->lto_file_data->order : -1; /* Order files same way as they appeared in the command line to reduce seeking while copying sections. */ if (file_order1 != file_order2) return file_order1 - file_order2; the non-presence of n{1,2}->lto_file_data represented as -1 makes whether non-presence is first dependent on the value of the order of the other. Maybe explicitly spelling that out would be better. if (file_order1 != file_order2) { if (file_order1 == -1) return -1; if (file_order2 == -1) return 1; return file_order1 - file_order2; } or so.
I've hit a similar ICE testing libbacktrace with LTO bootstrapped GCC on LoongArch: during IPA pass: static-var lto1: internal compiler error: qsort checking failed 0x7ffff1b13047 __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 0x7ffff1b1316f __libc_start_main_impl /sources/glibc-2.39/csu/libc-start.c:360 0x12019a547 _start ../sysdeps/loongarch/start.S:75 Please submit a full bug report, with preprocessed source (by using -freport-bug). Please include the complete backtrace with any bug report. See <https://gcc.gnu.org/bugs/> for instructions. lto-wrapper: fatal error: /sources/gcc-14-8722+hotfix/build/./prev-gcc/xgcc returned 1 exit status compilation terminated. /usr/bin/ld: error: lto-wrapper failed collect2: error: ld returned 1 exit status make[3]: *** [Makefile:1340: btest_lto] Error 1 Unfortunately I didn't enable the debug info. I'll keep my build tree and try to gather some info when I get some time...
I managed to run the test manually in `/usr/lib/go/src/cmd/link` with `go test cgo_test.go -x -v` and made writeTempFile dump the name/contents. At least now I can reproduce with just one command outside of the test suite: ``` /tmp/go-test $ cat add.go package main import "C" /* test */ //export myadd func myadd(a C.int, b C.int) C.int { return a + b } ``` ``` /tmp/go-test $ cat main.go package main import "fmt" /* #include <stdio.h> void hello(void) { printf("hello\n"); } */ import "C" func main() { hello := C.hello fmt.Printf("%v\n", hello) } ``` ``` /tmp/go-test $ cat go.mod module cgolto ``` Then I get: ``` /tmp/go-test $ CGO_CFLAGS="-O2 -g -flto" /usr/lib/go/bin/go build # cgolto /usr/lib/go/pkg/tool/linux_amd64/link: running x86_64-pc-linux-gnu-gcc failed: exit status 1 lto1: error: qsort comparator non-negative on sorted output: 64 during IPA pass: static-var lto1: internal compiler error: qsort checking failed 0x63b5c1913cd6 qsort_chk_error /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.cc:204 0x63b5c0397d4d qsort_chk(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.cc:246 0x63b5c1d34457 gcc_sort_r(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/sort.cc:296 0x63b5c1b289f4 vec<symtab_node*, va_heap, vl_embed>::sort(int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.h:1232 0x63b5c1b289f4 vec<symtab_node*, va_heap, vl_ptr>::sort(int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.h:2252 0x63b5c1b289f4 lto_output() /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto-streamer-out.cc:2799 0x63b5c21e6365 write_lto /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/passes.cc:2779 0x63b5c21e2c9e ipa_write_optimization_summaries(lto_symtab_encoder_d*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/passes.cc:2969 0x63b5c21e1f94 stream_out /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:179 0x63b5c21e1f94 stream_out_partitions_1 /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:231 0x63b5c21abd7a stream_out_partitions /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:246 0x63b5c21abd7a lto_wpa_write_files /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:419 0x63b5c21abd7a do_whole_program_analysis /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:594 0x63b5c21abd7a lto_main() /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:668 Please submit a full bug report, with preprocessed source (by using -freport-bug). Please include the complete backtrace with any bug report. See <https://bugs.gentoo.org/> for instructions. lto-wrapper: fatal error: x86_64-pc-linux-gnu-gcc returned 1 exit status compilation terminated. /usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: error: lto-wrapper failed collect2: error: ld returned 1 exit status ```
With some finagling: ``` /tmp/go-PR113204 $ ./run.sh + gcc -o test -Wl,--export-dynamic-symbol=_cgo_panic -Wl,--export-dynamic-symbol=_cgo_topofstack -Wl,--export-dynamic-symbol=crosscall2 -Wl,--export-dynamic-symbol=myadd -Wl,--compress-debug-sections=zlib go.o 000000.o 000001.o 000002.o 000003.o 000004.o 000005.o 000006.o 000007.o 000008.o 000009.o 000010.o 000011.o 000012.o 000013.o 000014.o 000015.o 000016.o -O2 -g -O2 -g -lpthread lto1: error: qsort comparator non-negative on sorted output: 64 during IPA pass: static-var lto1: internal compiler error: qsort checking failed 0x56bcc8c9bcd6 qsort_chk_error /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.cc:204 0x56bcc771fd4d qsort_chk(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.cc:246 0x56bcc90bc457 gcc_sort_r(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/sort.cc:296 0x56bcc8eb09f4 vec<symtab_node*, va_heap, vl_embed>::sort(int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.h:1232 0x56bcc8eb09f4 vec<symtab_node*, va_heap, vl_ptr>::sort(int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.h:2252 0x56bcc8eb09f4 lto_output() /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto-streamer-out.cc:2799 0x56bcc956e365 write_lto /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/passes.cc:2779 0x56bcc956ac9e ipa_write_optimization_summaries(lto_symtab_encoder_d*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/passes.cc:2969 0x56bcc9569f94 stream_out /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:179 0x56bcc9569f94 stream_out_partitions_1 /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:231 0x56bcc9533d7a stream_out_partitions /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:246 0x56bcc9533d7a lto_wpa_write_files /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:419 0x56bcc9533d7a do_whole_program_analysis /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:594 0x56bcc9533d7a lto_main() /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:668 Please submit a full bug report, with preprocessed source (by using -freport-bug). Please include the complete backtrace with any bug report. See <https://bugs.gentoo.org/> for instructions. lto-wrapper: fatal error: gcc returned 1 exit status compilation terminated. /usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: error: lto-wrapper failed collect2: error: ld returned 1 exit status ``` Tarball with just object files which reproduces it for me: https://dev.gentoo.org/~sam/bugs/gcc/gcc-go-qsort/go-PR113204.tar.xz
(For my own benefit for future reference: I ran go build -work -v -x, then went into the work dir it made, then ran /usr/lib/go/pkg/tool/linux_amd64/link -v ..., then started pulling out the bits it ran manually.)
(In reply to Xi Ruoyao from comment #5) > I've hit a similar ICE testing libbacktrace with LTO bootstrapped GCC on > LoongArch: I hit this today too. Unfortunately, it seems that libbacktrace gets relinked (and maybe sometimes rebuilt?) repeatedly and I couldn't convert all the files to preprocessed variants in time before the build finished, and I was too lazy to properly suspend it. But I can reproduce it with: ``` $ gcc btest-btest.i btest_lto-testlib.i fileline.i dwarf.o elf.o -O2 -flto /var/tmp/portage/sys-devel/gcc-15.0.9999/work/gcc-15.0.9999/libbacktrace/btest.c:52:1: warning: ‘optnone’ attribute directive ignored [-Wattributes] /var/tmp/portage/sys-devel/gcc-15.0.9999/work/gcc-15.0.9999/libbacktrace/btest.c:166:1: warning: ‘optnone’ attribute directive ignored [-Wattributes] lto-wrapper: warning: Extra option to ‘-Xassembler’: -O2, dropping all ‘-Xassembler’ and ‘-Wa’ options. lto1: error: qsort comparator non-negative on sorted output: 204 during IPA pass: static-var lto1: internal compiler error: qsort checking failed 0x558d86f3778f internal_error(char const*, ...) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/diagnostic-global-context.cc:491 0x558d86f3784f qsort_chk_error /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/vec.cc:204 0x558d858bc2f0 qsort_chk(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/vec.cc:246 0x558d873b50f2 gcc_sort_r(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/sort.cc:296 0x558d871d971b vec<symtab_node*, va_heap, vl_embed>::sort(int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/vec.h:1232 0x558d871d971b vec<symtab_node*, va_heap, vl_ptr>::sort(int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/vec.h:2252 0x558d871d971b lto_output() /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto-streamer-out.cc:2792 0x558d878d6805 write_lto /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/passes.cc:2780 0x558d878cf8f7 ipa_write_optimization_summaries(lto_symtab_encoder_d*) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/passes.cc:2970 0x558d878cf1e4 stream_out /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:179 0x558d878cf1e4 stream_out_partitions_1 /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:231 0x558d8788ed99 stream_out_partitions /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:290 0x558d8788ed99 lto_wpa_write_files /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:419 0x558d8788ed99 do_whole_program_analysis /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:596 0x558d8788ed99 lto_main() /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:670 Please submit a full bug report, with preprocessed source (by using -freport-bug). Please include the complete backtrace with any bug report. See <https://bugs.gentoo.org/> for instructions. ``` and at least dwarf.o and elf.o are from C this time so Go is out of the equation. This keeps irritating me so at some point I'll try playing with it like richi suggested I guess.
(My reluctance is because getting a proper set of C sources to reproduce it seems to keep hiding from me.)
(In reply to Richard Biener from comment #4) > the non-presence of n{1,2}->lto_file_data represented as -1 makes whether > non-presence is first dependent on the value of the order of the other. > That might explain why it's such a pain to try isolate it, actually.
*** Bug 118788 has been marked as a duplicate of this bug. ***