| Summary: | qsort checking ICE with Go LTO (lto1: error: qsort comparator non-negative on sorted output: 64) | ||
|---|---|---|---|
| Product: | gcc | Reporter: | Matthias Klose <doko> |
| Component: | lto | Assignee: | Not yet assigned to anyone <unassigned> |
| Status: | NEW --- | ||
| Severity: | normal | CC: | egallager, ian, marxin, sjames, xry111 |
| Priority: | P3 | Keywords: | ice-checking, ice-on-valid-code, lto, testsuite-fail |
| Version: | 14.0 | ||
| Target Milestone: | --- | ||
| See Also: | https://bugs.gentoo.org/show_bug.cgi?id=912152 | ||
| Host: | Target: | ||
| Build: | Known to work: | 13.2.1 | |
| Known to fail: | 14.0 | Last reconfirmed: | 2024-08-02 00:00:00 |
| Bug Depends on: | |||
| Bug Blocks: | 82407 | ||
The comparison function in question:
```
static int
cmp_symbol_files (const void *pn1, const void *pn2, void *id_map_)
{
const symtab_node *n1 = *(const symtab_node * const *)pn1;
const symtab_node *n2 = *(const symtab_node * const *)pn2;
hash_map<lto_file_decl_data *, int> *id_map
= (hash_map<lto_file_decl_data *, int> *)id_map_;
int file_order1 = n1->lto_file_data ? n1->lto_file_data->order : -1;
int file_order2 = n2->lto_file_data ? n2->lto_file_data->order : -1;
/* Order files same way as they appeared in the command line to reduce
seeking while copying sections. */
if (file_order1 != file_order2)
return file_order1 - file_order2;
/* Order within static library. */
if (n1->lto_file_data && n1->lto_file_data->id != n2->lto_file_data->id)
return *id_map->get (n1->lto_file_data) - *id_map->get (n2->lto_file_data);
/* And finaly order by the definition order. */
return n1->order - n2->order;
}
```
I wonder if the map changes ...
I hit this a little while ago (https://bugs.gentoo.org/912152) but I gave up after I couldn't get Go to preserve the files I needed. 13 is affected too at least, I'm pretty sure, but I can't check right now. Just needs checking. it's really odd, I don't see how it could fail. The only suspicious bit is
int file_order1 = n1->lto_file_data ? n1->lto_file_data->order : -1;
int file_order2 = n2->lto_file_data ? n2->lto_file_data->order : -1;
/* Order files same way as they appeared in the command line to reduce
seeking while copying sections. */
if (file_order1 != file_order2)
return file_order1 - file_order2;
the non-presence of n{1,2}->lto_file_data represented as -1 makes whether
non-presence is first dependent on the value of the order of the other.
Maybe explicitly spelling that out would be better.
if (file_order1 != file_order2)
{
if (file_order1 == -1)
return -1;
if (file_order2 == -1)
return 1;
return file_order1 - file_order2;
}
or so.
I've hit a similar ICE testing libbacktrace with LTO bootstrapped GCC on LoongArch:
during IPA pass: static-var
lto1: internal compiler error: qsort checking failed
0x7ffff1b13047 __libc_start_call_main
../sysdeps/nptl/libc_start_call_main.h:58
0x7ffff1b1316f __libc_start_main_impl
/sources/glibc-2.39/csu/libc-start.c:360
0x12019a547 _start
../sysdeps/loongarch/start.S:75
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.
lto-wrapper: fatal error: /sources/gcc-14-8722+hotfix/build/./prev-gcc/xgcc returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:1340: btest_lto] Error 1
Unfortunately I didn't enable the debug info. I'll keep my build tree and try to gather some info when I get some time...
I managed to run the test manually in `/usr/lib/go/src/cmd/link` with `go test cgo_test.go -x -v` and made writeTempFile dump the name/contents.
At least now I can reproduce with just one command outside of the test suite:
```
/tmp/go-test $ cat add.go
package main
import "C"
/* test */
//export myadd
func myadd(a C.int, b C.int) C.int {
return a + b
}
```
```
/tmp/go-test $ cat main.go
package main
import "fmt"
/*
#include <stdio.h>
void hello(void) {
printf("hello\n");
}
*/
import "C"
func main() {
hello := C.hello
fmt.Printf("%v\n", hello)
}
```
```
/tmp/go-test $ cat go.mod
module cgolto
```
Then I get:
```
/tmp/go-test $ CGO_CFLAGS="-O2 -g -flto" /usr/lib/go/bin/go build
# cgolto
/usr/lib/go/pkg/tool/linux_amd64/link: running x86_64-pc-linux-gnu-gcc failed: exit status 1
lto1: error: qsort comparator non-negative on sorted output: 64
during IPA pass: static-var
lto1: internal compiler error: qsort checking failed
0x63b5c1913cd6 qsort_chk_error
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.cc:204
0x63b5c0397d4d qsort_chk(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*)
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.cc:246
0x63b5c1d34457 gcc_sort_r(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*)
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/sort.cc:296
0x63b5c1b289f4 vec<symtab_node*, va_heap, vl_embed>::sort(int (*)(void const*, void const*, void*), void*)
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.h:1232
0x63b5c1b289f4 vec<symtab_node*, va_heap, vl_ptr>::sort(int (*)(void const*, void const*, void*), void*)
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.h:2252
0x63b5c1b289f4 lto_output()
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto-streamer-out.cc:2799
0x63b5c21e6365 write_lto
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/passes.cc:2779
0x63b5c21e2c9e ipa_write_optimization_summaries(lto_symtab_encoder_d*)
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/passes.cc:2969
0x63b5c21e1f94 stream_out
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:179
0x63b5c21e1f94 stream_out_partitions_1
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:231
0x63b5c21abd7a stream_out_partitions
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:246
0x63b5c21abd7a lto_wpa_write_files
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:419
0x63b5c21abd7a do_whole_program_analysis
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:594
0x63b5c21abd7a lto_main()
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:668
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
lto-wrapper: fatal error: x86_64-pc-linux-gnu-gcc returned 1 exit status
compilation terminated.
/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
```
With some finagling: ``` /tmp/go-PR113204 $ ./run.sh + gcc -o test -Wl,--export-dynamic-symbol=_cgo_panic -Wl,--export-dynamic-symbol=_cgo_topofstack -Wl,--export-dynamic-symbol=crosscall2 -Wl,--export-dynamic-symbol=myadd -Wl,--compress-debug-sections=zlib go.o 000000.o 000001.o 000002.o 000003.o 000004.o 000005.o 000006.o 000007.o 000008.o 000009.o 000010.o 000011.o 000012.o 000013.o 000014.o 000015.o 000016.o -O2 -g -O2 -g -lpthread lto1: error: qsort comparator non-negative on sorted output: 64 during IPA pass: static-var lto1: internal compiler error: qsort checking failed 0x56bcc8c9bcd6 qsort_chk_error /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.cc:204 0x56bcc771fd4d qsort_chk(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.cc:246 0x56bcc90bc457 gcc_sort_r(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/sort.cc:296 0x56bcc8eb09f4 vec<symtab_node*, va_heap, vl_embed>::sort(int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.h:1232 0x56bcc8eb09f4 vec<symtab_node*, va_heap, vl_ptr>::sort(int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/vec.h:2252 0x56bcc8eb09f4 lto_output() /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto-streamer-out.cc:2799 0x56bcc956e365 write_lto /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/passes.cc:2779 0x56bcc956ac9e ipa_write_optimization_summaries(lto_symtab_encoder_d*) /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/passes.cc:2969 0x56bcc9569f94 stream_out /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:179 0x56bcc9569f94 stream_out_partitions_1 /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:231 0x56bcc9533d7a stream_out_partitions /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:246 0x56bcc9533d7a lto_wpa_write_files /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:419 0x56bcc9533d7a do_whole_program_analysis /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:594 0x56bcc9533d7a lto_main() /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/lto/lto.cc:668 Please submit a full bug report, with preprocessed source (by using -freport-bug). Please include the complete backtrace with any bug report. See <https://bugs.gentoo.org/> for instructions. lto-wrapper: fatal error: gcc returned 1 exit status compilation terminated. /usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: error: lto-wrapper failed collect2: error: ld returned 1 exit status ``` Tarball with just object files which reproduces it for me: https://dev.gentoo.org/~sam/bugs/gcc/gcc-go-qsort/go-PR113204.tar.xz (For my own benefit for future reference: I ran go build -work -v -x, then went into the work dir it made, then ran /usr/lib/go/pkg/tool/linux_amd64/link -v ..., then started pulling out the bits it ran manually.) (In reply to Xi Ruoyao from comment #5) > I've hit a similar ICE testing libbacktrace with LTO bootstrapped GCC on > LoongArch: I hit this today too. Unfortunately, it seems that libbacktrace gets relinked (and maybe sometimes rebuilt?) repeatedly and I couldn't convert all the files to preprocessed variants in time before the build finished, and I was too lazy to properly suspend it. But I can reproduce it with: ``` $ gcc btest-btest.i btest_lto-testlib.i fileline.i dwarf.o elf.o -O2 -flto /var/tmp/portage/sys-devel/gcc-15.0.9999/work/gcc-15.0.9999/libbacktrace/btest.c:52:1: warning: ‘optnone’ attribute directive ignored [-Wattributes] /var/tmp/portage/sys-devel/gcc-15.0.9999/work/gcc-15.0.9999/libbacktrace/btest.c:166:1: warning: ‘optnone’ attribute directive ignored [-Wattributes] lto-wrapper: warning: Extra option to ‘-Xassembler’: -O2, dropping all ‘-Xassembler’ and ‘-Wa’ options. lto1: error: qsort comparator non-negative on sorted output: 204 during IPA pass: static-var lto1: internal compiler error: qsort checking failed 0x558d86f3778f internal_error(char const*, ...) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/diagnostic-global-context.cc:491 0x558d86f3784f qsort_chk_error /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/vec.cc:204 0x558d858bc2f0 qsort_chk(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/vec.cc:246 0x558d873b50f2 gcc_sort_r(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/sort.cc:296 0x558d871d971b vec<symtab_node*, va_heap, vl_embed>::sort(int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/vec.h:1232 0x558d871d971b vec<symtab_node*, va_heap, vl_ptr>::sort(int (*)(void const*, void const*, void*), void*) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/vec.h:2252 0x558d871d971b lto_output() /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto-streamer-out.cc:2792 0x558d878d6805 write_lto /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/passes.cc:2780 0x558d878cf8f7 ipa_write_optimization_summaries(lto_symtab_encoder_d*) /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/passes.cc:2970 0x558d878cf1e4 stream_out /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:179 0x558d878cf1e4 stream_out_partitions_1 /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:231 0x558d8788ed99 stream_out_partitions /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:290 0x558d8788ed99 lto_wpa_write_files /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:419 0x558d8788ed99 do_whole_program_analysis /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:596 0x558d8788ed99 lto_main() /usr/src/debug/sys-devel/gcc-15.0.9999/gcc-15.0.9999/gcc/lto/lto.cc:670 Please submit a full bug report, with preprocessed source (by using -freport-bug). Please include the complete backtrace with any bug report. See <https://bugs.gentoo.org/> for instructions. ``` and at least dwarf.o and elf.o are from C this time so Go is out of the equation. This keeps irritating me so at some point I'll try playing with it like richi suggested I guess. (My reluctance is because getting a proper set of C sources to reproduce it seems to keep hiding from me.) (In reply to Richard Biener from comment #4) > the non-presence of n{1,2}->lto_file_data represented as -1 makes whether > non-presence is first dependent on the value of the order of the other. > That might explain why it's such a pain to try isolate it, actually. *** Bug 118788 has been marked as a duplicate of this bug. *** *** Bug 124125 has been marked as a duplicate of this bug. *** |
seen with trunk 20240101, building the golang-1.21 package with -flto (with testsuite running during the build), many architectures: ok cmd/internal/src 0.113s ok cmd/internal/test2json 0.422s --- FAIL: TestCGOLTO (0.25s) cgo_test.go:32: /home/packages/tmp/golang-1.21-1.21.5/bin/go env CC: "gcc" cgo_test.go:32: /home/packages/tmp/golang-1.21-1.21.5/bin/go env CGO_CFLAGS: "-O2 -g" --- FAIL: TestCGOLTO/0 (9.48s) cgo_test.go:123: CGO_CFLAGS="-O2 -g -flto" /home/packages/tmp/golang-1.21-1.21.5/bin/go build cgo_test.go:125: # cgolto /home/packages/tmp/golang-1.21-1.21.5/pkg/tool/linux_amd64/link: running gcc failed: exit status 1 lto1: error: qsort comparator non-negative on sorted output: 64 during IPA pass: static-var lto1: internal compiler error: qsort checking failed 0x18e4cee qsort_chk_error ../../src/gcc/vec.cc:204 0x6518ea qsort_chk(void*, unsigned long, unsigned long, int (*)(void const*, void co nst*, void*), void*) ../../src/gcc/vec.cc:246 0x1c50705 gcc_sort_r(void*, unsigned long, unsigned long, int (*)(void const*, void const*, void*), void*) ../../src/gcc/sort.cc:296 0x203e8c1 vec<symtab_node*, va_heap, vl_embed>::sort(int (*)(void const*, void const*, void*), void*) ../../src/gcc/vec.h:1232 0x203e8c1 vec<symtab_node*, va_heap, vl_ptr>::sort(int (*)(void const*, void const*, void*), void*) ../../src/gcc/vec.h:2252 0x203e8c1 lto_output() ../../src/gcc/lto-streamer-out.cc:2799 0x203e453 write_lto ../../src/gcc/passes.cc:2779 0x8b2643 ipa_write_optimization_summaries(lto_symtab_encoder_d*) ../../src/gcc/passes.cc:2969 0x2039557 stream_out ../../src/gcc/lto/lto.cc:177 0x2039557 stream_out_partitions_1 ../../src/gcc/lto/lto.cc:229 0x1fcd178 stream_out_partitions ../../src/gcc/lto/lto.cc:244 0x1fcd178 lto_wpa_write_files ../../src/gcc/lto/lto.cc:417 0x1fcd178 do_whole_program_analysis ../../src/gcc/lto/lto.cc:592 0x1fcd178 lto_main() ../../src/gcc/lto/lto.cc:666 Please submit a full bug report, with preprocessed source (by using -freport-bug). Please include the complete backtrace with any bug report. See <file:///usr/share/doc/gcc-14/README.Bugs> for instructions. lto-wrapper: fatal error: gcc returned 1 exit status compilation terminated. /usr/bin/ld: error: lto-wrapper failed collect2: error: ld returned 1 exit status cgo_test.go:128: go build failed: exit status 1 cgo_test.go:146: failed FAIL FAIL cmd/link 10.473s ok cmd/link/internal/benchmark 0.020s