| Summary: | linux hard lock-up when gcc output file is on filesystem with insufficient disk space | ||
|---|---|---|---|
| Product: | gcc | Reporter: | Jason Vas Dias <jason.vas.dias> |
| Component: | other | Assignee: | Not yet assigned to anyone <unassigned> |
| Status: | RESOLVED INVALID | ||
| Severity: | normal | CC: | gcc-bugs, jason.vas.dias |
| Priority: | P3 | ||
| Version: | unknown | ||
| Target Milestone: | --- | ||
| Host: | x86_64-linux-gnu | Target: | x86_64-linux-gnu |
| Build: | x86_64-linux-gnu | Known to work: | |
| Known to fail: | Last reconfirmed: | ||
|
Description
Jason Vas Dias
2010-05-01 22:04:01 UTC
GCC does not write infinitely. This is a kernel bug. OK, but you cannot state "this is a kernel bug" without proving this to be the case . Every time I make a mistake about available disk space and start a gcc compile, and disk space is exhausted, the machine totally locks up: no response from keyboard, (including magic "Sys-Rq" keystroke) , mouse, network or any attached serial or USB device . Why does this happen with no other command ? GCC is an user level program. No user program should cause the kernel to lock up therefore it is a kernel issue. Has there been any history of reported linux kernel lock-ups for gcc compilations outputting to filesystems with insufficient disk space ? It could be something to do with using "make -j2" on my dual processor machine and outputting stderr + stdout to a "tee logfile" process. Of course, because the machine totally ceases to respond to any device, and does not create any crash report, the only evidence left is a few corrupt *.o files and terminated log files such as : gcc -Wp,-MD,fs/nfs/.proc.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include -I/usr/src/linux-2.6/arch/x86/include -Iinclude -include include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -O2 -m64 -march=k8 -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wframe-larger-than=2048 -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -pg -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fno-dwarf2-cfi-asm -fconserve-stack -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(proc)" -D"KBUILD_MODNAME=KBUILD_STR(nfs)" -D"DEBUG_HASH=59" -D"DEBUG_HASH2=19" -c -o fs/nfs/proc.o fs/nfs/proc.c set -e ; perl /usr/src/linux-2.6/scripts/recordmcount.pl "x86_64" "little" "64" "objdump" "objcopy" "gcc" "ld" "nm" "" "" "1" "drivers/block/paride/on20.o"; gcc -Wp,-MD,drivers/block/paride/.on26.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include -I/usr/src/linux-2.6/arch/x86/include -Iinclude -include include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -O2 -m64 -march=k8 -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wframe-larger-than=2048 -fno-omit-fram Sometimes disk space runs out before the machine locks up,
but I get loads of invalid objects ;
I think it is a bug that gcc can continue after producing invalid object files:
gcc -Wp,-MD,arch/x86/power/.cpu.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include -I/usr/src/linux-2.6/arch/x86/include -Iinclude -include include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -O2 -m64 -march=k8 -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wframe-larger-than=2048 -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -pg -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fno-dwarf2-cfi-asm -fconserve-stack -fno-stack-protector -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(cpu)" -D"KBUILD_MODNAME=KBUILD_STR(cpu)" -D"DEBUG_HASH=28" -D"DEBUG_HASH2=39" -c -o arch/x86/power/cpu.o arch/x86/power/cpu.c
{standard input}: Assembler messages:
{standard input}:66218: Fatal error: can't write drivers/infiniband/ulp/iser/iser_initiator.o: No space left on device
{standard input}:66218: Fatal error: can't close drivers/infiniband/ulp/iser/iser_initiator.o: No space left on device
make[3]: *** [drivers/infiniband/ulp/iser/iser_initiator.o] Error 1
make[2]: *** [drivers/infiniband/ulp/iser] Error 2
make[1]: *** [drivers/infiniband] Error 2
make: *** [drivers] Error 2
make: *** Waiting for unfinished jobs....
gcc -Wp,-MD,arch/x86/power/.hibernate_64.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include -I/usr/src/linux-2.6/arch/x86/include -Iinclude -include include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -O2 -m64 -march=k8 -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wframe-larger-than=2048 -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -pg -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fno-dwarf2-cfi-asm -fconserve-stack -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(hibernate_64)" -D"KBUILD_MODNAME=KBUILD_STR(hibernate_64)" -D"DEBUG_HASH=15" -D"DEBUG_HASH2=16" -c -o arch/x86/power/hibernate_64.o arch/x86/power/hibernate_64.c
{standard input}: Assembler messages:
{standard input}:29504: Fatal error: can't write arch/x86/power/cpu.o: No space left on device
{standard input}:29504: Fatal error: can't close arch/x86/power/cpu.o: No space left on device
make[1]: *** [arch/x86/power/cpu.o] Error 1
make[1]: *** Waiting for unfinished jobs....
{standard input}: Assembler messages:
{standard input}:25073: Fatal error: can't write arch/x86/power/hibernate_64.o: No space left on device
{standard input}:25073: Fatal error: can't close arch/x86/power/hibernate_64.o: No space left on device
make[1]: *** [arch/x86/power/hibernate_64.o] Error 1
make: *** [arch/x86/power] Error 2
[ root@jvdspc:/usr/src/linux 23:47:05 545:44 ]
$ find . -type f -a '(' '!' -type l ')' -a '(' -name '*.[oa]' -o -name '.so*' ')' | while read f; do if [[ ! $(file $f) =~ ELF ]]; then echo bad obj: $f; rm -f $obj; else d=$(dirname $f); mkdir -p /mnt/sda3/linux/${d} && cp -fp $f /mnt/sda3/linux/$d && rm -f $f && ln -s /mnt/sda3/linux/$f $f; fi; done
bad obj: ./drivers/block/aoe/built-in.o
bad obj: ./drivers/block/drbd/built-in.o
bad obj: ./drivers/block/paride/built-in.o
bad obj: ./drivers/bluetooth/built-in.o
bad obj: ./drivers/char/ip2/built-in.o
bad obj: ./drivers/char/ipmi/built-in.o
bad obj: ./drivers/char/mwave/built-in.o
bad obj: ./drivers/char/pcmcia/ipwireless/built-in.o
bad obj: ./drivers/crypto/built-in.o
bad obj: ./drivers/dca/built-in.o
bad obj: ./drivers/dma/ioat/built-in.o
bad obj: ./drivers/firewire/built-in.o
bad obj: ./drivers/gpu/drm/i2c/built-in.o
bad obj: ./drivers/gpu/drm/i810/built-in.o
bad obj: ./drivers/gpu/drm/i830/built-in.o
bad obj: ./drivers/gpu/drm/i915/built-in.o
bad obj: ./drivers/gpu/drm/mga/built-in.o
bad obj: ./drivers/gpu/drm/r128/built-in.o
bad obj: ./drivers/gpu/drm/savage/built-in.o
bad obj: ./drivers/gpu/drm/sis/built-in.o
bad obj: ./drivers/gpu/drm/tdfx/built-in.o
bad obj: ./drivers/gpu/drm/via/built-in.o
bad obj: ./drivers/hid/usbhid/built-in.o
bad obj: ./drivers/hid/built-in.o
bad obj: ./drivers/i2c/busses/built-in.o
bad obj: ./drivers/idle/built-in.o
bad obj: ./drivers/ieee1394/built-in.o
bad obj: ./drivers/ieee802154/built-in.o
bad obj: ./drivers/infiniband/core/built-in.o
bad obj: ./drivers/infiniband/hw/amso1100/built-in.o
bad obj: ./drivers/infiniband/hw/cxgb3/built-in.o
bad obj: ./drivers/infiniband/hw/ipath/built-in.o
bad obj: ./drivers/infiniband/hw/mlx4/built-in.o
bad obj: ./drivers/infiniband/hw/mthca/built-in.o
bad obj: ./drivers/infiniband/hw/nes/built-in.o
bad obj: ./drivers/infiniband/ulp/ipoib/built-in.o
bad obj: ./drivers/infiniband/ulp/iser/built-in.o
bad obj: ./drivers/infiniband/built-in.o
bad obj: ./drivers/input/joystick/built-in.o
bad obj: ./fs/ncpfs/.sock.o.cmd
bad obj: ./fs/nfs_common/built-in.o
bad obj: ./fs/nfsd/built-in.o
bad obj: ./fs/nilfs2/built-in.o
bad obj: ./fs/ntfs/built-in.o
bad obj: ./fs/ocfs2/cluster/built-in.o
bad obj: ./fs/ocfs2/dlm/built-in.o
bad obj: ./fs/ocfs2/dlmfs/built-in.o
bad obj: ./fs/ocfs2/built-in.o
bad obj: ./fs/omfs/built-in.o
bad obj: ./fs/proc/.softirqs.o.cmd
bad obj: ./fs/qnx4/built-in.o
bad obj: ./fs/reiserfs/built-in.o
bad obj: ./fs/smbfs/built-in.o
bad obj: ./fs/smbfs/.sock.o.cmd
bad obj: ./fs/squashfs/built-in.o
bad obj: ./fs/sysv/built-in.o
bad obj: ./fs/ubifs/built-in.o
bad obj: ./fs/ufs/built-in.o
bad obj: ./fs/xfs/built-in.o
bad obj: ./sound/arm/built-in.o
bad obj: ./sound/atmel/built-in.o
bad obj: ./sound/core/oss/built-in.o
bad obj: ./sound/core/seq/oss/built-in.o
bad obj: ./sound/core/seq/built-in.o
bad obj: ./sound/drivers/mpu401/built-in.o
bad obj: ./sound/drivers/opl3/built-in.o
bad obj: ./sound/drivers/opl4/built-in.o
bad obj: ./sound/drivers/pcsp/built-in.o
bad obj: ./sound/drivers/vx/built-in.o
bad obj: ./sound/drivers/built-in.o
bad obj: ./sound/i2c/other/built-in.o
bad obj: ./sound/i2c/built-in.o
bad obj: ./sound/isa/ad1816a/built-in.o
bad obj: ./sound/isa/ad1848/built-in.o
bad obj: ./sound/isa/cs423x/built-in.o
bad obj: ./sound/isa/es1688/built-in.o
bad obj: ./sound/isa/gus/built-in.o
bad obj: ./sound/isa/msnd/built-in.o
bad obj: ./sound/isa/opti9xx/built-in.o
bad obj: ./sound/isa/sb/built-in.o
bad obj: ./sound/isa/wavefront/built-in.o
bad obj: ./sound/isa/wss/built-in.o
bad obj: ./sound/isa/built-in.o
bad obj: ./sound/mips/built-in.o
bad obj: ./sound/oss/built-in.o
bad obj: ./sound/oss/.soundcard.o.cmd
bad obj: ./sound/oss/.sound_timer.o.cmd
bad obj: ./sound/oss/.sound.o.cmd
bad obj: ./sound/parisc/built-in.o
bad obj: ./sound/pci/ac97/built-in.o
bad obj: ./sound/pci/ali5451/built-in.o
bad obj: ./sound/pci/au88x0/built-in.o
bad obj: ./sound/pci/aw2/built-in.o
bad obj: ./sound/pci/ca0106/built-in.o
bad obj: ./sound/pci/cs46xx/built-in.o
bad obj: ./sound/pci/cs5535audio/built-in.o
bad obj: ./sound/pci/ctxfi/built-in.o
bad obj: ./sound/pci/echoaudio/built-in.o
bad obj: ./sound/pci/emu10k1/built-in.o
bad obj: ./sound/pci/hda/built-in.o
bad obj: ./sound/pci/ice1712/built-in.o
bad obj: ./sound/pci/korg1212/built-in.o
bad obj: ./sound/pci/lx6464es/built-in.o
bad obj: ./sound/pci/mixart/built-in.o
bad obj: ./sound/pci/nm256/built-in.o
bad obj: ./sound/pci/oxygen/built-in.o
bad obj: ./sound/pci/pcxhr/built-in.o
bad obj: ./sound/pci/riptide/built-in.o
bad obj: ./sound/pci/rme9652/built-in.o
bad obj: ./sound/pci/trident/built-in.o
bad obj: ./sound/pci/vx222/built-in.o
bad obj: ./sound/pci/ymfpci/built-in.o
bad obj: ./sound/pci/built-in.o
bad obj: ./sound/pcmcia/pdaudiocf/built-in.o
bad obj: ./sound/pcmcia/vx/built-in.o
bad obj: ./sound/ppc/built-in.o
bad obj: ./sound/sh/built-in.o
bad obj: ./sound/soc/atmel/built-in.o
bad obj: ./sound/soc/au1x/built-in.o
bad obj: ./sound/soc/blackfin/built-in.o
bad obj: ./sound/soc/codecs/built-in.o
bad obj: ./sound/soc/davinci/built-in.o
bad obj: ./sound/soc/fsl/built-in.o
bad obj: ./sound/soc/imx/built-in.o
bad obj: ./sound/soc/omap/built-in.o
bad obj: ./sound/soc/pxa/built-in.o
bad obj: ./sound/soc/s3c24xx/built-in.o
bad obj: ./sound/soc/s6000/built-in.o
bad obj: ./sound/soc/sh/built-in.o
bad obj: ./sound/soc/txx9/built-in.o
bad obj: ./sound/soc/built-in.o
bad obj: ./sound/sparc/built-in.o
bad obj: ./sound/spi/built-in.o
bad obj: ./sound/synth/emux/built-in.o
bad obj: ./sound/synth/built-in.o
bad obj: ./sound/usb/caiaq/built-in.o
bad obj: ./sound/usb/usx2y/built-in.o
bad obj: ./sound/usb/built-in.o
Aha ! I think I see the problem, or at least A problem : $ strace -f gcc ~jason/t.c -o /tmp/t 2>&1 | grep sync $ Nothing is doing a fsync(int fd) for any object file written by gcc . Hence writes by gcc can be entirely to the filesystem cache, and when disk space runs out, many files may not have been sync()-ed to disk within a large make(1) build. I propose that gcc (and binutils!) should do an fsync() of each output file written before closing and returning success. It is the responsibility of your filesystem to ensure that write or close return error status if there isn't sufficient disk space; if they do not, this is a kernel bug. Most applications should never call fsync; that's only if the application is meant to be making data integrity guarantees across unclean shutdown, and it never makes sense to make such guarantees for compiler and assembler output since those files can always be recreated from the associated input files; it doesn't matter if the last few seconds' compilation output before a crash is lost or corrupted. |