I'm reporting something that looks like a bug, at least I can't imagine how this behavior could be considered right, but I admit I haven't read the entire C and Fortran specifications that gfortran is following. When specifying the "STAT" option with the ALLOCATE statement, no error is returned for some extremely large arrays that obviously can not fit into memory. test.f90: PROGRAM MAIN IMPLICIT NONE DOUBLE PRECISION, ALLOCATABLE :: E(:,:,:,:) INTEGER :: N, ierr CHARACTER(LEN=32) :: arg CALL get_command_argument(1, arg) READ(arg,*) N ALLOCATE(E(N,N,N,N),STAT=ierr) IF (ierr/=0) THEN PRINT *,'Could not allocate array of size ',N,'**4, error code=', ierr ELSE PRINT *,'Sucesfully allocated array of size ',N,'**4' E(N,N,N,N)=1.0D0 DEALLOCATE(E) ENDIF END PROGRAM MAIN Run with: $ gfortran test.f90; for i in `seq -w 10 10 400`; do ./a.out $i; done Sucesfully allocated array of size 10 **4 Sucesfully allocated array of size 20 **4 Sucesfully allocated array of size 30 **4 Sucesfully allocated array of size 40 **4 Sucesfully allocated array of size 50 **4 Sucesfully allocated array of size 60 **4 Sucesfully allocated array of size 70 **4 Sucesfully allocated array of size 80 **4 Sucesfully allocated array of size 90 **4 Sucesfully allocated array of size 100 **4 Sucesfully allocated array of size 110 **4 Sucesfully allocated array of size 120 **4 Could not allocate array of size 130 **4, error code= 5014 Could not allocate array of size 140 **4, error code= 5014 Could not allocate array of size 150 **4, error code= 5014 Sucesfully allocated array of size 160 **4 Could not allocate array of size 170 **4, error code= 5014 Could not allocate array of size 180 **4, error code= 5014 Sucesfully allocated array of size 190 **4 Could not allocate array of size 200 **4, error code= 5014 Could not allocate array of size 210 **4, error code= 5014 Sucesfully allocated array of size 220 **4 Sucesfully allocated array of size 230 **4 Sucesfully allocated array of size 240 **4 Sucesfully allocated array of size 250 **4 Could not allocate array of size 260 **4, error code= 5014 It's obvious that on my 3Gb-RAM machine, the 250**4 array was NOT allocated.
What operating system and does it use a virtual memory system? Change 'e(n,n,n,n) = 1' to 'e = 1' and see what happens. In one case, you are touching only the last page in vm while in the other you are touching all pages.
With "E=1.0D0" instead of "E(N,N,N,N)=1.0D0" $ gfortran test.f90; for i in `seq -w 10 10 400`; do LANG=C ./a.out $i; done Sucesfully allocated array of size 10 **4 Sucesfully allocated array of size 20 **4 Sucesfully allocated array of size 30 **4 Sucesfully allocated array of size 40 **4 Sucesfully allocated array of size 50 **4 Sucesfully allocated array of size 60 **4 Sucesfully allocated array of size 70 **4 Sucesfully allocated array of size 80 **4 Sucesfully allocated array of size 90 **4 Sucesfully allocated array of size 100 **4 Sucesfully allocated array of size 110 **4 Sucesfully allocated array of size 120 **4 Could not allocate array of size 130 **4, error code= 5014 Could not allocate array of size 140 **4, error code= 5014 Could not allocate array of size 150 **4, error code= 5014 Sucesfully allocated array of size 160 **4 Erreur de segmentation Could not allocate array of size 170 **4, error code= 5014 Could not allocate array of size 180 **4, error code= 5014 Sucesfully allocated array of size 190 **4 Erreur de segmentation Could not allocate array of size 200 **4, error code= 5014 Could not allocate array of size 210 **4, error code= 5014 Sucesfully allocated array of size 220 **4 Erreur de segmentation Sucesfully allocated array of size 230 **4 Erreur de segmentation Sucesfully allocated array of size 240 **4 Erreur de segmentation Sucesfully allocated array of size 250 **4 Erreur de segmentation Could not allocate array of size 260 **4, error code= 5014 Could not allocate array of size 270 **4, error code= 5014 Sucesfully allocated array of size 280 **4 Erreur de segmentation Sucesfully allocated array of size 290 **4 Erreur de segmentation Sucesfully allocated array of size 300 **4 Erreur de segmentation Sucesfully allocated array of size 310 **4 Erreur de segmentation Could not allocate array of size 320 **4, error code= 5014 Sucesfully allocated array of size 330 **4 Erreur de segmentation Could not allocate array of size 340 **4, error code= 5014 Could not allocate array of size 350 **4, error code= 5014 Sucesfully allocated array of size 360 **4 Erreur de segmentation Could not allocate array of size 370 **4, error code= 5014 Could not allocate array of size 380 **4, error code= 5014 Sucesfully allocated array of size 390 **4 Erreur de segmentation Could not allocate array of size 400 **4, error code= 5014 (Excuse the non-English words, "Erreur de segmentation" really means "Segmentation fault"). I am using Ubuntu 10.04.3 LTS (32-bit) with: $ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : stepping : 10 cpu MHz : 2493.666 cache size : 1024 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm xsave lahf_lm tpr_shadow vnmi flexpriority bogomips : 4987.33 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual
From the metadata, it seems you're using version 4.4.3, where the overflow check when calculating the size to allocate was a bit stupid. Basically it did the calculation, and if the result was negative, overflow was detected. But this check missed the cases where the calculated values would wrap around to a positive value again. I believe this is what you're seeing when some of the intermediate values succeed. In current versions of the compiler, overflow check is bit smarter. See PR 28105. Also, as Steve Kargl hinted at, many operating systems allow memory to be overcommitted. I.e. allocating memory succeeds, but the program is killed later on if it actually tries to use that memory. Gfortran has no checks whether the amount of memory you try to allocate is some reasonable fraction of the memory installed in the machine, it just does what you tell it to do.
I'm using: $ gfortran --version GNU Fortran (Ubuntu 4.4.3-4ubuntu5) 4.4.3 I've now tested the same program on a 64-bit CentOs machine with 16-Gb RAM, but wasn't able to reproduce the problem there: $ gfortran --version GNU Fortran (GCC) 4.1.2 $ for i in `seq -w 10 10 400`; do LANG=C ./a.out $i; done Sucesfully allocated array of size 10 **4 Sucesfully allocated array of size 20 **4 Sucesfully allocated array of size 30 **4 Sucesfully allocated array of size 40 **4 Sucesfully allocated array of size 50 **4 Sucesfully allocated array of size 60 **4 Sucesfully allocated array of size 70 **4 Sucesfully allocated array of size 80 **4 Sucesfully allocated array of size 90 **4 Sucesfully allocated array of size 100 **4 Sucesfully allocated array of size 110 **4 Sucesfully allocated array of size 120 **4 Sucesfully allocated array of size 130 **4 Sucesfully allocated array of size 140 **4 Sucesfully allocated array of size 150 **4 Sucesfully allocated array of size 160 **4 Sucesfully allocated array of size 170 **4 Sucesfully allocated array of size 180 **4 Sucesfully allocated array of size 190 **4 Sucesfully allocated array of size 200 **4 Sucesfully allocated array of size 210 **4 Could not allocate array of size 220 **4, error code= 1 Could not allocate array of size 230 **4, error code= 1 Could not allocate array of size 240 **4, error code= 1 Could not allocate array of size 250 **4, error code= 1 Could not allocate array of size 260 **4, error code= 1 Could not allocate array of size 270 **4, error code= 1 Could not allocate array of size 280 **4, error code= 1 Could not allocate array of size 290 **4, error code= 1 Could not allocate array of size 300 **4, error code= 1 Could not allocate array of size 310 **4, error code= 1 Could not allocate array of size 320 **4, error code= 1 Could not allocate array of size 330 **4, error code= 1 Could not allocate array of size 340 **4, error code= 1 Could not allocate array of size 350 **4, error code= 1 Could not allocate array of size 360 **4, error code= 1 Could not allocate array of size 370 **4, error code= 1 Could not allocate array of size 380 **4, error code= 1 Could not allocate array of size 390 **4, error code= 1 Could not allocate array of size 400 **4, error code= 1
(In reply to comment #4) > I've now tested the same program on a 64-bit CentOs machine > with 16-Gb RAM, but wasn't able to reproduce the problem there: Yes, because on a 64-bit platform the size calculation is done with 64-bit integers, and long before such integers overflow the OS decides to deny the allocation request (as can be seen e.g. by the different error code). I'm not really seeing any bug here.
Has the bug been corrected in recent versions of gfortran, or do you really mean it's OK that gfortran claims an array has been allocated when it really has not been?
On Mon, Oct 31, 2011 at 07:25:38PM +0000, fwi at inducks dot org wrote: > > Has the bug been corrected in recent versions of gfortran, or do you really > mean it's OK that gfortran claims an array has been allocated when it really > has not been? > Yes, the problem of integer overflow that Janne mentioned has been corrected. As to gfortran claiming "that an array has been allocated when it really has not been", suggests that you may not understand how your operating system works. The concise story is that gfortran asks your OS for a gazillion bytes of memory and please confirm the status. The OS tells gfortran (actually your compiled code), here's your gazillion bytes of virtual memory and everything is a-okay. As long as you do not actually touch the pages of allocated virtual memory, no physical memory has been allocated to your program. Once you start touching pages, at some point the OS notices a shortage of physical memory. This shortage leads to swapping other (idle) processes out to secondary storage, utilizing swap space as the backing storage for your virtual memory, evidently your process exceeds its limits or the system limits and the process segfaults.
I do not(In reply to comment #7) > On Mon, Oct 31, 2011 at 07:25:38PM +0000, fwi at inducks dot org wrote: > Yes, the problem of integer overflow that Janne mentioned has > been corrected. Great. I indeed do not know everything about the OS and what it does when I "allocate" an array. But that's exactly the purpose of a programming language like Fortran, an abstraction that should be "good enough" for programing without having to know everything about the OS. Secondly, users are sometimes better than programmers at telling them if something is really useful or not. In that case, the question is: what is the purpose of the STAT flag in an allocate STATEMENT if it won't give you any reasonable indication if the array you have can be used or not.
On Mon, Oct 31, 2011 at 08:17:51PM +0000, fwi at inducks dot org wrote: > http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937 > > --- Comment #8 from fwi at inducks dot org 2011-10-31 20:17:51 UTC --- > I do not(In reply to comment #7) > > On Mon, Oct 31, 2011 at 07:25:38PM +0000, fwi at inducks dot org wrote: > > Yes, the problem of integer overflow that Janne mentioned has > > been corrected. > > Great. > > I indeed do not know everything about the OS and what it does when I "allocate" > an array. But that's exactly the purpose of a programming language like > Fortran, an abstraction that should be "good enough" for programing without > having to know everything about the OS. > Secondly, users are sometimes better than programmers at telling them if > something is really useful or not. In that case, the question is: what is the > purpose of the STAT flag in an allocate STATEMENT if it won't give you any > reasonable indication if the array you have can be used or not. > Use a newer version of gfortran 4.1.2 and 4.4.3 are old. Install 4.6.2 and see what happens. You've been told twice that the integer overflow has been fixed.
(In reply to comment #8) > I indeed do not know everything about the OS and what it does when I "allocate" > an array. But that's exactly the purpose of a programming language like > Fortran, an abstraction that should be "good enough" for programing without > having to know everything about the OS. Sometimes abstractions leak, unfortunately. There's really not anything gfortran can do about that. And, it's not unique to gfortran either. gfortran ALLOCATE uses the C malloc(), as does e.g. the default implementation of the C++ new operator and presumably a lot of other language runtimes as well. So they all share the same issues in case the system overcommits memory. I can certainly sympathize with the notion that memory overcommit is inane and shouldn't be enabled by default, but that's a system policy issue and nothing gfortran can do anything about. As you're on Linux, FWIW you can disable overcommit by setting the "vm.overcommit_memory" sysctl to the value 2. See http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting > Secondly, users are sometimes better than programmers at telling them if > something is really useful or not. In that case, the question is: what is the > purpose of the STAT flag in an allocate STATEMENT if it won't give you any > reasonable indication if the array you have can be used or not. Indeed, on a system which overcommits memory, ALLOCATE with STAT is not particularly useful. But, again, memory overcommitting is a system policy issue and gfortran can't do anything about it. And, one might add, if all you're going to do with the STAT result is checking whether it's nonzero and stopping the program in that case, you might as well not bother because that's exactly what ALLOCATE without STAT already does.
(In reply to comment #10) > Sometimes abstractions leak, unfortunately. There's really not anything > gfortran can do about that. And, it's not unique to gfortran either. gfortran > ALLOCATE uses the C malloc(), as does e.g. the default implementation of the > C++ new operator and presumably a lot of other language runtimes as well. So > they all share the same issues in case the system overcommits memory. I hate to continue this discussion, but I think the integer overflow problem, which is what I was reporting, was a different issue. The fact that you solved it in recent gfortran versions proves that there was something you could do about that. > I can certainly sympathize with the notion that memory overcommit is inane and > shouldn't be enabled by default, but that's a system policy issue and nothing > gfortran can do anything about. As you're on Linux, FWIW you can disable > overcommit by setting the "vm.overcommit_memory" sysctl to the value 2. See > http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting Thanks for the info. > And, one might add, if all you're going to do with the STAT result is checking > whether it's nonzero and stopping the program in that case, you might as well > not bother because that's exactly what ALLOCATE without STAT already does. Not really, because then I can directly tell users how they can solve the issue, that is either switch to a 64bit OS or compile the MPI version of the same program (because then the main array is splitted in several chunks, all of them small enough to be indexed with 32bit integers). Without the STAT option, users will be left in the dark with just "it doesn't work". Or with "I need to spend time reading the manual". And please - I hope you all do not take my remarks as harsh criticism, I do appreciate the efforts and job you did with gfortran. A few years ago, my code was much faster with ifort than gcc's Fortran, nowadays it's comparable and I can tell people to use gfortran if they like.