I have been getting this error on one of my programs that does extremely complex I/O operations for a few months now, but I have been unable to isolate it. I finally got the same error message from a program that generates random I/O patterns. About half of the FORTRANS that I run this program with fail, but I found four including Lahey Fortran and Compaq Fortran that have no complain. testio.f is the original program. testio2.f is a program made from the output of testio.f that duplicates the I/O sequence that failed. [dranta:~/tests/gfortran-D] dir% gfortran -o testio testio.f [dranta:~/tests/gfortran-D] dir% testio > out At line 37 of file testio.f Fortran runtime error: Invalid argument [dranta:~/tests/gfortran-D] dir% cat testio.f program test dimension ia(100) i=rand(500) do 10 i=1,100 call doio(ia,100) 10 continue stop end subroutine doio(ia,n) dimension ia(n) i=1 10 continue ia(i)=1+3*rand(0)+.5 if(i.gt.1)then if(ia(i-1).eq.4.and.ia(i).eq.3)goto 10 endif i=i+1 if(i.le.n)goto 10 write(6,1000)ia call dowrite(ia,n) return 1000 format(20i4) end subroutine dowrite(ia,n) dimension ia(n) dimension data(500) nend=1000 open(unit=11,status='scratch',form='unformatted') do 100 i=1,n goto (10,20,30,40),ia(i) 10 continue write(6,*)' rewind 11' rewind 11 goto 100 20 continue write(6,*)' backspace 11' backspace 11 goto 100 30 continue write(6,*)' read(11,end=',nend,')data' write(6,1000)nend nend=nend+1 read(11,end=50)data goto 100 40 continue write(6,*)' write(11)data' write(11)data goto 100 50 continue write(6,*)' backspace 11' backspace 11 goto 100 100 continue close(11) return 1000 format(' ',i4,' continue') end [dranta:~/tests/gfortran-D] dir% gfortran -o testio2 testio2.f [dranta:~/tests/gfortran-D] dir% testio2 At line 72 of file testio2.f Fortran runtime error: Invalid argument [dranta:~/tests/gfortran-D] dir% cat testio2.f program test dimension data(100) read(11,end= 1000 )data 1000 continue backspace 11 backspace 11 backspace 11 read(11,end= 1001 )data 1001 continue backspace 11 rewind 11 backspace 11 backspace 11 backspace 11 rewind 11 write(11)data backspace 11 backspace 11 backspace 11 backspace 11 backspace 11 backspace 11 rewind 11 backspace 11 write(11)data write(11)data write(11)data backspace 11 read(11,end= 1002 )data 1002 continue backspace 11 rewind 11 backspace 11 read(11,end= 1003 )data 1003 continue write(11)data write(11)data write(11)data rewind 11 read(11,end= 1004 )data 1004 continue rewind 11 rewind 11 read(11,end= 1005 )data 1005 continue backspace 11 write(11)data backspace 11 rewind 11 backspace 11 rewind 11 read(11,end= 1006 )data 1006 continue rewind 11 write(11)data backspace 11 read(11,end= 1007 )data 1007 continue read(11,end= 1008 )data 1008 continue backspace 11 read(11,end= 1009 )data 1009 continue backspace 11 write(11)data backspace 11 read(11,end= 1010 )data 1010 continue read(11,end= 1011 )data 1011 continue backspace 11 backspace 11 stop end
Reduced test case: program test dimension data(100) read(11,end= 1000 )data 1000 continue backspace 11 backspace 11 end I will have to check to see if this is legal code.
Well, since it relies on uninitialized variables, it's a bit hard to reproduce. The following code (I believe it is legal) reproducibly fails for me on i686-linux: integer dat(5) dat = (/ 0, 0, 0, 0, 18651460 /) write(11) dat,dat,dat rewind 11 write(11) dat,dat,dat,dat rewind 11 write(11) dat rewind 11 write(11) dat rewind 11 read(11) dat read(11,end=1008) dat 1008 continue backspace 11 write(11) dat backspace 11 read(11) dat read(11,end=1011) dat 1011 continue backspace 11 backspace 11 end I think it has to do with array transfer, and overwriting the record markes with data. Adding Janne in CC list for opinion.
The testcase can be reduced further: integer dat(5) dat = (/ 0, 0, 0, 0, 1 /) write(11) dat,dat,dat,dat rewind 11 write(11) dat read(11,end=1008) dat 1008 continue backspace 11 write(11) dat read(11,end=1011) dat 1011 continue backspace 11 backspace 11 end At line 13 of file testio2.f Fortran runtime error: Invalid argument From looking at the fort.11 file, I think the output is correct at every point of the program. Also note that if we comment two lines of this testcase, like that: integer dat(5) dat = (/ 0, 0, 0, 0, 1 /) write(11) dat,dat,dat,dat rewind 11 write(11) dat read(11,end=1008) dat 1008 continue ! backspace 11 ! write(11) dat read(11,end=1011) dat 1011 continue backspace 11 backspace 11 end then we get another error message (which shouldn't happen): At line 10 of file testio2.f Fortran runtime error: Read past ENDFILE record I don't know yet if these bugs have the same cause.
It doesn't have anything to do with array transfers. The following test program is equivalent to the one in #3, but uses the scalar transfer_integer instead of transfer_array: program test3 integer dat(5) dat = (/ 1, 2, 3, 4, 5 /) write(11) dat(1), dat(2), dat(3), dat(4), dat(5), & dat(1), dat(2), dat(3), dat(4), dat(5), & dat(1), dat(2), dat(3), dat(4), dat(5), & dat(1), dat(2), dat(3), dat(4), dat(5) rewind 11 write(11) dat(1), dat(2), dat(3), dat(4), dat(5) read(11,end=1008) dat(1), dat(2), dat(3), dat(4), dat(5) 1008 continue backspace 11 write(11) dat(1), dat(2), dat(3), dat(4), dat(5) read(11,end=1011) dat(1), dat(2), dat(3), dat(4), dat(5) 1011 continue backspace 11 backspace 11 end program test3 Depending on whether the two lines after 1008 continue are commented or not, it shows the same errors as the ones in #3.
There are two errors on this one. The first is not a regression (happens with 4.0.3): $ cat a.f90 integer :: i = 1 open(11,status="replace",form="unformatted") read(11,end=1008) i 1008 continue read(11,end=1011) i 1011 continue end $ gfortran a.f90 && ./a.out At line 5 of file a.f90 Fortran runtime error: Read past ENDFILE record The second one is a regression (doesn't happen with 4.0.3). $ cat a.f90 integer :: i = 1 open(11,status="replace",form="unformatted") write(11) dat read(11,end=1008) i 1008 continue backspace 11 backspace 11 backspace 11 end $ gfortran a.f90 && ./a.out At line 8 of file a.f90 Fortran runtime error: Invalid argument PS: sorry Janne for hinting that it might have been related to your patch. Reducing this has been a bit painful.
I found the problem in the second one, which is fixed by the one-line patch: Index: io/file_pos.c =================================================================== --- io/file_pos.c (revision 108353) +++ io/file_pos.c (working copy) @@ -109,13 +109,14 @@ length = sizeof (gfc_offset); - p = salloc_r_at (u->s, &length, - file_position (u->s) - length); + /* length is modified by this function call, and most probably + set to zero. */ + p = salloc_r_at (u->s, &length, file_position (u->s) - length); if (p == NULL) goto io_error; memcpy (&m, p, sizeof (gfc_offset)); - new = file_position (u->s) - m - 2*length; + new = file_position (u->s) - m - sizeof (gfc_offset) - length; if (sseek (u->s, new) == FAILURE) goto io_error; (well, more than one line since I added a comment so that we don't do this mistake again). Since this does not fixed the other problem, I filed it separately under PR 25340.
This patch works for the reduced test case, but the original test case still fails for me - [dranta:~/tests/gfortran-D] dir% gfortran -o write08 write08.f [dranta:~/tests/gfortran-D] dir% write08 [dranta:~/tests/gfortran-D] dir% cat write08.f integer :: i = 1 open(11,status="replace",form="unformatted") write(11) dat read(11,end=1008) i 1008 continue backspace 11 backspace 11 backspace 11 end [dranta:~/tests/gfortran-D] dir% gfortran -o testio2 testio2.f [dranta:~/tests/gfortran-D] dir% testio2 At line 72 of file testio2.f Fortran runtime error: Invalid argument [dranta:~/tests/gfortran-D] dir% cat testio2.f program test dimension data(100) read(11,end= 1000 )data 1000 continue backspace 11 backspace 11 backspace 11 read(11,end= 1001 )data 1001 continue backspace 11 rewind 11 backspace 11 backspace 11 backspace 11 rewind 11 write(11)data backspace 11 backspace 11 backspace 11 backspace 11 backspace 11 backspace 11 rewind 11 backspace 11 write(11)data write(11)data write(11)data backspace 11 read(11,end= 1002 )data 1002 continue backspace 11 rewind 11 backspace 11 read(11,end= 1003 )data 1003 continue write(11)data write(11)data write(11)data rewind 11 read(11,end= 1004 )data 1004 continue rewind 11 rewind 11 read(11,end= 1005 )data 1005 continue backspace 11 write(11)data backspace 11 rewind 11 backspace 11 rewind 11 read(11,end= 1006 )data 1006 continue rewind 11 write(11)data backspace 11 read(11,end= 1007 )data 1007 continue read(11,end= 1008 )data 1008 continue backspace 11 read(11,end= 1009 )data 1009 continue backspace 11 write(11)data backspace 11 read(11,end= 1010 )data 1010 continue read(11,end= 1011 )data 1011 continue backspace 11 backspace 11 stop end [dranta:~/tests/gfortran-D] dir% gfortran --v Using built-in specs. Target: powerpc-apple-darwin8.3.0 Configured with: ./configure --prefix=/Users/dir/gfortran --enable-languages=c,f95 Thread model: posix gcc version 4.2.0 20051212 (experimental) [dranta:~/tests/gfortran-D] dir%
Looking into unformatted_backspace, there does not seem to be anything there to prevent "u->last_record" from going negative - may be that is problem.
I generated random I/O sequences doing 100,000 tests at each I/O operation count staring at five here is the shortest one to fail (somewhere in the eights) - [dranta:~/tests/gfortran-D] dir% gfortran -o write10 write10.f [dranta:~/tests/gfortran-D] dir% write10 At line 18 of file write10.f Fortran runtime error: Invalid argument [dranta:~/tests/gfortran-D] dir% cat write10.f integer :: i = 1 dimension data(500) do 10 i=1,500 data(i)=i 10 continue open(unit=11,status='scratch',form='unformatted') write(11)data write(11)data rewind 11 write(11)data backspace 11 read(11,end= 1000 )data 1000 continue read(11,end= 1001 )data 1001 continue backspace 11 backspace 11 backspace 11 close(11) end
Removing the constraint of no read after write gives a slightly shorter one - [dranta:~/tests/gfortran-D] dir% gfortran -o write11 write11.f [dranta:~/tests/gfortran-D] dir% write11 At line 14 of file write11.f Fortran runtime error: Invalid argument [dranta:~/tests/gfortran-D] dir% cat write11.f integer :: i = 1 dimension data(5) do 10 i=1,5 data(i)=i 10 continue open(unit=11,status='scratch',form='unformatted') write(11)data read(11,end= 1000 )data 1000 continue backspace 11 rewind 11 read(11,end= 1001 )data 1001 continue read(11,end= 1002 )data 1002 continue close(11) end
When the array of size 2045 or larger the error goes away - [dranta:~/tests/gfortran-D] dir% gfortran -o write11 write11.f [dranta:~/tests/gfortran-D] dir% write11 At line 14 of file write11.f Fortran runtime error: Invalid argument [dranta:~/tests/gfortran-D] dir% cat write11.f integer :: i = 1 dimension data(2044) do 10 i=1,5 data(i)=i 10 continue open(unit=11,status='scratch',form='unformatted') write(11)data read(11,end= 1000 )data 1000 continue c backspace 11 rewind 11 read(11,end= 1001 )data 1001 continue read(11,end= 1002 )data 1002 continue close(11) end [dranta:~/tests/gfortran-D] dir% gfortran -o write11 write11.f [dranta:~/tests/gfortran-D] dir% write11 [dranta:~/tests/gfortran-D] dir% cat write11.f integer :: i = 1 dimension data(2045) do 10 i=1,5 data(i)=i 10 continue open(unit=11,status='scratch',form='unformatted') write(11)data read(11,end= 1000 )data 1000 continue c backspace 11 rewind 11 read(11,end= 1001 )data 1001 continue read(11,end= 1002 )data 1002 continue close(11) end
Removing patch keyword and reassigning it to nobody as my patch doesn't fix it and I won't have free time before january. Sorry for not doing this earlier.
Fortran problems are not release critical: P5.
After tracing the errors for a while, it became clean that "active" pointer into the read/write buffer was not being correctly updated. Adding one line ( s->active=0; ) to the bottom of routine "fd_truncate" in file "unix.c" fixes these errors. All the examples here run OK with this patch and I built 16 of my programs and ran their many test problems with no errors. I tried to run the fortran testsuite with "make check-gfortran", but check-gfortran is not in the makefile. It would be nice if just fortran testsuite could be run. *** unix.c Wed Dec 21 08:11:12 2005 --- unixoriginal.c Wed Dec 21 08:10:27 2005 *************** *** 601,608 **** } s->physical_offset = s->file_length = s->logical_offset; - - s->active=0; return SUCCESS; } --- 601,606 ----
(In reply to comment #14) > I tried to run the > fortran testsuite with "make check-gfortran", but check-gfortran is not > in the > makefile. It would be nice if just fortran testsuite could be run. > Dale, move into the gcc/ directory of your build directory and run the "make check-gfortran" command. Yes, it's an unexpected PITA, but it should work.
I down loaded gfortran and built it on the Macintosh with - svn -q checkout svn://gcc.gnu.org/svn/gcc/trunk gcc cd gcc configure --prefix=/Users/dir/gfortran --enable-languages=c,f95 make -j 4 make install When, I try the "check-gfortran" in the directory 'gcc' where I did the configure,make -j 4,make install, I get - [dranta:~/gfortran/gcc] dir% make check-gfortran make: *** No rule to make target `check-gfortran'. Stop. When I move to the directory 'gcc' below that, I get - [dranta:~/gfortran/gcc] dir% cd gcc [dranta:~/gfortran/gcc/gcc] dir% make check-gfortran make: *** No rule to make target `check-gfortran'. Stop. [dranta:~/gfortran/gcc/gcc] dir% Is there another directory called gcc or does it need to be build some other way ?
(In reply to comment #16) > When, I try the "check-gfortran" in the directory 'gcc' where I did the > configure,make -j 4,make install, I get Dale, you should really stop building your compiler inside the source tree. This isn't very actively supported (actually, it breaks every so often) and it makes everything harder. For example, I usually create a directory parent directory build/, including build/gcc which is the source tree and build/ibin which is the build. Inside build/ibin, do: ../gcc/configure [...] && make && make install What Steve is saying is that you have to go inside the build/ibin/gcc directory and run make check-gfortran *there*. > Is there another directory called gcc or does it need to be build some other > way ? I guess with your scheme, you need to go inside gcc/gcc/ and run make check-gfortran. But please, don't do in-tree builds :)
This is great! I have regression tested and NIST tested on i686 and all pass. I have not tried some of the problem cases yet, but will do later tonight. I was just getting ready to start working on this one and I am tickled pink that it appears to be a fix. Who will approve and who will commit this. I will be happy to commit with a test case if needed.
Ran some test cases. The second example in Comment #3 fails. With or without FX patch given in Comment #6 At line 10 of file back3.f Fortran runtime error: Read past ENDFILE record Dale, are you getting this?
Thanks FX - You gave me the first good explaination of how it should be done.
The second example in comment 3 should fail as it does - since it does try read beyond the end of file. As far as I can tell, the FX patch does not fix any of the test cases on the Macintosh and I think that it actually is incorrect. It should actually read - new = file_position (u->s) - m - 2*sizeof (gfc_offset); as "2*sizeof (gfc_offset)" is the size of the green words before and after the data and m is the size of the data. (m+2*sizeof (gfc_offset)) is the total size of the binary record that is being skipped backwards over.
(In reply to comment #21) > new = file_position (u->s) - m - 2*sizeof (gfc_offset); > > as "2*sizeof (gfc_offset)" is the size of the green words before and after the > data and m is the size of the data. (m+2*sizeof (gfc_offset)) is the total size > of the binary record that is being skipped backwards over. Did you test that patch? Does it work? I would be very surprised if it did. You need to consider that the previous read (or salloc_r_at, to be exact) might or might not read all length bytes. So, after that call, length might be sizeof(gfc_offset), or it might be 0, or it might be something in between. You need to do something that takes into account all these cases, and that's what I thought my patch did. I don't have time now to make all the reasonning again, but I just wanted to underline that issue so that you're aware of it.
What is happening here is actually quite simple. The program reads the green word for the previous record from the file from location "file_position (u->s) - length)", that word gives the length of the previous record in bytes. It then pulls that number out of the buffer and puts it into correct integer form for the current machine, that is the number 'm'. The length of a binary record is always the length of the two green words plus the length of the data or (m + 2*sizeof (gfc_offset)). The position before the currect record is then file_position (u->s) - m - 2*sizeof (gfc_offset). I did not change the current coding and it works for all of my tests. Any time that p is not NULL, length returns the number of bytes read, which should be sizeof (gfc_offset). If length is not equal to sizeof (gfc_offset), there is an error in the file data - so may be that should be put in as an error test.
Also, I just noticed that, if length != sizeof (gfc_offset) then 'm' has not been correctly read - so the routine should quit. Using a bad number to calculate 'new' will not likely useful.
The failure in the second example in comment #3 is that the reading past the record is not being caught by the end specifier in the read statement. It should go to 1011 and continue. This is a different bug. I will try to hunt this down this weekend unless Dale does. Let me know off bugzilla.
I ran the "check-gfortran" tests before and after putting in the two changes that I suggested on the Macintosh and the results were identical and my real programs now run Ok with the changes, but someone is currectly making changes to the fortran io routines and some of the tests given here are failing in new ways - Test Run By dir on Thu Dec 22 09:47:44 2005 Native configuration is powerpc-apple-darwin8.3.0 === gfortran tests === Schedule of variations: unix Running target unix Using /usr/local/share/dejagnu/baseboards/unix.exp as board description file for target. Using /usr/local/share/dejagnu/config/unix.exp as generic interface file for target. Using /Users/dir/gfortran/gcc/gcc/testsuite/config/default.exp as tool-and-target-specific interface file. Running /Users/dir/gfortran/gcc/gcc/testsuite/gfortran.dg/dg.exp ... FAIL: gfortran.dg/large_real_kind_2.F90 execution test FAIL: gfortran.dg/large_real_kind_2.F90 execution test FAIL: gfortran.dg/large_real_kind_2.F90 execution test FAIL: gfortran.dg/large_real_kind_2.F90 execution test FAIL: gfortran.dg/large_real_kind_2.F90 execution test FAIL: gfortran.dg/large_real_kind_2.F90 execution test FAIL: gfortran.dg/large_real_kind_2.F90 execution test FAIL: gfortran.dg/large_real_kind_2.F90 execution test Running /Users/dir/gfortran/gcc/gcc/testsuite/gfortran.dg/vect/vect.exp ... Running /Users/dir/gfortran/gcc/gcc/testsuite/gfortran.fortran-torture/compile/compile.exp ... Running /Users/dir/gfortran/gcc/gcc/testsuite/gfortran.fortran-torture/execute/execute.exp ... === gfortran Summary === # of expected passes 10985 # of unexpected failures 8 # of expected failures 7 # of unsupported tests 42
In the second example in comment #3, the first read reads the end of file and the end=1008 works correctly. On the second read, the end of file has been passed so that the end=1011 does not and should not catch it - it is an error condition and a correct error message is printed. If err=1011 was used, then the error message should not be printed and the programs should jump to 1011 with out comment.
On i686-pc-linux-gnu: $ cat back3.f integer dat(5) dat = (/ 0, 0, 0, 0, 1 /) write(11) dat,dat,dat,dat rewind 11 write(11) dat read(11,end=1008) dat call abort() 1008 continue ! backspace 11 ! write(11) dat read(11,end=1011) dat print *, 'failed' 1011 continue backspace 11 backspace 11 end $ gfc back3.f $ ./a.out At line 11 of file back3.f Fortran runtime error: Read past ENDFILE record $ ifc back3.f $ ./a.out $ gfortran and ifort give different results with those two lines commented out. The other cases work fine.
Dale, do you have copyright assignment and/or commit priviliges? Also, I have gone to comp.lang.fortran for an interpretation on my comment #28. I see what you are saying and I see what ifort is doing. Ifort is usually right, but not always, so we will se what others say. BTW Good work.
Based on feedback on comp.lang.fortran, the code in #28 is illegal and so the behavior is acceptable and probably appropriate. Based on this I believe we have a working solution to this bug. We just need to get it committed. Thanks Dale.
Hi Jerry, Would you go ahead and commit this ? A test case something like that in Comment #9 shows the problem before and the fix after or may be you have a better one from NIST. The change suggest in comment #14 fixes the problem that started all of this for me. There are other bugs, that I have hit looking into this, but I think that they should be in seperate bug reports.
Yes, I will work this up with a proper test case and submit for approval.
Subject: Bug 25139 Author: jvdelisle Date: Wed Dec 28 20:58:08 2005 New Revision: 109122 URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=109122 Log: 2005-12-28 Jerry DeLisle <jvdelisle@gcc.gnu.org> PR libgfortran/25139 * io/unix.c (fd_truncate): Set s->active to zero. PR libgfortran/25510 * libgfortran.h: Add ERROR_INTERNAL and ERROR_INTERNAL_UNIT. * runtime/error.c (translate_error): Add messages for new errors. * io/list_read.c (next_char): Use new errors. * io/transfer.c (next_record_r) (next_record_w): Use new errors. Modified: trunk/libgfortran/ChangeLog trunk/libgfortran/io/list_read.c trunk/libgfortran/io/transfer.c trunk/libgfortran/io/unix.c trunk/libgfortran/libgfortran.h trunk/libgfortran/runtime/error.c
Subject: Bug 25139 Author: jvdelisle Date: Wed Dec 28 21:12:52 2005 New Revision: 109123 URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=109123 Log: 2005-12-28 Jerry DeLisle <jvdelisle@gcc.gnu.org> PR libgfortran/25139 * gfortran.dg/backspace_2.f: New test. Added: trunk/gcc/testsuite/gfortran.dg/backspace_2.f Modified: trunk/gcc/testsuite/ChangeLog
Subject: Bug 25139 Author: jvdelisle Date: Sun Jan 1 03:49:00 2006 New Revision: 109212 URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=109212 Log: 2005-12-31 Jerry DeLisle <jvdelisle@gcc.gnu.org> PR libgfortran/25139 * io/unix.c (fd_truncate): Set s->active to zero. PR libgfortran/25510 * libgfortran.h: Add ERROR_INTERNAL and ERROR_INTERNAL_UNIT. * runtime/error.c (translate_error): Add messages for new errors. * io/list_read.c (next_char): Use new errors. * io/transfer.c (next_record_r) (next_record_w): Use new errors. Modified: branches/gcc-4_1-branch/libgfortran/ChangeLog branches/gcc-4_1-branch/libgfortran/io/list_read.c branches/gcc-4_1-branch/libgfortran/io/transfer.c branches/gcc-4_1-branch/libgfortran/io/unix.c branches/gcc-4_1-branch/libgfortran/libgfortran.h branches/gcc-4_1-branch/libgfortran/runtime/error.c
Subject: Bug 25139 Author: jvdelisle Date: Sun Jan 1 03:53:12 2006 New Revision: 109213 URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=109213 Log: 2005-12-31 Jerry DeLisle <jvdelisle@gcc.gnu.org> PR libgfortran/25139 * gfortran.dg/backspace_2.f: New test. Added: branches/gcc-4_1-branch/gcc/testsuite/gfortran.dg/backspace_2.f Modified: branches/gcc-4_1-branch/gcc/testsuite/ChangeLog
Fixed on 4.1 and 4.2