Bug 20257 - Fortran runtime error: End of record occurs when writing large arrays
Summary: Fortran runtime error: End of record occurs when writing large arrays
Status: RESOLVED FIXED
Alias: None
Product: gcc
Classification: Unclassified
Component: libfortran (show other bugs)
Version: 4.1.0
: P2 minor
Target Milestone: ---
Assignee: Jerry DeLisle
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2005-03-01 02:50 UTC by Kamaraju Kusumanchi
Modified: 2006-04-29 05:47 UTC (History)
2 users (show)

See Also:
Host:
Target:
Build:
Known to work:
Known to fail:
Last reconfirmed: 2006-04-15 07:15:20


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Kamaraju Kusumanchi 2005-03-01 02:50:22 UTC
Consider the following test program (fortran 90)

program explore
 implicit none

 integer, parameter :: NX = 32, NR = 16, NPHI=2

 real :: input(0:NX-1, 0:NR-1, 0:NPHI-1)

 write(*,*) size(input)

 call random_number(input)
 write(*,*) input
end program explore


This program compiles fine. It executes fine. But when the output is redirected
to a file, it produces the following error.

$gfortran explore.f90
$./a.out > temp
At line 11 of file explore.f90
Fortran runtime error: End of record




system & build information:

$gfortran -v
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v
--enable-languages=c,c++,java,f95,objc,ada--prefix=/usr/lib/gcc-snapshot
--enable-shared --with-system-zlib --enable-nls --enable-threads=posix
--without-included-gettext --disable-werror --enable-__cxa_atexit
--enable-libstdcxx-allocator=mt --enable-clocale=gnu
--enable-libstdcxx-debug --enable-mpfr --enable-java-gc=boehm
--enable-java-awt=gtk --enable-gtk-cairo --enable-checking=release
i486-linux-gnu
Thread model: posix
gcc version 4.1.0 20050227 (experimental)

$gcc -v
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --enable-languages=c,c++,java,f95,objc,ada
--prefix=/usr/lib/gcc-snapshot --enable-shared --with-system-zlib --enable-nls
--enable-threads=posix --without-included-gettext --disable-werror
--enable-__cxa_atexit --enable-libstdcxx-allocator=mt --enable-clocale=gnu
--enable-libstdcxx-debug --enable-mpfr --enable-java-gc=boehm
--enable-java-awt=gtk --enable-gtk-cairo --enable-checking=release i486-linux-gnu
Thread model: posix
gcc version 4.1.0 20050227 (experimental)

$uname -a
Linux kusumanchi 2.6.9-1-686 #1 Thu Nov 25 03:48:29 EST 2004 i686 GNU/Linux

Other information:
  System is Debian unstable (sid).
Comment 1 Jerry DeLisle 2005-03-01 03:14:00 UTC
I noticed on a the list mention that the error occurs only when the output is
redirected.  When I run ./a.out the error message is embedded in the middle of
the output before the last numbers are printed.

The error message gets printed out before the stdout buffer finishes.
Comment 2 Francois-Xavier Coudert 2005-03-01 13:10:28 UTC
This has indeed nothing to do with the output being redirected to a file, or
stdout, or even being connected (in Fortran way) to a file:

$ cat pr20257.f90 
 integer, parameter :: NX = 32, NR = 16, NPHI=2
 real :: input(0:NX-1, 0:NR-1, 0:NPHI-1)

 input(:,:,:) = 0
 open (10, file='foo')
 write(10,*) input
end
$ gfortran pr20257.f90 && ./a.out
At line 6 of file pr20257.f90
Fortran runtime error: End of record
Comment 3 Francois-Xavier Coudert 2005-03-02 09:34:50 UTC
It happens for all kind of output (formatted (direct or sequential) or
unformatted), and is due to gfortran setting an arbitrary maximal length of
10000 chars (this is in current_unit->recl). Since this code doesn't specify a
recl, we use the default:

io.h:#define DEFAULT_RECL 10000

I don't see a reason for setting so low a value. I will ask later today on the
gfortran mailing-list about the reasons behind this choice. Hope we can fix this
quickly.
Comment 4 Francois-Xavier Coudert 2005-03-03 10:34:17 UTC
Partial fix proposed here: http://gcc.gnu.org/ml/fortran/2005-03/msg00023.html

This fixes the bug when writing to files, but there still is the case of
preconnected units (stdout, stderr). This will need a little more thinking.

Reduced test case for preconnected units:

 real :: array(10000)
 array(:) = 0
 write(*,*) array
end
Comment 5 Francois-Xavier Coudert 2005-03-03 13:08:33 UTC
Complete patch is here: http://gcc.gnu.org/ml/fortran/2005-03/msg00024.html

Note: this just makes the limit higher (1 Gb), but there still is a limit on the
size of things you can write to preconnected units.
Comment 6 Jerry DeLisle 2006-01-25 07:30:45 UTC
I don't see this problem any more.  Was this fixed?  Is the default record length an issue any more?  Can we close this PR?  I wonder if the fix to P25835 fixed this?  If so I propose we close this unless it is related to the new PR Janne is working on, large records with little writes, 25949.
Comment 7 Francois-Xavier Coudert 2006-01-25 09:32:19 UTC
(In reply to comment #6)
> I don't see this problem any more.  Was this fixed?  Is the default record
> length an issue any more?  Can we close this PR?  I wonder if the fix to P25835
> fixed this?  If so I propose we close this unless it is related to the new PR
> Janne is working on, large records with little writes, 25949.

As I said in comment #5, the limit was just pushed higher, but big writes on preconnected units still fail:

$ gfortran a.f90 && ./a.out > bigfile
At line 3 of file a.f90
Fortran runtime error: End of record
$ cat a.f90
 real :: array(100000000)
 array(:) = 0
 write(*,*) array
end
$ ll bigfile 
-rw-r--r--  1 coudert users 1.0G Jan 25 10:28 bigfile
$ gfortran -v
Using built-in specs.
Target: i386-linux
Configured with: ../gcc/configure --prefix=/tmp/gfortran-20060124/irun --enable-languages=c,fortran --host=i386-linux --with-gmp=/tmp/gfortran-20060124/gfortran_libs
Thread model: posix
gcc version 4.2.0 20060124 (experimental)


You can look at the patch cited in comment #5 for where the problem lies. And this is not really related to Janne's PR, I guess. It's specific to preconnected units.
Comment 8 Jerry DeLisle 2006-04-15 07:15:20 UTC
I have the beginnings of a patch going.  It covers formatted write to stdout.  Seems to work OK.  I just need to extend the concept to the other cases.
Comment 9 patchapp@dberlin.org 2006-04-20 23:20:15 UTC
Subject: Bug number PR20257

A patch for this bug has been added to the patch tracker.
The mailing list url for the patch is http://gcc.gnu.org/ml/gcc-patches/2006-04/msg00785.html
Comment 10 Jerry DeLisle 2006-04-23 02:05:07 UTC
Subject: Bug 20257

Author: jvdelisle
Date: Sun Apr 23 02:04:58 2006
New Revision: 113190

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=113190
Log:
2006-04-22  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

	PR libgfortran/20257
	* io/io.h: Add prototypes for get_internal_unit and free_internal_unit.
	* io/unit.c (get_internal_unit): Initialize unit number, not zero.
	(free_internal_unit): New function to consolidate freeing memory.
	(get_unit): Initialize internal_unit_desc to NULL when unit is
	external.
	* io/unix.c (mem_close): Check for not NULL before freeing memory.
	* io/transfer.c (read_block): Reset bytes_left and skip error if unit
	is preconnected and default record length is reached.
	(read_block_direct): Ditto.
	(write_block): Ditto.
	(write_buf): Ditto.
	(data_transfer_init): Only flush if not internal unit.
	(finalize_transfer): Ditto and delete code to free memory used by
	internal units.
	(st_read_done): Use new function - free_internal_unit.
	(st_write_done): Use new function - free_internal unit.

Modified:
    trunk/libgfortran/ChangeLog
    trunk/libgfortran/io/io.h
    trunk/libgfortran/io/transfer.c
    trunk/libgfortran/io/unit.c
    trunk/libgfortran/io/unix.c

Comment 11 Jerry DeLisle 2006-04-29 04:27:15 UTC
Subject: Bug 20257

Author: jvdelisle
Date: Sat Apr 29 04:27:09 2006
New Revision: 113373

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=113373
Log:
2006-04-28  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

	PR libgfortran/20257
	* io/io.h: Add prototypes for get_internal_unit and free_internal_unit.
	* io/unit.c (get_internal_unit): Initialize unit number, not zero.
	(free_internal_unit): New function to consolidate freeing memory.
	(get_unit): Initialize internal_unit_desc to NULL when unit is
	external.
	* io/unix.c (mem_close): Check for not NULL before freeing memory.
	* io/transfer.c (read_block): Reset bytes_left and skip error if unit
	is preconnected and default record length is reached.
	(read_block_direct): Ditto.
	(write_block): Ditto.
	(write_buf): Ditto.
	(data_transfer_init): Only flush if not internal unit.
	(finalize_transfer): Ditto and delete code to free memory used by
	internal units.
	(st_read_done): Use new function - free_internal_unit.
	(st_write_done): Use new function - free_internal unit.

Modified:
    branches/gcc-4_1-branch/libgfortran/ChangeLog
    branches/gcc-4_1-branch/libgfortran/io/io.h
    branches/gcc-4_1-branch/libgfortran/io/transfer.c
    branches/gcc-4_1-branch/libgfortran/io/unit.c
    branches/gcc-4_1-branch/libgfortran/io/unix.c

Comment 12 Jerry DeLisle 2006-04-29 05:47:23 UTC
Fixed on Trunk and now 4.1 branch