[patch, libgfortran] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[Jerry DeLisle]

With the attached patch I create a special version of fbuf_flush that is only 
called with list directed I/O.  There can be no tabbing back and forth so it is 
safe to flush the buffer whenever we want.  The bug occurs when the buffer keeps 
growing to no end until no more allocations can be made and the OS gives an error.

fbuf_flush can be called as much as one wants to, but to keep overhead low, I 
introduce a new version with an arbitrary limit.  If below the limit, just 
return doing no flushing.  When the limit is exceeded, fbuf is flushed.  The 
code to do this is duplicated from the original fbuf_flush so it is very safe 
and well tested.

I played with the allowable maximum position limit for flushing and ran some 
timing tests.  My machine here uses a solid state drive which may bias the 
results somewhat.  Others are welcome to test and see what values they get as 
well.  I settled on 524588 based on these results, favoring writing over reading 
a little.

The patch has zero impact on any other type of I/O including normal formatted 
I/O.  I also tested to confirm that formatted I/O does not have the problem.  It 
is flushed regularly in the main transfer loop.  As far as I can tell only list 
directed has the issue.

I get the following timings using the attached test program.

WRITING:
  Array Size-->	1000000		100000000
  Buf Limit

   16384		2.107		210.9
   32768		2.026		292.1
   65636		2.232		235.8
   524288	1.958		193.5
  1048576	2.023		203.5


READING:
  Buf Limit

   16384		1.843		184.4
   32768		1.841		186.8
   65636		1.816		197.6
   524288	1.879		186.5
  1048576	1.834		185.2

Regression tested on x86-64 Linux.

OK for trunk followed by backports?

I can not include a specific testsuite test case, it would take way too long to run.

Regards,

Jerry

2015-02-07  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

	PR libgfortran/60956
	* io/fbuf.c (fbuf_flush_list): New function that only flushes
	if current fbuf position exceeds a limit.
	* io/fbuf.h: Declare the new function.
	* io/io.h (enum unit_mode): Add two new modes.
	* io/list_read.c (list_formatted_read_scalar): Call new function.
	* io/write.c: Include fbuf.h. (list_formatted_write_scalar):
	Call new function.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pr60956.diff
Type: text/x-patch
Size: 2792 bytes
Desc: not available
URL: <http://gcc.gnu.org/pipermail/gcc-patches/attachments/20150207/4e5df304/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_complex_io.f90
Type: text/x-fortran
Size: 1024 bytes
Desc: not available
URL: <http://gcc.gnu.org/pipermail/gcc-patches/attachments/20150207/4e5df304/attachment-0001.bin>