Re: [patch, libgfortran] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

["H.J. Lu" <hjl dot tools at gmail dot com>]

On Sat, Feb 7, 2015 at 6:52 AM, Paul Richard Thomas
<paul.richard.thomas@gmail.com> wrote:
> Dear Jerry,
>
> This is OK for trunk. Maybe it is best to leave it for a week or two
> before committing to the branches?
>
> Thanks for the patch.
>
> Paul
>
> On 7 February 2015 at 03:08, Jerry DeLisle <jvdelisle@charter.net> wrote:
>> With the attached patch I create a special version of fbuf_flush that is
>> only called with list directed I/O.  There can be no tabbing back and forth
>> so it is safe to flush the buffer whenever we want.  The bug occurs when the
>> buffer keeps growing to no end until no more allocations can be made and the
>> OS gives an error.
>>
>> fbuf_flush can be called as much as one wants to, but to keep overhead low,
>> I introduce a new version with an arbitrary limit.  If below the limit, just
>> return doing no flushing.  When the limit is exceeded, fbuf is flushed.  The
>> code to do this is duplicated from the original fbuf_flush so it is very
>> safe and well tested.
>>
>> I played with the allowable maximum position limit for flushing and ran some
>> timing tests.  My machine here uses a solid state drive which may bias the
>> results somewhat.  Others are welcome to test and see what values they get
>> as well.  I settled on 524588 based on these results, favoring writing over
>> reading a little.
>>
>> The patch has zero impact on any other type of I/O including normal
>> formatted I/O.  I also tested to confirm that formatted I/O does not have
>> the problem.  It is flushed regularly in the main transfer loop.  As far as
>> I can tell only list directed has the issue.
>>
>> I get the following timings using the attached test program.
>>
>> WRITING:
>>  Array Size-->  1000000         100000000
>>  Buf Limit
>>
>>   16384         2.107           210.9
>>   32768         2.026           292.1
>>   65636         2.232           235.8
>>   524288        1.958           193.5
>>  1048576        2.023           203.5
>>
>>
>> READING:
>>  Buf Limit
>>
>>   16384         1.843           184.4
>>   32768         1.841           186.8
>>   65636         1.816           197.6
>>   524288        1.879           186.5
>>  1048576        1.834           185.2
>>
>> Regression tested on x86-64 Linux.
>>
>> OK for trunk followed by backports?
>>
>> I can not include a specific testsuite test case, it would take way too long
>> to run.
>>
>> Regards,
>>
>> Jerry
>>
>> 2015-02-07  Jerry DeLisle  <jvdelisle@gcc.gnu.org>
>>
>>         PR libgfortran/60956
>>         * io/fbuf.c (fbuf_flush_list): New function that only flushes
>>         if current fbuf position exceeds a limit.
>>         * io/fbuf.h: Declare the new function.
>>         * io/io.h (enum unit_mode): Add two new modes.
>>         * io/list_read.c (list_formatted_read_scalar): Call new function.
>>         * io/write.c: Include fbuf.h. (list_formatted_write_scalar):
>>         Call new function.
>

I think this caused:

FAIL: gfortran.dg/io_constraints_8.f90  -O   (test for errors, line 64)
FAIL: gfortran.dg/io_constraints_8.f90  -O   (test for errors, line 65)
FAIL: gfortran.dg/io_constraints_8.f90  -O   (test for errors, line 66)
FAIL: gfortran.dg/io_constraints_8.f90  -O   (test for errors, line 67)
FAIL: gfortran.dg/io_constraints_8.f90  -O   (test for errors, line 68)
FAIL: gfortran.dg/io_constraints_8.f90  -O   (test for errors, line 70)
FAIL: gfortran.dg/io_constraints_8.f90  -O   (test for errors, line 71)
FAIL: gfortran.dg/io_constraints_8.f90  -O  (test for excess errors)

on 4.8 branch on Linux/x86-32.

-- 
H.J.