Re: [patch, libgfortran] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[Paul Richard Thomas <paul dot richard dot thomas at gmail dot com>]

Dear Jerry,

This is OK for trunk. Maybe it is best to leave it for a week or two
before committing to the branches?

Thanks for the patch.

Paul

On 7 February 2015 at 03:08, Jerry DeLisle <jvdelisle@charter.net> wrote:
> With the attached patch I create a special version of fbuf_flush that is
> only called with list directed I/O.  There can be no tabbing back and forth
> so it is safe to flush the buffer whenever we want.  The bug occurs when the
> buffer keeps growing to no end until no more allocations can be made and the
> OS gives an error.
>
> fbuf_flush can be called as much as one wants to, but to keep overhead low,
> I introduce a new version with an arbitrary limit.  If below the limit, just
> return doing no flushing.  When the limit is exceeded, fbuf is flushed.  The
> code to do this is duplicated from the original fbuf_flush so it is very
> safe and well tested.
>
> I played with the allowable maximum position limit for flushing and ran some
> timing tests.  My machine here uses a solid state drive which may bias the
> results somewhat.  Others are welcome to test and see what values they get
> as well.  I settled on 524588 based on these results, favoring writing over
> reading a little.
>
> The patch has zero impact on any other type of I/O including normal
> formatted I/O.  I also tested to confirm that formatted I/O does not have
> the problem.  It is flushed regularly in the main transfer loop.  As far as
> I can tell only list directed has the issue.
>
> I get the following timings using the attached test program.
>
> WRITING:
>  Array Size-->  1000000         100000000
>  Buf Limit
>
>   16384         2.107           210.9
>   32768         2.026           292.1
>   65636         2.232           235.8
>   524288        1.958           193.5
>  1048576        2.023           203.5
>
>
> READING:
>  Buf Limit
>
>   16384         1.843           184.4
>   32768         1.841           186.8
>   65636         1.816           197.6
>   524288        1.879           186.5
>  1048576        1.834           185.2
>
> Regression tested on x86-64 Linux.
>
> OK for trunk followed by backports?
>
> I can not include a specific testsuite test case, it would take way too long
> to run.
>
> Regards,
>
> Jerry
>
> 2015-02-07  Jerry DeLisle  <jvdelisle@gcc.gnu.org>
>
>         PR libgfortran/60956
>         * io/fbuf.c (fbuf_flush_list): New function that only flushes
>         if current fbuf position exceeds a limit.
>         * io/fbuf.h: Declare the new function.
>         * io/io.h (enum unit_mode): Add two new modes.
>         * io/list_read.c (list_formatted_read_scalar): Call new function.
>         * io/write.c: Include fbuf.h. (list_formatted_write_scalar):
>         Call new function.



-- 
Outside of a dog, a book is a man's best friend. Inside of a dog it's
too dark to read.

Groucho Marx