This is the mail archive of the
mailing list for the GCC project.
Re: [patch, libgfortran] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran
- From: Paul Richard Thomas <paul dot richard dot thomas at gmail dot com>
- To: Jerry DeLisle <jvdelisle at charter dot net>
- Cc: gfortran <fortran at gcc dot gnu dot org>, gcc patches <gcc-patches at gcc dot gnu dot org>
- Date: Sat, 7 Feb 2015 15:52:48 +0100
- Subject: Re: [patch, libgfortran] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran
- Authentication-results: sourceware.org; auth=none
- References: <54D57387 dot 9080608 at charter dot net>
This is OK for trunk. Maybe it is best to leave it for a week or two
before committing to the branches?
Thanks for the patch.
On 7 February 2015 at 03:08, Jerry DeLisle <firstname.lastname@example.org> wrote:
> With the attached patch I create a special version of fbuf_flush that is
> only called with list directed I/O. There can be no tabbing back and forth
> so it is safe to flush the buffer whenever we want. The bug occurs when the
> buffer keeps growing to no end until no more allocations can be made and the
> OS gives an error.
> fbuf_flush can be called as much as one wants to, but to keep overhead low,
> I introduce a new version with an arbitrary limit. If below the limit, just
> return doing no flushing. When the limit is exceeded, fbuf is flushed. The
> code to do this is duplicated from the original fbuf_flush so it is very
> safe and well tested.
> I played with the allowable maximum position limit for flushing and ran some
> timing tests. My machine here uses a solid state drive which may bias the
> results somewhat. Others are welcome to test and see what values they get
> as well. I settled on 524588 based on these results, favoring writing over
> reading a little.
> The patch has zero impact on any other type of I/O including normal
> formatted I/O. I also tested to confirm that formatted I/O does not have
> the problem. It is flushed regularly in the main transfer loop. As far as
> I can tell only list directed has the issue.
> I get the following timings using the attached test program.
> Array Size--> 1000000 100000000
> Buf Limit
> 16384 2.107 210.9
> 32768 2.026 292.1
> 65636 2.232 235.8
> 524288 1.958 193.5
> 1048576 2.023 203.5
> Buf Limit
> 16384 1.843 184.4
> 32768 1.841 186.8
> 65636 1.816 197.6
> 524288 1.879 186.5
> 1048576 1.834 185.2
> Regression tested on x86-64 Linux.
> OK for trunk followed by backports?
> I can not include a specific testsuite test case, it would take way too long
> to run.
> 2015-02-07 Jerry DeLisle <email@example.com>
> PR libgfortran/60956
> * io/fbuf.c (fbuf_flush_list): New function that only flushes
> if current fbuf position exceeds a limit.
> * io/fbuf.h: Declare the new function.
> * io/io.h (enum unit_mode): Add two new modes.
> * io/list_read.c (list_formatted_read_scalar): Call new function.
> * io/write.c: Include fbuf.h. (list_formatted_write_scalar):
> Call new function.
Outside of a dog, a book is a man's best friend. Inside of a dog it's
too dark to read.