Bug 30617 - Implement a run time diagnostic for invalid recursive I/O
Implement a run time diagnostic for invalid recursive I/O
Status: REOPENED
Product: gcc
Classification: Unclassified
Component: libfortran
4.3.0
: P3 enhancement
: ---
Assigned To: Not yet assigned to anyone
: diagnostic
: 51286 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2007-01-27 19:47 UTC by Dominique d'Humieres
Modified: 2013-02-27 18:18 UTC (History)
6 users (show)

See Also:
Host:
Target:
Build:
Known to work:
Known to fail:
Last reconfirmed: 2007-03-03 13:31:17


Attachments
assembly generated with -frename-registers (12.49 KB, text/plain)
2009-12-14 20:43 UTC, Dominique d'Humieres
Details
assembly generated without -frename-registers (12.51 KB, text/plain)
2009-12-14 20:45 UTC, Dominique d'Humieres
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dominique d'Humieres 2007-01-27 19:47:33 UTC
The executable of the following code:

external fun
real fun
real a
a = fun()
print *, a
print *, fun()
end
real function fun()
print *, 'test'
fun = 1.0
end

compiled with 4.3.0 20070126, hangs. -fdump-tree-original gives:

MAIN__ ()
{
  real4 a;

  _gfortran_set_std (70, 127, 0, 0);
  a = fun ();
  {
    struct __st_parameter_dt dt_parm.0;

    dt_parm.0.common.filename = "fun_external.f90";
    dt_parm.0.common.line = 5;
    dt_parm.0.common.unit = 6;
    dt_parm.0.common.flags = 128;
    _gfortran_st_write (&dt_parm.0);
    _gfortran_transfer_real (&dt_parm.0, &a, 4);
    _gfortran_st_write_done (&dt_parm.0);
  }
  {
    struct __st_parameter_dt dt_parm.1;

    dt_parm.1.common.filename = "fun_external.f90";
    dt_parm.1.common.line = 6;
    dt_parm.1.common.unit = 6;
    dt_parm.1.common.flags = 128;
    _gfortran_st_write (&dt_parm.1);
    {
      real4 D.951;

      D.951 = fun ();
      _gfortran_transfer_real (&dt_parm.1, &D.951, 4);
    }
    _gfortran_st_write_done (&dt_parm.1);
  }
}


fun ()
{
  real4 __result_fun;

  {
    struct __st_parameter_dt dt_parm.2;

    dt_parm.2.common.filename = "fun_external.f90";
    dt_parm.2.common.line = 9;
    dt_parm.2.common.unit = 6;
    dt_parm.2.common.flags = 128;
    _gfortran_st_write (&dt_parm.2);
    _gfortran_transfer_character (&dt_parm.2, "test", 4);
    _gfortran_st_write_done (&dt_parm.2);
  }
  __result_fun = 1.0e+0;
  return __result_fun;
}

and if run under gdb, after ^C, where gives:

#0  0x90017238 in semaphore_wait_signal_trap ()
#1  0x90001d90 in pthread_mutex_lock ()
#2  0x0020093c in get_external_unit (n=6, do_create=3331) at ../../../gcc-4.3-20070127/libgfortran/../gcc/gthr-posix.h:604
#3  0x001ff6e0 in data_transfer_init (dtp=0x6004d8, read_flag=3331) at ../../../gcc-4.3-20070127/libgfortran/io/transfer.c:1698
#4  0x00002b48 in fun_ () at fun_external.f90:9
#5  0x00002abc in MAIN__ () at fun_external.f90:6
#6  0x00002bac in main (argc=14, argv=0xd03) at ../../../gcc-4.3-20070127/libgfortran/fmain.c:18

Note that I see the same problem on OSX 10.4 with gcc version 4.2.0 20060617.

Any idea around?
Comment 1 kargl 2007-01-27 21:31:31 UTC
I believe recursive IO is undefined (except maybe in some very limited
situations), but I can't locate anything in the F95 Standard that says
this. :(
Comment 2 kargl 2007-01-27 21:38:38 UTC
(In reply to comment #1)
> I believe recursive IO is undefined (except maybe in some very limited
> situations), but I can't locate anything in the F95 Standard that says
> this. :(
> 

Okay I found the relevant passage.

9.7   Restrictions on function references and list items

A function reference shall not appear in an expression anywhere in an input/output statement if such a reference causes another input/output
statement to be executed.

I believe your "print *, fun()" is nonconforming.

Note F2003 changes the restrictions of recursive IO.  It may take a 
long time before gfortran supports this.
Comment 3 Dominique d'Humieres 2007-01-27 21:43:05 UTC
> I believe recursive IO is undefined

Probably, but the same code works with 

Target: x86_64-unknown-linux-gnu
gcc version 4.3.0 20061231 (experimental)

on a linux AMD64.
Comment 4 Andrew Pinski 2007-01-27 21:47:51 UTC
This might be a bug in Mac OS X's libraries.
Comment 5 Jerry DeLisle 2007-01-27 21:51:29 UTC
I recall some time ago working on the patch that enabled this feature.

I agree this is probably platform specific.
Comment 6 Dominique d'Humieres 2007-01-27 22:01:32 UTC
Subject: Re:  recursive I/O hangs under OSX 10.3

> I agree this is probably platform specific.

Can someone do the test on a Macintel? TIA

Comment 7 kargl 2007-01-27 23:34:14 UTC
(In reply to comment #3)
> > I believe recursive IO is undefined
> 
> Probably, but the same code works with 
> 
> Target: x86_64-unknown-linux-gnu
> gcc version 4.3.0 20061231 (experimental)
> 

Undefined means undefined.  It might do what
you want on one platform and something entirely
different on another.

If I read the F2003 standrad correctly, then 
your program conforms to F2003.  You may want
to change this to an enhancement request.
Comment 8 Dale Ranta 2007-01-29 13:45:57 UTC
It hangs on a Intel iMac -

[pactech01:~/tests] dir% gfortran -o recursiveio recursiveio.f90
[pactech01:~/tests] dir% recursiveio
 test
   1.000000    


^C
[pactech01:~/tests] dir% cat recursiveio.f90
      external fun
      real fun
      real a
      a = fun()
      print *, a
      print *, fun()
      end
      real function fun()
      print *, 'test'
      fun = 1.0
      end
[dranta:~] dir% 


Comment 9 Dominique d'Humieres 2007-01-29 14:13:08 UTC
Subject: Re:  recursive I/O hangs under OSX 10.3

> It hangs on a Intel iMac

Thanks for the answer. So it hangs for the different flavor of OSX, but not
for x86_64-unknown-linux-gnu (4.3.0 20061231) and
i386-pc-linux-gnu (4.3.0 20070129). What about the other supported OS and
architecture?
Comment 10 Francois-Xavier Coudert 2007-02-05 22:15:24 UTC
When the program hangs, we're in

#0  0x9002bbc8 in semaphore_wait_signal_trap ()
#1  0x90001410 in pthread_mutex_lock ()
#2  0x0025595c in get_external_unit (n=6, do_create=275) at ../../../gcc/libgfortran/../gcc/gthr-posix.h:604
#3  0x00254164 in data_transfer_init (dtp=0xbffff5cc, read_flag=275) at ../../../gcc/libgfortran/io/transfer.c:1698
#4  0x00002bb4 in fun_ () at a.f90:9
#5  0x00002b28 in MAIN__ () at a.f90:6
#6  0x00002c18 in main (argc=14, argv=0x113) at ../../../gcc/libgfortran/fmain.c:18

I don't know how all this works, maybe Jakub could give us some help here?
Comment 11 Jerry DeLisle 2007-02-06 03:04:05 UTC
See the commentary near the top of io/unit.c .  This is a basic explanation.
Comment 12 Brooks Moses 2007-02-07 09:10:29 UTC
(In reply to comment #7)
> If I read the F2003 standrad correctly, then 
> your program conforms to F2003.  You may want
> to change this to an enhancement request.

This is incorrect -- the code does not conform to F2003 either.  Section 9.11, paragraph 3, says that a recursive I/O statement -- that is, one that occurs while another is in progress, such as the "print *, 'test'" in this code --  may not identify an external unit, unless it is a "child I/O statement".  Section 9.5.3.7.1, paragarph 2, defines a "child I/O statement" as one that's occuring within a user-defined derived-type I/O function -- which is definitely not applicable here.

Therefore, I believe that this bug should be closed as INVALID, but given the amount of commentary (and the fact that hanging is an unfortunate response even to invalid code), I'll let someone else make that call.
Comment 13 Dominique d'Humieres 2007-02-07 09:24:32 UTC
Subject: Re:  recursive I/O hangs under OSX 10.3

> Section 9.5.3.7.1, paragarph 2, defines a "child I/O statement" as one that's
> occuring within a user-defined derived-type I/O function -- which is definitely
> not applicable here.

Could you please translate this in plain English?

I have looked at the standard draft and have been unable to figure out
a single conforming example, not speaking about a single useful one.
If someone can give me such example(s), I am ready to test it (them)
on both OSX and Linux.

From the above quotation, am I correct to infer that replacing * by 6
will not make the code standard conforming?

TIA
Comment 14 Dominique d'Humieres 2007-02-07 19:55:42 UTC
The test is known to fail on OSX 10.3 (gcc 4.3.0 20070202) and 10.4 (PPC with gcc 4.2.0 20070124
and MacIntel gcc unknown).

When I have filled the PR I have forgotten to say that the same bug was present in

http://gcc.gnu.org/viewcvs/trunk/gcc/testsuite/gfortran.dg/actual_array_interface_1.f90

see http://gcc.gnu.org/ml/fortran/2006-11/msg00394.html and following mails in the thread
for details.

Note that the slightly modified code:

external fun
real fun
character(10) :: string
real a
a = fun()
print *, a
write(string,*) fun()
print *, string
end
real function fun()
print *, 'test'
fun = 1.0
end

gives

test
  1.000000    
test
At line 7 of file recursive_io.f90
Fortran runtime error: End of record

on both OSX and Linux.
Comment 15 Jerry DeLisle 2007-02-08 03:12:24 UTC
Try :

external fun
real fun
character(10) :: string
real a
a = fun()
print *, a,a
write(string,'(f10.6)') fun()
print *, string
end
real function fun()
print *, 'test'
fun = 1.0
end

Or increase the size of string.
Comment 16 Dominique d'Humieres 2007-02-08 08:45:56 UTC
Subject: Re:  recursive I/O hangs under OSX

> Try :
> ...
> Or increase the size of string.

I have increased the length to 20 and in both cases the executable
(for an invalid code!-) works on OSX.

I have also tried:

external fun
character(20) :: fun
i=2
print trim(fun(i)), 99
end
function fun(i)
character(20) :: fun
integer :: i
write(fun,'(A2,I0,A)') "(I", I, ")"
end

which works. Is this last code valid for f95 (-std=f95 does give any diagnostic)
and/or f2003?

Anyway the hanging seems due to the recursive access to external I/O.
If I want to corner the problem, where should I look?  With directions I can
try to locate the problem and test possible fixes.
Comment 17 Jakub Jelinek 2007-02-08 09:36:13 UTC
When writing the unit locking code, I relied on Fortran95 requiring that
the same unit isn't used recursively, you can use different units, but not the
same.
gfortran.dg/intrinsic_actual_2.f90 fails on Linux too when linked with
-fopenmp (or -lpthread, -fopenmp implies that), but that is fine, the testcase
is IMHO not valid Fortran95 source.

Now, if Fortran2003 allows some recursive access to the same unit (under which conditions?), then the gfc_unit's lock could very well be changed into
__gthread_recursive_mutex_t as well as the initializer, perhaps together with
adding a recursion count (as the recursion count in __gthread_recursive_mutex_t
isn't portably accessible) to gfc_unit as well.  The question is how much is libgfortran prepared for changes happening to gfc_unit happening between
say st_write and st_write_done calls (st_write normally calls get_unit and thus
acquires gfc_unit's lock, which is held until st_write_done calls unlock_unit).

For Fortran95, if a hang is in your opinion not a good implementation of undefined behavior when running a buggy program, if recursive mutex is used
for gfc_unit's lock and a recursion counter is present, then anyone calling
gfc_unit could then check the recursion count and if it is bigger than 1,
unlock_unit it again and report an error.
Comment 18 Dominique d'Humieres 2007-02-08 10:03:29 UTC
Subject: Re:  recursive I/O hangs under OSX

> gfortran.dg/intrinsic_actual_2.f90 fails on Linux too when linked with
> -fopenmp (or -lpthread, -fopenmp implies that), but that is fine, the testcase
> is IMHO not valid Fortran95 source.

When you say "fails" what kind of failure do you get? Is this a timeout or not?
If this is a runtime error, a "seg fault", or whatever but hanging, this is fine
with me.

> Now, if Fortran2003 allows some recursive access to the same unit (under which
> conditions?)

I think the question is settled: recursive access to external units is invalid.

> then anyone calling gfc_unit could then check the recursion count and if it is bigger than 1,
> unlock_unit it again and report an error.

This seems the path to follow to implement a runtime error, isn't it?
To be frank, I have a very limited understanding of what you are saying, but
I could look again to the code to figure out what it looks like: I understand
more or less the technical concepts, but not the actual implementation.

Thanks for your time.
Comment 19 Dominique d'Humieres 2007-02-08 12:06:56 UTC
The following invalid code:

external fun
real fun
real a
a = fun()
! print *, a
write(10,*) fun(), a, 'try it', ' 1.23'
end
real function fun()
print *, 'test'
fun = 1.0
end

gives a working executable when compiled with gfortran on OSX.
However, if I replace "print *," by "write(10,*)" the executable hangs again.
So the problem seems limited to use of the same external unit.
Comment 20 kargl 2007-02-08 16:18:11 UTC
(In reply to comment #17)
> 
> Now, if Fortran2003 allows some recursive access to the same unit
> (under which conditions?),

Recursive IO to external units is simply not allowed by the Fortran
95 and Fortran 2003 standards.  You could wade through Brooks
posts in fortran@ to see where recursion is allowed.


> __gthread_recursive_mutex_t as well as the initializer, perhaps together with
> adding a recursion count (as the recursion count in __gthread_recursive_mutex_t
> isn't portably accessible) to gfc_unit as well.  The question is how much is
> libgfortran prepared for changes happening to gfc_unit happening between
> say st_write and st_write_done calls (st_write normally calls get_unit and thus
> acquires gfc_unit's lock, which is held until st_write_done calls unlock_unit).
> 

This sound like a performance penalty imposed on anyone writing standard
conforming code to permit nonconforming code to run.  The fact 
remains that a fortran processor can do anything it wants with nonconforming
code.  It might "work" on some architecture and it may "fail" another for
some definition of "work" and "fail".  So be it.  
Comment 21 Paul Thomas 2007-02-11 20:59:09 UTC
Subject: Bug 30617

Author: pault
Date: Sun Feb 11 20:58:48 2007
New Revision: 121824

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=121824
Log:
2007-02-11  Paul Thomas  <pault@gcc.gnu.org>

	PR fortran/30554
	* module.c (find_symtree_for_symbol): New function to return
	a symtree that is not a "unique symtree" given a symbol.
	(read_module): Do not automatically set pointer_info to
	referenced because this inhibits the generation of a unique
	symtree.  Recycle the existing symtree if possible by calling
	find_symtree_for_symbol.

	PR fortran/30319
	* decl.c (add_init_expr_to_sym): Make new charlen for an array
	constructor initializer.

2007-02-11  Paul Thomas  <pault@gcc.gnu.org>

	PR fortran/30554
	* gfortran.dg/used_dummy_types_6.f90: Add the "privatized"
	versions of the modules.

	PR fortran/30617
	* gfortran.dg/intrinsic_actual_2.f90: Make this legal fortran
	by getting rid of recursive I/O and providing functions with
	results.

	PR fortran/30319
	* gfortran.dg/char_array_constructor_2.f90

Added:
    trunk/gcc/testsuite/gfortran.dg/char_array_constructor_2.f90
Modified:
    trunk/gcc/fortran/ChangeLog
    trunk/gcc/fortran/decl.c
    trunk/gcc/fortran/module.c
    trunk/gcc/testsuite/ChangeLog
    trunk/gcc/testsuite/gfortran.dg/intrinsic_actual_2.f90
    trunk/gcc/testsuite/gfortran.dg/used_dummy_types_6.f90

Comment 22 Dominique d'Humieres 2007-02-12 07:37:12 UTC
Subject: Re:  recursive I/O hangs under OSX

> PR fortran/30319

Thanks Paul ;-)
Comment 23 Paul Thomas 2007-02-17 17:17:11 UTC
Subject: Bug 30617

Author: pault
Date: Sat Feb 17 17:16:56 2007
New Revision: 122074

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=122074
Log:
2007-02-17  Paul Thomas  <pault@gcc.gnu.org>

	PR fortran/30554
	* module.c (find_symtree_for_symbol): New function to return
	a symtree that is not a "unique symtree" given a symbol.
	(read_module): Do not automatically set pointer_info to
	referenced, if the symbol has a namespace, because this
	inhibits the generation of a unique symtree.  Recycle the
	symtree if possible by calling find_symtree_for_symbol. If a
	symbol is excluded by an ONLY clause, check to see if there is
	a symtree already loaded. If so, attach the symtree to the
	pointer_info.

	PR fortran/30319
	* decl.c (add_init_expr_to_sym): Make new charlen for an array
	constructor initializer.

2007-02-17  Paul Thomas  <pault@gcc.gnu.org>

	PR fortran/30554
	* gfortran.dg/used_dummy_types_6.f90: New test.
	* gfortran.dg/used_dummy_types_7.f90: New test..

	PR fortran/30617
	* gfortran.dg/intrinsic_actual_2.f90: Make this legal fortran
	by getting rid of recursive I/O and providing functions with
	results.

	PR fortran/30319
	* gfortran.dg/char_array_constructor_2.f90



Added:
    branches/gcc-4_2-branch/gcc/testsuite/gfortran.dg/char_array_constructor_2.f90
    branches/gcc-4_2-branch/gcc/testsuite/gfortran.dg/used_dummy_types_6.f90
    branches/gcc-4_2-branch/gcc/testsuite/gfortran.dg/used_dummy_types_7.f90
Modified:
    branches/gcc-4_2-branch/gcc/fortran/ChangeLog
    branches/gcc-4_2-branch/gcc/fortran/decl.c
    branches/gcc-4_2-branch/gcc/fortran/module.c
    branches/gcc-4_2-branch/gcc/testsuite/ChangeLog
    branches/gcc-4_2-branch/gcc/testsuite/gfortran.dg/intrinsic_actual_2.f90

Comment 24 Tobias Burnus 2007-03-03 20:55:59 UTC
Is this actually now fixed or not? I see a 4.2 and a trunk commit. Can this bug now be closed, is something missing or should it be checked in for 4.1?
Comment 25 Dominique d'Humieres 2007-03-03 21:46:32 UTC
> Is this actually now fixed or not?

No, it is not. The commits are for the side effect of test case intrinsic_actual_2.f90 that has
been fixed.
Comment 26 Jakub Jelinek 2007-04-24 11:59:41 UTC
Please define "fixed".  You run an invalid program, all you need is change
your expectations what will happen.
It is the same as if you write
pthread_mutex_t m;
pthread_mutex_init (&m, NULL);
pthread_mutex_lock (&m);
pthread_mutex_lock (&m);
The deadlock might be detected or might not, at which point it would hang.
Comment 27 Dominique d'Humieres 2007-04-24 13:03:56 UTC
> Please define "fixed".

For me "fixed"=="no hanging" period.

> You run an invalid program, 

Yes, I know, repeating that won't help!

> all you need is change your expectations what will happen.

Bugzilla has plenty of entries "ICE on invalid", I am just asking that "hanging on invalid" is not brushed aside just because it does happen on a platform some people don't like.

> It is the same as if you write
> pthread_mutex_t m;
> pthread_mutex_init (&m, NULL);
> pthread_mutex_lock (&m);
> pthread_mutex_lock (&m);
> The deadlock might be detected or might not, at which point it would hang.

As far as I can tell, without knowledge of the 'pthread_mutex' mechanism, the problem does not occur at this level, but at the level of:

pthread_mutex_unlock (&m)
pthread_mutex_unlock (&m)

where the first 'unlock' releases all the previous locks dealing with the same unit number, leaving the second 'unlock' "waiting for something that is not coming".  Something like:

pthread_mutex_init (&m, NULL);
pthread_mutex_lock (&m);
pthread_mutex_init (&m1, NULL);
pthread_mutex_lock (&m1);
...
pthread_mutex_unlock (&m1)
pthread_mutex_unlock (&m)

is working (see comment #19), so probably if &m refers to some "event" and not to some "unit number" (not being ashamed to show my ignorance) the problem would disappear.

I think the following answer from

http://groups.google.com/group/comp.lang.fortran/browse_thread/thread/d3caadeb9083808e/78c9e65c8b4f930e?lnk=gst&q=recursive&rnum=11#78c9e65c8b4f930e

summarize the discussion

Even if this restriction on recursive input/output is
unnecessary or undesireable, it is, nonetheless, a part
of the Fortran 2003 standard.

Note the "unnecessary or undesireable", leading me to the following comments:

(1) the use of recursive I/O can be detected at run time: xlf does it;
I guess that a second locking of '&m' would not be too difficult to detect and I don't buy the efficiency argument at this level (gfortran did not strike me as being any faster than xlf on I/O operations);

(2) the behavior, however "processor" (fortran meaning) dependent it is, should be platform independent: just imagine the port to OSX of a 100kloc code where one such recusive I/O used for debuging on Linux (it works) has been left;

(3) I repeat that I am not asking for an immediate fix, only that the problem is not forgotten due to the 'invalid' label.

Anyway, thanks for the answer.

Comment 28 Andrew Pinski 2007-04-24 15:46:34 UTC
> Bugzilla has plenty of entries "ICE on invalid", I am just asking that "hanging
> on invalid" is not brushed aside just because it does happen on a platform some
> people don't like.

 It is hanging on undefined, I think there is a difference here, a big difference.
Comment 29 Dominique d'Humieres 2007-04-24 15:59:32 UTC
> It is hanging on undefined, I think there is a difference here, a big
> difference.

What is the difference for you?

Just as a side note, that's not me who classed the PR as invalid and so far I did not see a class "ICE on undefined".
Comment 30 Dominique d'Humieres 2008-07-11 22:18:59 UTC
From pr36806 I think the mutex behavior on Darwin needs to be better understood (handled).

To answer comment #28, this bug has been "resolved" as "invalid" and not as "undefined"!

Comment 31 Dominique d'Humieres 2008-07-11 22:21:56 UTC
Forgot to say that the problem is still there for ppc/intel Darwin9.

Comment 32 Dominique d'Humieres 2009-05-07 22:00:31 UTC
Presently invalid recursive I/Os are accepted by gfortran without any diagnostic at run time. It could be helpful to implement such a diagnostic with something such as -fcheck=recursive_IO.

Note that xlf gives a run time error for invalid recursive I/Os.
Comment 33 Dominique d'Humieres 2009-12-14 20:43:07 UTC
Created attachment 19296 [details]
assembly generated with -frename-registers
Comment 34 Dominique d'Humieres 2009-12-14 20:45:58 UTC
Created attachment 19297 [details]
assembly generated without -frename-registers
Comment 35 Dominique d'Humieres 2009-12-14 20:51:45 UTC
Wrong pr. Sorry for the noise.
Comment 36 Dominique d'Humieres 2013-02-27 18:18:38 UTC
*** Bug 51286 has been marked as a duplicate of this bug. ***