For writing, I think this fails because we

- write a bogus record marker with value 0
- write out the data
- write out the trailing record marker
- seek to the first record marker
- write out its value
- seek past the end

These seeks actually translate to OS calls, which of course fails
for pipes.

For reading, there is another problem:  If I

- redirect the output from a to a file
- cat it into the named pipe
- read it using b

it gives a bogus "corrupt unformatted sequential file" error,
for a reason that I don't understand at the moment.

Ouch.  This one is nasty.

(In reply to comment #1)

> These seeks actually translate to OS calls, which of course fails
> for pipes.

We could get by for writing if we

- eliminated the seeks as long as we don't actually go past our buffer
- flushed after every write statement if we have a non-seekable file
- set the subrecord size to the buffer length - 2*length of the record marker.

This is deep regression country; in the time
I have to devote to this, I couldn't work it out.

Unassigning myself (for now).

Additional data points:

Reading/writing unformatted sequential doesn't work with g77
(fails with an error message on open), and it fails with ifort 8
for records longer than 2**18 bytes (ifort's default buffer size).

Because of the complications with record markers, this might be
much easier to implement for stream access.

 

I will work at it.

(In reply to comment #5)
> I will work at it.

Thanks, I'll be happy to assist with discussions and review.

(Those who can, fix; those who can't, review :-)

I have formatted named pipe I/O working, at least for the equivalent test cases given here.

(In reply to comment #7)
> I have formatted named pipe I/O working, at least for the equivalent test cases
> given here.

Great!

If you want me to, I'll be willing to test your patch.

Thomas

Preliminary patch for formatted only.

Index: io/unix.c
===================================================================
*** io/unix.c	(revision 120301)
--- io/unix.c	(working copy)
*************** fd_flush (unix_stream * s)
*** 349,355 ****
    size_t writelen;
  
    if (s->ndirty == 0)
!     return SUCCESS;;
  
    if (s->physical_offset != s->dirty_offset &&
        lseek (s->fd, s->dirty_offset, SEEK_SET) < 0)
--- 349,358 ----
    size_t writelen;
  
    if (s->ndirty == 0)
!     return SUCCESS;
!   
!   if (s->file_length == -1)
!     return SUCCESS;
  
    if (s->physical_offset != s->dirty_offset &&
        lseek (s->fd, s->dirty_offset, SEEK_SET) < 0)
*************** fd_sfree (unix_stream * s)
*** 562,567 ****
--- 565,574 ----
  static try
  fd_seek (unix_stream * s, gfc_offset offset)
  {
+ 
+   if (s->file_length == -1)
+     return SUCCESS;
+ 
    if (s->physical_offset == offset) /* Are we lucky and avoid syscall?  */
      {
        s->logical_offset = offset;
*************** static try
*** 583,589 ****
  fd_truncate (unix_stream * s)
  {
    if (lseek (s->fd, s->logical_offset, SEEK_SET) == -1)
!     return FAILURE;
  
    /* non-seekable files, like terminals and fifo's fail the lseek.
       Using ftruncate on a seekable special file (like /dev/null)
--- 590,596 ----
  fd_truncate (unix_stream * s)
  {
    if (lseek (s->fd, s->logical_offset, SEEK_SET) == -1)
!     return SUCCESS;
  
    /* non-seekable files, like terminals and fifo's fail the lseek.
       Using ftruncate on a seekable special file (like /dev/null)
*************** fd_to_stream (int fd, int prot)
*** 1009,1015 ****
    /* Get the current length of the file. */
  
    fstat (fd, &statbuf);
!   s->file_length = S_ISREG (statbuf.st_mode) ? statbuf.st_size : -1;
    s->special_file = !S_ISREG (statbuf.st_mode);
  
    fd_open (s);
--- 1016,1027 ----
    /* Get the current length of the file. */
  
    fstat (fd, &statbuf);
! 
!   if (lseek (fd, 0, SEEK_CUR) == (off_t) -1)
!     s->file_length = -1;
!   else
!     s->file_length = S_ISREG (statbuf.st_mode) ? statbuf.st_size : -1;
! 
    s->special_file = !S_ISREG (statbuf.st_mode);
  
    fd_open (s);

(In reply to comment #9)

> Preliminary patch for formatted only.

Looks pretty good, at least the tests pass.


> Index: io/unix.c
> ===================================================================
> *** io/unix.c   (revision 120301)
> --- io/unix.c   (working copy)
> *************** fd_flush (unix_stream * s)
> *** 349,355 ****
>     size_t writelen;
> 
>     if (s->ndirty == 0)
> !     return SUCCESS;;
> 
>     if (s->physical_offset != s->dirty_offset &&
>         lseek (s->fd, s->dirty_offset, SEEK_SET) < 0)
> --- 349,358 ----
>     size_t writelen;
> 
>     if (s->ndirty == 0)
> !     return SUCCESS;
> !   
> !   if (s->file_length == -1)
> !     return SUCCESS;

I don't understand this part.  Shouldn't we flush the
buffer anyway, even without a seek?


>   fd_truncate (unix_stream * s)
>   {
>     if (lseek (s->fd, s->logical_offset, SEEK_SET) == -1)
> !     return SUCCESS;

A small matter, we should test for seekable files first:

      if (s->file_length == -1)
        return SUCCESS;

      if (lseek(s->fd, s->logical_offset, SEEK_SET) == -1)
        return FAILURE;

If this fails, something is really wrong.

As an aside, this already works for ACCESS="stream" and unformatted
files, so I think we should think about committing this.

Subject: Bug number PR30162

A patch for this bug has been added to the patch tracker.
The mailing list url for the patch is http://gcc.gnu.org/ml/gcc-patches/2007-01/msg00325.html

Subject: Bug 30162

Author: jvdelisle
Date: Sat Jan  6 00:14:38 2007
New Revision: 120512

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=120512
Log:
2007-01-05  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

	PR libgfortran/30162
	* io/unix.c (fd_flush): Don't seek if file is not seekable, defined as
	s->file_length == -1.
	(fd_alloc_w_at): Do not adjust file_length if file is not seekable.
	(fd_seek): If not seekable, just return success.
	(fd_truncate): If not seekable, no need to truncate.  Return failure if
	seek fails and the stream is not a pipe.
	(fd_to_stream): Make test for non-seekable file more robust.

Modified:
    trunk/libgfortran/ChangeLog
    trunk/libgfortran/io/unix.c

A difficult case to deal with is:

program main
  print *,foo(0)
contains
  function foo (n) result(res)
    integer, intent(in) :: n
    integer, allocatable :: res(:)
    logical :: init = .false.

    if (.not. init) then
       allocate (res(1))
       init = .true.
    else
       allocate (res(2))
    end if

    res = n
  end function foo
end program main

Another difficult case is:

program main
  character(len=4) c
  c = 'ab  '
  write (10) trim(c)
end program main

Subject: Bug 30162

Author: jvdelisle
Date: Sun Jan  7 01:34:03 2007
New Revision: 120544

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=120544
Log:
2007-01-06  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

	PR libgfortran/30162
	* io/unix.c (fd_flush): Don't seek if file is not seekable, defined as
	s->file_length == -1.
	(fd_alloc_w_at): Do not adjust file_length if file is not seekable.
	(fd_seek): If not seekable, just return success.
	(fd_truncate): If not seekable, no need to truncate.  Return failure if
	seek fails and the stream is not a pipe.
	(fd_to_stream): Make test for non-seekable file more robust.

2007-01-06  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

	* ChangeLog: Copied to ...
	* ChangeLog-2006: here.


Added:
    branches/gcc-4_2-branch/libgfortran/ChangeLog-2006
Modified:
    branches/gcc-4_2-branch/libgfortran/ChangeLog
    branches/gcc-4_2-branch/libgfortran/io/unix.c

Is this bug fixed or not? I see a 4.3 and a 4.2 check in.
Or is something missing, if yes, what?

We do not have this working for unformatted I/O.  It will require frontend work as well as library to implement.  Specifically, the size of records, determined from the output list, must be determined prior to performing transfers.

On the library side, once the record size is known, seeking backwards in named pipes must be avoided.  Right now we seek backward to save the beginning record marker after the length of the record is determined by writing the record.  This backward seek is costly in performance.

I am convinced that these changes to frontend and library can be done.  I just do not have time right now to pursue it.  I will be studying the problem in the meantime, but I do not want to hold back anyone else from making the attempt, thinking I may be working it.

I think this is the same bug.  If not let me know and I'll repost it.
This is a real bummer.  I use this to turn off noise from routines I don't have control of (and I mean ALOT of noise).  Maybe there is another way to do what I want though.  Anyway, the following produces a seek error on the write line.  It worked well in f77.  Now using gcc-4.1.2

       program bug
        close(6)
        open(6,file='/dev/null',status='old')
        close(6)
c  would normally do some noisy stuff here 
c    but not needed to make the bug.
        open(6,file='/dev/stdout',status='old')
        write (*,*)"Hello"
       end

(In reply to comment #18)
> I think this is the same bug.  If not let me know and I'll repost it.
> This is a real bummer.  I use this to turn off noise from routines I don't have
> control of (and I mean ALOT of noise).  Maybe there is another way to do what I
> want though.  Anyway, the following produces a seek error on the write line. 
> It worked well in f77.  Now using gcc-4.1.2
> 
>        program bug
>         close(6)
>         open(6,file='/dev/null',status='old')
>         close(6)
> c  would normally do some noisy stuff here 
> c    but not needed to make the bug.
>         open(6,file='/dev/stdout',status='old')
>         write (*,*)"Hello"
>        end
> 

of course the comment should be one line higher.

I will try to have a look in the next few days.

This works on 4.2 and 4.3.  Not planning to backport to 4.1.  See if you can get a later version of gfortran.  If you have problems doing so, check the gfortran wiki binaries.  If that fails, let me know.

Revision 180701 removed all checks for special files in unit.c:unit_truncate(). As a consequence programs compiled with gfortran 4.6.3 and newer cannot to write (formatted) to Unix named pipes as ftruncate() fails with EINVAL:

(64-bit code)

lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
write(3, " Hello, world!\n", 15)        = 15
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
ftruncate(3, 18446744073709551615)      = -1 EINVAL (Invalid argument)

(32-bit code)

_llseek(3, 0, 0xffba8fa0, SEEK_CUR)     = -1 ESPIPE (Illegal seek)
write(3, " Hello, world!\n", 15)        = 15
_llseek(3, 0, 0xffba9080, SEEK_CUR)     = -1 ESPIPE (Illegal seek)
ftruncate64(3, 18446744073709551615)    = -1 EINVAL (Invalid argument)

(In reply to comment #22)
> Revision 180701 removed all checks for special files in unit.c:unit_truncate().

Yes. As the email message introducing the patch explained, special files are special in many ways, and trying to impose some kind of uniform behavior on them is bound to fail in more or less surprising and inconvenient ways. For the same reason, the code for handling special files were removed from the buffered IO functions, and all non-regular files now just use the raw IO functions instead. The only sane way is to seek/truncate only when the user code requires such a behavior, and if that then fails, return an error and let the user handle it.

The real bug is thus that we shouldn't try to seek or truncate the fd at all, as it's not necessary in this case.

I recall that I tried some simple programs with named pipes when I implemented the patch, but maybe some subsequent patch then broke it again.

(In reply to comment #23)
> The real bug is thus that we shouldn't try to seek or truncate the fd at all,
> as it's not necessary in this case.

I think the issue occurs for:

st_write_done (st_parameter_dt *dtp)
...
  if (dtp->u.p.current_unit != NULL
      && dtp->u.p.current_unit->flags.access == ACCESS_SEQUENTIAL)
    switch (dtp->u.p.current_unit->endfile)
...
      case NO_ENDFILE:
        /* Get rid of whatever is after this record.  */
        if (!is_internal_unit (dtp))
          unit_truncate (dtp->u.p.current_unit,
                         stell (dtp->u.p.current_unit->s),
                         &dtp->common);
        dtp->u.p.current_unit->endfile = AT_ENDFILE;

with a file as simple as

  open (99, file="foo.dat")
  write (99, '(a)') 'Hello world'
  end


$ mkfifo foo.dat
$ tail -f foo.dat &
[2] 342
$ gfortran test.f90 && ./a.out
At line 2 of file test.f90 (unit = 99, file = 'foo.dat')
Fortran runtime error: Invalid argument
Hello world
$ fg
tail -f foo.dat
^C

As Janne just mentioned on the gfortran mailing list, this PR is a regression. Thus, I have marked it as such. Based on the information below, It fails since GCC 4.7.

In reply to comment #22)
> Revision 180701 removed all checks for special files in unit.c:unit_truncate().

Is this caused by

http://gcc.gnu.org/viewcvs?view=revision&revision=180701

?

Maybe we need to remember if we have a special file after all, or just ignore the error on the truncate.

(In reply to comment #26)
> Is this caused by
> 
> http://gcc.gnu.org/viewcvs?view=revision&revision=180701
> 
> ?
> 
> Maybe we need to remember if we have a special file after all, or just ignore
> the error on the truncate.

IMHO the correct fix is to not seek and/or truncate the file unless the Fortran semantics require it; that way libgfortran does not need to care whether the file is special or not. As explained in #c23, special files are special in different ways (also different on different OS'es), and trying to enumerate all the ways in which they are special is bound to fail. 

I think Tobias comment #c24 pinpoints the place which needs to be fixed, but unfortunately I haven't had time to look into it.

(In reply to comment #27)
> (In reply to comment #26)
> > Is this caused by
> > 
> > http://gcc.gnu.org/viewcvs?view=revision&revision=180701
> > 
> > ?
> > 
> > Maybe we need to remember if we have a special file after all, or just ignore
> > the error on the truncate.
> 
> IMHO the correct fix is to not seek and/or truncate the file unless the Fortran
> semantics require it; that way libgfortran does not need to care whether the
> file is special or not. As explained in #c23, special files are special in
> different ways (also different on different OS'es), and trying to enumerate all
> the ways in which they are special is bound to fail. 
> 
> I think Tobias comment #c24 pinpoints the place which needs to be fixed, but
> unfortunately I haven't had time to look into it.

Well, we need to make sure that the (very basic) program

      program main
      character*10 x
      open(10,file="foo.dat")
      write (10,'(A)') '1111'
      write (10,'(A)') '2222'
      close (10)
      open(10,file="foo.dat")
      write (10,'(A)') '3333'
      close (10)
      open(10,file="foo.dat")
 100  continue
        read (10,'(A)',end=200) x
        write (*,'(A)') x
      goto 100
 200  continue
      close(10,status="delete")
      end

continues to work as expected: That probably means truncating on
close and rewind.

I can see what I can do, but I have little time... (as always)

(In reply to comment #28)
> (In reply to comment #27)
> > (In reply to comment #26)
> > > Is this caused by
> > > 
> > > http://gcc.gnu.org/viewcvs?view=revision&revision=180701
> > > 
> > > ?
> > > 
> > > Maybe we need to remember if we have a special file after all, or just ignore
> > > the error on the truncate.
> > 
> > IMHO the correct fix is to not seek and/or truncate the file unless the Fortran
> > semantics require it; that way libgfortran does not need to care whether the
> > file is special or not. As explained in #c23, special files are special in
> > different ways (also different on different OS'es), and trying to enumerate all
> > the ways in which they are special is bound to fail. 
> > 
> > I think Tobias comment #c24 pinpoints the place which needs to be fixed, but
> > unfortunately I haven't had time to look into it.
> 
> Well, we need to make sure that the (very basic) program
> 
>       program main
>       character*10 x
>       open(10,file="foo.dat")
>       write (10,'(A)') '1111'
>       write (10,'(A)') '2222'
>       close (10)
>       open(10,file="foo.dat")
>       write (10,'(A)') '3333'
>       close (10)
>       open(10,file="foo.dat")
>  100  continue
>         read (10,'(A)',end=200) x
>         write (*,'(A)') x
>       goto 100
>  200  continue
>       close(10,status="delete")
>       end
> 
> continues to work as expected: That probably means truncating on
> close and rewind.

I'm not sure about truncation being necessary here, actually. Where we do need to truncate is 

1) Following a write in access='sequential' mode, when we are not at the end of the file (that is, we have opened an existing file and are at the beginning, we have previously used rewind/backspace or such). This might also apply to formatted writes in access='stream' mode, need to check the standard.

2) Executing an ENDFILE statement.

3) Opening a file with status='replace' (actually, probably not, as a sensible implementation would be to open() with O_TRUNC instead of a separate ftruncate() call following open(), and while I'm too lazy to look it up right now I'm quite sure this is exactly what we already do)

I think the current code goes wrong with pipes because as they are reported to be of size 0, it is undefined whether we're at the beginning, middle, or end of the file, and hence we have made some arbitrary choice, and hit_eof() think's we're not at the end, and tries to truncate. Perhaps we need something more sophisticated such as some need_to_truncate_after_next_write flag which is set and tested at appropriate places..

This seems to do the trick.

Index: unix.c
===================================================================
--- unix.c      (Revision 194507)
+++ unix.c      (Arbeitskopie)
@@ -344,7 +344,15 @@
 static gfc_offset
 raw_tell (unix_stream * s)
 {
-  return lseek (s->fd, 0, SEEK_CUR);
+  gfc_offset x;
+  x = lseek (s->fd, 0, SEEK_CUR);
+
+  /* Non-seekable files should always be assumed to be at
+     current position.  */
+  if (x == -1 && errno == ESPIPE)
+    x = 0;
+
+  return x;
 }
 
 static gfc_offset

Author: tkoenig
Date: Fri Dec 21 20:50:48 2012
New Revision: 194679

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=194679
Log:
2012-12-21  Thomas Koenig  <tkoenig@gcc.gnu.org>

	PR libfortran/30162
	* io/unix.c (raw_tell):  If the lseek is done on a
	non-seekable file, return 0.


Modified:
    trunk/libgfortran/ChangeLog
    trunk/libgfortran/io/unix.c

Author: tkoenig
Date: Sat Dec 22 10:46:37 2012
New Revision: 194694

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=194694
Log:
2012-12-22  Thomas Koenig  <tkoenig@gcc.gnu.org>

	PR libfortran/30162
	Backport from trunk
	* io/unix.c (raw_tell):  If the lseek is done on a
	non-seekable file, return 0.


Modified:
    branches/gcc-4_7-branch/libgfortran/ChangeLog
    branches/gcc-4_7-branch/libgfortran/io/unix.c

Fixed on trunk and 4.7, closing.

I still see

 status from open is            0
 status from open is            0
At line 7 of file pr30162_1.f (unit = 20, file = 'np')
Fortran runtime error: Illegal seek

[1]    Exit 2                        a.exe
At line 8 of file pr30162_2.f (unit = 21, file = 'np')
Fortran runtime error: I/O past end of record on unformatted file

on x86_64-apple-darwin10 r194681.

> I still see
>
>   status from open is            0
>   status from open is            0
> At line 7 of file pr30162_1.f (unit = 20, file = 'np')
> Fortran runtime error: Illegal seek
>
> [1]    Exit 2                        a.exe
> At line 8 of file pr30162_2.f (unit = 21, file = 'np')
> Fortran runtime error: I/O past end of record on unformatted file
>
> on x86_64-apple-darwin10 r194681.

This is strange.

Can you strace (or whatever) the code and tell me what lseek()
returns?  Is it really EINVAL like the error message suggests?
If so, this would be strange, because

http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man2/lseek.2.html

tells me (as a return value)

      [EINVAL]           Whence is not a proper value.

      [ESPIPE]           Fildes is associated with a pipe, socket, or FIFO.

which indicates a bug in the library, but maybe I am looking
at the wrong manpage here.

> Can you strace (or whatever) the code and tell me what lseek()
> returns?  Is it really EINVAL like the error message suggests?

The best I can do (without understanding it) is

(1) if I use dtruss (should be equivalent to strace) on a, I get

[macbook] f90/bug% 25210/0x2baef9e:  lseek(0x3, 0x0, 0x1)		 = -1 Err#29
25210/0x2baef9e:  lseek(0x3, 0xFFFFFFFFFFFFFFF4, 0x1)		 = -1 Err#29

(2) if I use it on b, I get

[macbook] f90/bug% 25218/0x2baf031:  lseek(0x3, 0x0, 0x1)		 = -1 Err#29
25218/0x2baf031:  lseek(0x3, 0x4, 0x1)		 = -1 Err#29

Hi Dominique,

> The best I can do (without understanding it) is
>
> (1) if I use dtruss (should be equivalent to strace) on a, I get
>
> [macbook] f90/bug% 25210/0x2baef9e:  lseek(0x3, 0x0, 0x1)         = -1 Err#29
> 25210/0x2baef9e:  lseek(0x3, 0xFFFFFFFFFFFFFFF4, 0x1)         = -1 Err#29
>
> (2) if I use it on b, I get
>
> [macbook] f90/bug% 25218/0x2baf031:  lseek(0x3, 0x0, 0x1)         = -1 Err#29
> 25218/0x2baf031:  lseek(0x3, 0x4, 0x1)         = -1 Err#29

That doesn't point to anything suspicious.

Can you tell me what happens if you apply this patch?

Index: io/unix.c
===================================================================
--- io/unix.c   (Revision 194679)
+++ io/unix.c   (Arbeitskopie)
@@ -349,7 +349,7 @@

    /* Non-seekable files should always be assumed to be at
       current position.  */
-  if (x == -1 && errno == ESPIPE)
+  if (x == -1)
      x = 0;

    return x;

> Can you tell me what happens if you apply this patch?

The behavior stays unchanged;-(

Unfortunately, I cannot really debug this without access to a machine
where it fails.

Does this also fail on another BSD-based machine?  Is such a machine
available on the gcc compile farm?

> Unfortunately, I cannot really debug this without access to a machine
> where it fails.

Would it be possible to write a simple C test that can be run on the different platforms in order to check the behavior of lseek?

Note that I'll be off the hook for the next ten days , so don't expect feedback before the beginning of next year.

Am 23.12.2012 15:16, schrieb dominiq at lps dot ens.fr:
>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162
>
> --- Comment #40 from Dominique d'Humieres <dominiq at lps dot ens.fr> 2012-12-23 14:16:52 UTC ---
>> Unfortunately, I cannot really debug this without access to a machine
>> where it fails.
>
> Would it be possible to write a simple C test that can be run on the different
> platforms in order to check the behavior of lseek?

The tests that you ran with the patchlet show that the error is not
in setting errno on the lseek.  It has to be somewhere else.

If you still have the time, could you attach the truss output of

program main
   open (10,file="pipe",status="old")
   write (10,'(A)') 'Hello, world!'
end program main

where pipe is a FIFO, and the same for

program read
   character(len=100) c
   open (10,file="pipe", status="old")
   read (10,'(A)') c
   print *,trim(c)
end program read

(first starting read, then write).

I'll try to find a system I have access to where this also fails;
unassigning myself until then.

I finally had a look at this PR.
First the test on comment #41 works on x86_64-apple-darwin10 r196109:

macbook] f90/bug% mkfifo pipe
[macbook] f90/bug% read_f &
[1] 6998
[macbook] f90/bug% write_f
 Hello, world!
[macbook] f90/bug% 
[1]    Done                          read_f

I have done the following changes to the original test:

[macbook] f90/bug% cat pr30162_1_db.f
      integer status
      integer i
      open(unit=20,file='np',action='write',
     &     form='unformatted',iostat=status);
      print*,'status for write from open is ',status
      do i = 1,5
         print *, "write ", i
         write(20)i
      end do
      end
[macbook] f90/bug% cat pr30162_2_db.f
      integer status
      integer i
      integer val
      open(unit=21,file='np',action='read',
     &     form='unformatted',iostat=status);
      print*,'status for read from open is ',status
      do i = 1,5
         read(21)val
         print*,val
      end do
      end

and I got

[macbook] f90/bug% write_f
 status for write from open is            0
 write            1
At line 8 of file pr30162_1_db.f (unit = 20, file = 'np')
Fortran runtime error: Illegal seek
 status for read from open is            0
At line 8 of file pr30162_2_db.f (unit = 21, file = 'np')
Fortran runtime error: I/O past end of record on unformatted file
[macbook] f90/bug% 
[1]    Exit 2                        read_f

Then I have applied the following patch (based on r194679):

--- ../_clean/libgfortran/io/unix.c	2013-01-14 19:25:10.000000000 +0100
+++ libgfortran/io/unix.c	2013-02-17 20:23:43.000000000 +0100
@@ -336,7 +336,14 @@ raw_write (unix_stream * s, const void *
 static gfc_offset
 raw_seek (unix_stream * s, gfc_offset offset, int whence)
 {
-  return lseek (s->fd, offset, whence);
+  gfc_offset x;
+  x = lseek (s->fd, offset, whence);
+  /* Non-seekable files should always be assumed to be at
+     current position.  */
+  if (x == -1 && errno == ESPIPE)
+    x = 0;
+  return x;
+  /* return lseek (s->fd, offset, whence); */
 }
 
 static gfc_offset

and now I get

[macbook] f90/bug% write_f
 status for write from open is            0
 write            1
 write            2
 write            3
 write            4
 write            5
 status for read from open is            0
At line 8 of file pr30162_2_db.f (unit = 21, file = 'np')
Fortran runtime error: I/O past end of record on unformatted file
[macbook] f90/bug% 
[1]    Exit 2                        read_f

So the write seems now OK. For the read, dtrace gives

fstat64(0x3, 0x7FFF5FBFD580, 0x50)		 = 0 0
fstat64(0x3, 0x7FFF5FBFD5C0, 0x0)		 = 0 0
lseek(0x3, 0x0, 0x1)		 = -1 Err#29
write(0x1, " status for read from open is            0\n\0", 0x2B)		 = 43 0
read(0x3, "\0", 0x4)		 = 4 0
read(0x3, "\0", 0x0)		 = 0 0
lseek(0x3, 0x4, 0x1)		 = -1 Err#29
write(0x2, "At line 8 of file pr30162_2_db.f (unit = 21, file = 'np')\n\0", 0x3A)		 = 58 0
write(0x2, "Fortran runtime error: \0", 0x17)		 = 23 0
write(0x2, "I/O past end of record on unformatted file\0", 0x2A)		 = 42 0
write(0x2, "\n\0", 0x1)		 = 1 0
close(0x3)		 = 0 0

i.e., 2 errors #29 on lseek. I did not find out how this is connected to the error "I/O past end of record on unformatted file".

If unformatted sequential ever worked, it was by chance.

Look at this piece of code:

  /* Seek to the head and overwrite the bogus length with the real
     length.  */

  if (unlikely (sseek (dtp->u.p.current_unit->s, - m - 2 * record_marker, 
		       SEEK_CUR) < 0))
    goto io_error;

This works if it happens within a buffer, but you cannot rely on that.

I would therefore suggest to resolve this PR by issuing a well-defined
error if we encounter a pipe on opening.

I'll prepare a patch.

> ...
> I would therefore suggest to resolve this PR by issuing a well-defined
> error if we encounter a pipe on opening.

Are you suggesting to "fix" the non working named pipes on Darwin by removing them everywhere?
If yes, isn't it a little bit too radical?

If there are pieces of code working only by chance, would not it be better to fix them?

(In reply to comment #44)
> If unformatted sequential ever worked, it was by chance.
> 
> Look at this piece of code:
> 
>   /* Seek to the head and overwrite the bogus length with the real
>      length.  */
> 
>   if (unlikely (sseek (dtp->u.p.current_unit->s, - m - 2 * record_marker, 
>                SEEK_CUR) < 0))
>     goto io_error;
> 
> This works if it happens within a buffer, but you cannot rely on that.
> 
> I would therefore suggest to resolve this PR by issuing a well-defined
> error if we encounter a pipe on opening.
> 
> I'll prepare a patch.

As I explained in comment #23, this is already handled. When opening a file, we stat() the fd, and use the buffered I/O functions only if it's a regular file, otherwise the unbuffered raw I/O functions are used.

Am 18.02.2013 21:16, schrieb jb at gcc dot gnu.org:
>> Look at this piece of code:
>> >
>> >   /* Seek to the head and overwrite the bogus length with the real
>> >      length.  */
>> >
>> >   if (unlikely (sseek (dtp->u.p.current_unit->s, - m - 2 * record_marker,
>> >                SEEK_CUR) < 0))
>> >     goto io_error;
>> >
>> >This works if it happens within a buffer, but you cannot rely on that.
>> >
>> >I would therefore suggest to resolve this PR by issuing a well-defined
>> >error if we encounter a pipe on opening.
>> >
>> >I'll prepare a patch.
> As I explained in comment #23, this is already handled. When opening a file, we
> stat() the fd, and use the buffered I/O functions only if it's a regular file,
> otherwise the unbuffered raw I/O functions are used.

This is part of the precipitate, not part of the solution ;-)

For unformatted sequential, we first write a dummy record marker, write
the data and the final record marker, then seek to the first marker
and re-write that. Of course, this fails for an unbuffered fifo.

If we want this to work, we should make sure we always use
_buffered_ I/O for unformatted sequential, setting a maximum
record length on open that we can handle with our buffer.

Am 18.02.2013 20:15, schrieb dominiq at lps dot ens.fr:
> Are you suggesting to "fix" the non working named pipes on Darwin by removing
> them everywhere?

Actually, the error happens everywhere, I could reproduce the error with
unformatted sequential on my Linux box.

> If yes, isn't it a little bit too radical?

Formatted sequential, unformatted stream and formatted stream
would continue to work. As BACKSPACE most certainly will not work
on a pipe, I don't see the advantage of using unformatted sequential.

Here's a proof of concept patch which lets the
test cases "work" due to buffering.

Index: unix.c
===================================================================
--- unix.c      (Revision 195922)
+++ unix.c      (Arbeitskopie)
@@ -995,7 +995,8 @@
   s->file_length = statbuf.st_size;
 
   /* Only use buffered IO for regular files.  */
-  if (S_ISREG (statbuf.st_mode)
+  if ((S_ISREG (statbuf.st_mode)
+       || S_ISFIFO (statbuf.st_mode))
       && !options.all_unbuffered
       && !(options.unbuffered_preconnected && 
           (s->fd == STDIN_FILENO 

What it doesn't do is handle writes larger than
BUFSIZE, and this limitation is not communicated up
from the stream library to the record length, and also
not the other way.  This would break the abstraction
(but maybe we would not care, would we?)

(In reply to comment #47)
> Am 18.02.2013 21:16, schrieb jb at gcc dot gnu.org:
> >> Look at this piece of code:
> >> >
> >> >   /* Seek to the head and overwrite the bogus length with the real
> >> >      length.  */
> >> >
> >> >   if (unlikely (sseek (dtp->u.p.current_unit->s, - m - 2 * record_marker,
> >> >                SEEK_CUR) < 0))
> >> >     goto io_error;
> >> >
> >> >This works if it happens within a buffer, but you cannot rely on that.
> >> >
> >> >I would therefore suggest to resolve this PR by issuing a well-defined
> >> >error if we encounter a pipe on opening.
> >> >
> >> >I'll prepare a patch.
> > As I explained in comment #23, this is already handled. When opening a file, we
> > stat() the fd, and use the buffered I/O functions only if it's a regular file,
> > otherwise the unbuffered raw I/O functions are used.
> 
> This is part of the precipitate, not part of the solution ;-)
> 
> For unformatted sequential, we first write a dummy record marker, write
> the data and the final record marker, then seek to the first marker
> and re-write that. Of course, this fails for an unbuffered fifo.
> 
> If we want this to work, we should make sure we always use
> _buffered_ I/O for unformatted sequential, setting a maximum
> record length on open that we can handle with our buffer.

The buffered I/O functions generally assume that the underlying fd is a regular file; if you break that be prepared to handle a bunch of subtle fallout. I tried playing that whack-a-mole game a few years ago until I got tired of it and restricted buffered I/O to regular files.

Wrt to pipes, one regression we had (I don't remember the PR number) was a deadlock due to the buffering. E.g. one process writes "hello" to the pipe, the other one answers "hello, neighbor". The write part could be handled by executing the FLUSH statement, but I recall there was some problem on the read side as well, something to do with read() hanging due to there not being any data available in the pipe when it tried to fill the buffer.

Anyway, I don't think this is something libgfortran can, or should, try to fix. If the user tries to do something requiring seeking on a special file that doesn't support seeking, they get what they deserve. The real bug here is that we try to seek even though the Fortran semantics don't require it (mind that the complaint in comment #22 which reopened this PR refers to formatted I/O).

(In reply to comment #22)
> Revision 180701 removed all checks for special files in unit.c:unit_truncate().
> As a consequence programs compiled with gfortran 4.6.3 and newer cannot to
> write (formatted) to Unix named pipes as ftruncate() fails with EINVAL:
> 
> (64-bit code)
> 
> lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
> write(3, " Hello, world!\n", 15)        = 15
> lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
> ftruncate(3, 18446744073709551615)      = -1 EINVAL (Invalid argument)
> 
> (32-bit code)
> 
> _llseek(3, 0, 0xffba8fa0, SEEK_CUR)     = -1 ESPIPE (Illegal seek)
> write(3, " Hello, world!\n", 15)        = 15
> _llseek(3, 0, 0xffba9080, SEEK_CUR)     = -1 ESPIPE (Illegal seek)
> ftruncate64(3, 18446744073709551615)    = -1 EINVAL (Invalid argument)

This bug (http://stackoverflow.com/questions/11780556/write-to-fifo-named-pipe , presumably) should be fixed by the patch at

http://gcc.gnu.org/ml/gcc-patches/2013-02/msg00923.html

Author: jb
Date: Thu Feb 21 19:03:10 2013
New Revision: 196210

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=196210
Log:
Fix regression when writing formatted sequential to a pipe.

2013-02-21  Janne Blomqvist  <jb@gcc.gnu.org>

	PR libfortran/30162
	* io/open.c (test_endfile): Call stell only if size != 0.
	* io/unix.c (raw_tell): Revert r194679.
	(raw_size): Return size field only for regular files, otherwise 0.

Modified:
    trunk/libgfortran/ChangeLog
    trunk/libgfortran/io/open.c
    trunk/libgfortran/io/unix.c

Author: jb
Date: Thu Feb 21 20:13:04 2013
New Revision: 196212

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=196212
Log:
Fix regression when writing formatted sequential to a pipe.

2013-02-21  Janne Blomqvist  <jb@gcc.gnu.org>

	PR libfortran/30162
	* io/open.c (test_endfile): Call stell only if size != 0.
	* io/unix.c (raw_tell): Revert r194694.
	(raw_size): Return size field only for regular files, otherwise 0.

Modified:
    branches/gcc-4_7-branch/libgfortran/ChangeLog
    branches/gcc-4_7-branch/libgfortran/io/open.c
    branches/gcc-4_7-branch/libgfortran/io/unix.c

I'd suggest to close the unformatted sequential part of this PR as WONTFIX. Maybe it worked sometime in the past for small record sizes when buffered IO was used for pipes, but that was by accident not by design. I think re-enabling buffering for pipes would open up a can of worms for very little benefit.

A way to "properly" enable this would be to rework the IO library so that we know how much we're writing before writing the initial record marker (so that we don't need to seek back and fill in the correct value later), however that implies a rather major rework of how we do IO, so IMHO the benefit is not worth it.

(In reply to comment #54)
> I'd suggest to close the unformatted sequential part of this PR as WONTFIX. [...]

Should we document in http://gcc.gnu.org/onlinedocs/gfortran/ that I/O via pipes (and other nonseekable devices) is only supported for STREAM and for FORMATTED+SEQUENTIAL?

GCC 4.7.3 is being released, adjusting target milestone.

Downgrading to a documentation issue.

Closing.