I am trying to read the binary files produced with amd64 on a 32 bit machine. This is not possible because the record header changed from 4-bytes to 8-bytes. Can this be fixed? Example program: write(1)'Z' end produces fort.1 on opteron with the following content if opened in emacs: ^A^@^@^@^@^@^@^@Z^A^@^@^@^@^@^@^@ But it produces: ^A^@^@^@Z^A^@^@^@ on 32 bit machine or using -m32 on opteron. I beleive this is libfortran bug or feature, but I still thing the files should be readable across the platforms, or there should be some compile time option for this. I hope this is not against some stupid standard :-(
This is a mis-ferature inherited from g77. It's definitely premitted by the standard, but probably wants "fixing" anyway. Full discussion here: http://gcc.gnu.org/ml/fortran/2004-12/msg00139.html
Also note on some targets (like all *BSD's by the way), gfc_offset is always 64bits so really this is a miss feature in the OS rather than in gfortran in my mind.
As I said before this really should be 64bit on all targets instead as we should support LFS (large file support) (which is deafualt really on all *BSD, darwin is included with this *BSD).
Somewhat related patch here: http://gcc.gnu.org/ml/gcc-patches/2005-02/msg01085.html
*** Bug 21570 has been marked as a duplicate of this bug. ***
*** Bug 21673 has been marked as a duplicate of this bug. ***
proposed patch that should fix this problem: http://gcc.gnu.org/ml/fortran/2005-05/msg00431.html
*** Bug 21621 has been marked as a duplicate of this bug. ***
Great work! Thanks a million! Somehow I also have a problem with the name of the option, since g77 has exactly the same problem. When I compile with the g77 I get 8 byte headers on AMD64, so the name g77 doesn't really correspond to always get 4 byte headers. But this is very minor thing especially if it is documented how it works, the name shouldn't really matter.
Subject: Re: Unformatted record header is 4-bytes on 32-bit targets I second the "thanks a million", and also don't think g77 belongs in the option name. I have been using a very fast sort program for large files written in C++ (not by me) , and compiled with gcc-3.3.2 (c++) which produces a binary file that is read using Fortran code compiled with g77, and have not experienced any problems. I was testing this very large simulation using gcc-4.0 (c++) and g95 when I ran into the problem -- the binary file created with gcc-4.0 (c++) could not be read by g95 compiled code -- incompatible record markers (32 bit vice 64 bit). blime milan at cmm dot ki dot si wrote: >------- Additional Comments From milan at cmm dot ki dot si 2005-05-31 08:41 ------- >Great work! Thanks a million! > >Somehow I also have a problem with the name of the option, since g77 has exactly >the same problem. When I compile with the g77 I get 8 byte headers on AMD64, so >the name g77 doesn't really correspond to always get 4 byte headers. But this is >very minor thing especially if it is documented how it works, the name shouldn't >really matter. > > >
*** Bug 23814 has been marked as a duplicate of this bug. ***
FYI: Here's what Intel did for to address the record sizes larger than 2 GB: http://www.intel.com/software/products/compilers/flin/docs/main_for/mergedProjects/bldaps_for/format_of_record_types_.htm The nice thing about this approach is that the files are compatible with files generated by f77/g77 and other vendors' f90/f95 compilers. The tradeoff is that it makes the i/o logic a little more complicated. I believe that PGI (Portland Group) compiler generates f77 compliant unformatted files as well. I haven't found their approach yet for dealing with >2GB record sizes, but the docs are at http://www.pgroup.com/resources/docs.htm I'm going to check Cray, SGI and Sun as well to see what format they output.
FYI: The latest Cray, IRIX64 and Solaris fortran compilers all use 4 byte record markers in their unformatted files and are hence interoperable. FWIW, I think the Intel solution should be considered to support record lengths greater than 2GB.
Created attachment 10015 [details] changes unformatted record delimitters to 4 bytes for compatibility with other compilers I modified gfortran to produce unformatted records with 4 byte delimitters so the binaries would be compatible with the other Linux Fortran compilers such as Intel's and Portland Group's. I uploaded the patch just in case anybody needed the same capability and perhaps as a start to a more general patch. (Let me know if I screwed up on creating the patch...I just did a cvs diff on the related directory.)
I didn 't know how to compile gcc-4.1... so I couldn't reply before. I realised I have to install both mpfr and gmp libraries for gcc to compile. It complains only about gmp :-( Yes, this patch works OK. I had to patch the gcc-4.1-20051112 snapshot myself.
Created attachment 10296 [details] Patch to change delimitters to 4 bytes for unformatted records This is nearly the same patch that I posted before except for the head of the subversion repository.
(In reply to comment #16) > Created an attachment (id=10296) [edit] > Patch to change delimitters to 4 bytes for unformatted records > > This is nearly the same patch that I posted before except for the head of the > subversion repository. > I tried this one, but is now already out of date. I just made a hand patch of 4.1.0-beta20051223 and it seems to be OK. When will this be in the 4.1 snapshots, so I don't need to patch it by hand?
Created attachment 10564 [details] patch against gcc-4.1-20051223 As requested, I am posting Rob's patch which goes against the Dec 23 snapshot. I didn't change reverse_memcpy() which I think I should, but maybe the one who added it should be more appropriate person to do it.
(In reply to comment #18) > Created an attachment (id=10564) [edit] > patch against gcc-4.1-20051223 > > As requested, I am posting Rob's patch which goes against the Dec 23 snapshot. > I didn't change reverse_memcpy() which I think I should, but maybe the one who > added it should be more appropriate person to do it. Your patch works only on little-endian systems. I am currently thinking of making this a compile-time option. Thomas
I'll take this, implementing the simplistic approach (generating an error for >2GB record sizes). This should keep the complexity down.
Subject: Bug number PR 19303 A patch for this bug has been added to the patch tracker. The mailing list url for the patch is http://gcc.gnu.org/ml/gcc-patches/2006-03/msg00980.html
Subject: Bug 19303 Author: tkoenig Date: Wed Mar 22 19:09:11 2006 New Revision: 112290 URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=112290 Log: 2006-03-22 Thomas Koenig <Thomas.Koenig@onlien.de> PR fortran/19303 * gfortran.h (gfc_option_t): Add record_marker. * lang.opt: Add -frecord-marker=4 and -frecord-marker=8. * trans-decl.c: Add gfor_fndecl_set_record_marker. (gfc_build_builtin_function_decls): Set gfor_fndecl_set_record_marker. (gfc_generate_function_code): If we are in the main program and -frecord-marker was provided, call set_record_marker. * options.c (gfc_handle_option): Add handling for -frecord-marker=4 and -frecord-marker=8. * invoke.texi: Document -frecord-marker. 2006-03-22 Thomas Koenig <Thomas.Koenig@onlien.de> PR fortran/19303 * libgfortran.h (compile_options_t): Add record_marker. * runtime/compile_options.c (set_record_marker): New function. * io/open.c: If we have four-byte record markers, use GFC_INTEGER_4_HUGE as default record length. * io/file_pos.c (unformatted_backspace): Handle different size record markers. * io/transfer.c (us_read): Likewise. (us_write): Likewise. (next_record_r): Likewise. (write_us_marker): Likewise. (next_record_w): Likewise. 2006-03-22 Thomas Koenig <Thomas.Koenig@online.de> PR fortran/19303 * gfortran.dg/record_marker_1.f90: New test case. * gfortran.dg/record_marker_2.f: New test case. * gfortran.dg/record_marker_3.f90: New test case. Added: trunk/gcc/testsuite/gfortran.dg/record_marker_1.f90 trunk/gcc/testsuite/gfortran.dg/record_marker_2.f trunk/gcc/testsuite/gfortran.dg/record_marker_3.f90 Modified: trunk/gcc/fortran/ChangeLog trunk/gcc/fortran/gfortran.h trunk/gcc/fortran/invoke.texi trunk/gcc/fortran/lang.opt trunk/gcc/fortran/options.c trunk/gcc/fortran/trans-decl.c trunk/gcc/testsuite/ChangeLog trunk/libgfortran/ChangeLog trunk/libgfortran/io/file_pos.c trunk/libgfortran/io/open.c trunk/libgfortran/io/transfer.c trunk/libgfortran/libgfortran.h trunk/libgfortran/runtime/compile_options.c
Subject: Bug 19303 Author: tkoenig Date: Fri Mar 31 21:13:46 2006 New Revision: 112590 URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=112590 Log: PR fortran/19303 Backport from mainline * libgfortran.h (compile_options_t): Add record_marker. * runtime/compile_options.c (set_record_marker): New function. * io/open.c: If we have four-byte record markers, use GFC_INTEGER_4_HUGE as default record length. * io/file_pos.c (unformatted_backspace): Handle different size record markers. * io/transfer.c (us_read): Likewise. (us_write): Likewise. (next_record_r): Likewise. (write_us_marker): Likewise. (next_record_w): Likewise. 2006-03-31 Thomas Koenig <Thomas.Koenig@online.de> PR fortran/19303 Backport from mainline * gfortran.dg/record_marker_1.f90: New test case. * gfortran.dg/record_marker_2.f: New test case. * gfortran.dg/record_marker_3.f90: New test case. Added: branches/gcc-4_1-branch/gcc/testsuite/gfortran.dg/record_marker_1.f90 branches/gcc-4_1-branch/gcc/testsuite/gfortran.dg/record_marker_2.f branches/gcc-4_1-branch/gcc/testsuite/gfortran.dg/record_marker_3.f90 Modified: branches/gcc-4_1-branch/gcc/fortran/ChangeLog branches/gcc-4_1-branch/gcc/fortran/gfortran.h branches/gcc-4_1-branch/gcc/fortran/invoke.texi branches/gcc-4_1-branch/gcc/fortran/lang.opt branches/gcc-4_1-branch/gcc/fortran/options.c branches/gcc-4_1-branch/gcc/fortran/trans-decl.c branches/gcc-4_1-branch/gcc/testsuite/ChangeLog branches/gcc-4_1-branch/libgfortran/ChangeLog branches/gcc-4_1-branch/libgfortran/io/file_pos.c branches/gcc-4_1-branch/libgfortran/io/open.c branches/gcc-4_1-branch/libgfortran/io/transfer.c branches/gcc-4_1-branch/libgfortran/libgfortran.h branches/gcc-4_1-branch/libgfortran/runtime/compile_options.c
Fixed on 4.1. Closing.