Fwd: Re: the best practice to distribute Fortran libraries

[Tobias Burnus <tobias dot burnus at physik dot fu-berlin dot de>]

I forget to CC the list.

----- Forwarded message -----

Hi OndÅej,

OndÅej ÄertÃk wrote:
> It seems to me, that it is commonly accepted that:
>
> * .mods files are compiler dependent (even different versions of the
> same compiler can produce incompatible .mods)

Yes, that's unfortunately the case. That's similar to C/C++'s
precompiled headers, except that there is no real "header" file but
just normal Fortran files. (Fortran 2008's submodules help a bit,
though they are also a bit different - and most compilers do not
support submodules, yet.)

Mod files are essentially only (text or binary) dumps of the
internal structure. In case of gfortran, they are compatible
between releases (4.5.0 and 4.5.2) but not between minor versions
(4.5 vs 4.6); new features demand new information, which is with
few exceptions not compatible with the old structure. (Sometimes
but not often, additions do not require a new format.)


> * the .so library should (safely) also only be used by the same
> compiler (even the same version of the compiler?) due to name
> mangling, fortran runtime library and maybe other issues

Yes. For simple code without modules, one often has some
interoperability. But one needs to be careful; in particular:
- Tailing underscores (zero, one (most common), two)
- calling convention: Whether real is passed as double, whether
  a function returns the value as first argument, etc. Cf. -ff2c
  option in gfortran.
- Logical: Special Intel vs. gfortran problem: Intel has "-1" as
  .true. and gfortran "1". With higher optimization levels,
  gfortran only looks at one bit, hence -1 is .false.

Especially the second item, is difficult to test for.

I think the reason for the lacking interoperability is that there
are many Fortran compilers - someone counted 9 commercial ones for
the x86-64 architecture - and that it is common to recompile
libraries to get maximal performance. I think the only ABI which
was a bit wider supported was the one of f2c.


Regarding the run-time library compatibility: Most of the time
the library is backward compatibile. Thus, libgfortran of GCC 4.7
is compatible with 4.6, 4.5, 4.4 and 4.3; 4.5 is compatible with
4.4 and 4.3. But 4.2 has a different .so version and is
incompatible with with either 4.1 and 4.3.


> * one can use the iso_c_binding module to be able to safely call
> things from other languages (and compilers?), but that means only raw
> C functions (no modules, only C style arrays)

Yes. Though, you can use modules:
  module my
    interface
      subroutine sub1() bind (C)
      end subroutine sub1
    end interface
  contains
    subroutine sub2() bind(c, name="my_name")
    end subroutine sub2
  end module my

either just as wrapper (like for "sub1") or to implement the functions.
The binding names live in the global namespace, thus you need to make
sure yourself that you do not get some symbol clashes (manual mangling).

Regarding the arrays: It is true that you cannot have assumed-shape arrays
("A(:,:)") but using "A(n,m)" is also quite flexible. I think the bigger
issue is that pointer arrays and allocatable are not possible.

(The array issue is solved by an upcoming technical report, however, it
will takes a couple of years until it is implemented [1].)

Otherwise, using the C binding support should be safe, unless you need
to support compilers older than - let's say - 3-5 years.


> As such, it seems to me, that if I want my code to be used by people
> easily, they should really just take the .f90 source code and always
> compile it together with the rest of their program with the same
> compiler. Right?

That's right. It also allows to optimize for the system at hand
(cf. GCC's "-march=native"). You could also compile for a few selected
compiler (and versions) and offer the source as add on. The source
additionally allows the users to compile for more special machines
such as BlueGene, which you probably do not have access too.

(The situation for closed-source software is more difficult as they
have to compile for several compilers/system.)

So I will always distribute the .f90 source files. Now the question is
how to organize it. Possible options:


> 1) users just include the .f90 files in their project and use it just
> like any other file they wrote themselves

I think that only makes sense for very small (read: single file)
libraries.

In most cases, it is better to have the library in a separate directory
with its own makefile.


> 2) my .f90 files will be compiled into an .so  (or .a) library, and
> the user will use the compiler generated .mod files to access the
> library.

I think that should be the standard way for larger libraries. Probably,
a  .a file should be enough in many cases as it should be faster and
the number of programs sharing the library on any given system should be
fairly low.


> 2) my .f90 files will be compiled into an .so  (or .a) library, and
> the user will use the compiler generated .mod files to access the
> library. In particular, the best way to do it (it seems to me) is to
> create a simple dftatom.f90 file, like this one:

[USEing all other modules and exporting only selected entities]

I think as long as using parts of the library does not make sense and
as long as the the library does not have an excessive amount of symbols,
having a single module, which the user use-associates is the best.

Otherwise, I would offer a handful of modules.


> 3) my .f90 files will be compiled into an .so  (or .a) library, and I
> will provide a simple interface .f90 file, that users will include in
> their projects. I would have to write an interface to all of the
> functions in my library. Users will include this interface file, and
> then link with the .so library.

In a way, that's how one does with submodules:

module dftatom
  interface
    module subroutine stop_error(str)
      character(len=*) :: str
    end subroutine
  end interface
end module dftatom

and then:

submodule (dftatom) actual_implementation
contains
  module subroutine stop_error
    write(*,*) 'ERROR: '//trim(str)
    stop 1
  end subroutine
end submodule actual_implementation

The compiler ensures that the interface is the same (or one does
not repeat it, as I did here). Additionally, it is sufficient to
have the file contain "module dftatom" - the file containing the
submodules is only needed if the compiler produces a different ABI;
it is not needed if only the .mod format has changed.

However, to my knowledge, currently only the Cray compiler
supports submodules. [2]


> As such, I like the option 2) the most.
Ditto.

> Now let's say for the Debian and Fedora packages, one would create
> both .a and .so libraries (for static and dynamic linking) and then
> put the .mod files somewhere (eventually there will be some standard
> place for it).

Yes, but the question is where. Currently, gfortran - like most other
Fortran compilers - searches .mod files in /usr{/local,}/include, which
is not the best place for compiler vendor/version-dependent files.

Cf. http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49138 for GCC.

> So can I view the .mod files just like C .h files (and treat them as
> such), except that they are not human readable (to remind me, that
> they are platform and compiler dependent)?

Except for the big "except": Yes.

Tobias

[1] http://gcc.gnu.org/wiki/GFortranStandards#TR_29113_Further_Interoperability_of_Fortran_with_C
[2] http://fortranwiki.org/fortran/show/Fortran+2008+status