Coarrays via GNU Fortran's Coarray Communication Library

Contents

Coarrays via GNU Fortran's Coarray Communication Library

Coarray Compilation Modes

GNU Fortran currently supports three coarray modes, which can be selected via the -fcoarray= flag:

none: The default, which prints an error when a coarray construct is encountered
single: Optimized version for a single image, which allows for fast serial programs
lib: A communication-library-based coarray version, described on this page

A shared-memory version is considered.

Coarray Communication Libraries

Currently, four GNU Fortran coarray communication libraries exist. Part of the compiler is

A single-image library consisting of stubs, which allows to use a single image without recompiling and is also useful for debugging. (Using -fcoarray=single will produce faster code, however.)

The single library version is automatically installed with GCC. Thus it can be directly linked with -lcaf_single. The library is located in the libgfortran/caf of the GCC source code.

The OpenCoarrays (implementation status) project provides

A MPI version which is a wrapper to calls to a library implementing the Message Passing Interface (MPI). The current version is based on the one-sided communication (RMA) of MPI 2 and MPI 3.
A GASNet version using GASNet as communication library
An ARMCI version using the ARMCI library, which is part of the Global Array Toolkit. $/!\$ The ARMCI version has still to be updated for the ABI changes and, hence, should not be used.

By default, all libraries use static linkage. However, using dynamic linkage and the same shared-library name, it is possible to switch the library at execution time of the program. However, static linkage is recommended.

Compiling Coarray Programs

For a complete description of clean and convenient ways to compile coarray programs, see Getting Started.

In brief:

Simply compile the Fortran files as usually using the -fcoarray=lib flag.

For the single version, simply add -lcaf_single when linking the program. If you want to use your self-compiled version, you can also simply link single.o.
For the MPI version, you need to also link the MPI library; the easiest is to run mpif90 to link the files. Example:
```
  mpif90 *.o -lcaf_mpi
```
$/!\$ You may need to use an environment variable or a command-line option to make sure MPI uses the current gfortran version. For Open MPI use, e.g., OMPI_FC=gfortran and for MPICH2, e.g., -f90=gfortran. Note, however, that this still requires that the MPI library is compatible to the current gfortran version; if in doubt, recompile
For the ARMCI version it depends on the backend; with the MPI backend, use
```
  mpif90 *.o -lcaf_mpi -larmci
```

Running Coarray Programs

For the single version, simply run the program as usual.
For the MPI version and ARMCI with MPI backend: Start the program as you would start any other MPI program; for instance use
```
mpiexec -n 10  ./myCoarrayProgram
```

Combining Coarray Parallelization with OpenMP, MPI, pthreads, etc.

In principle, combining coarrays with other means of parallelization should work, but it is the users responsibility to avoid race conditions and other issues.

For the MPI version of the coarray library: Make sure that the user code does not call MPI_Init if the library is already intialized; e.g.

integer :: ierror
logical :: init
call MPI_Initialized (init, ierror)
if (.not. init) call MPI_Init (ierror)

Coarray Fortran with a Non-Fortran Main Program (for -fcoarray=lib)

If the main program is not written in Fortran, the initialization and finalization of the coarray communication library needs to be called manually. The other prototypes are listed for references and should usually not be needed. The value of the named constants and function prototypes can be found in the C header file libcaf.h; also the single-image implementation and the multi-image MPI/GASNet/ARMCI versions can be used as starting point for new libraries.

GCC Coarray ABI Documentation

List of additional ABIs, not yet covered in the documentation:

Calling SYNC ALL

The SYNC ALL statement has the following prototype:

void
_gfortran_caf_sync_all (int *stat, char *errmsg, int errmsg_len);

stat (intent(out)): If not NULL: set to 0 on success; set to STAT_STOPPED_IMAGE or other nonzero value on error
errmsg (intent(inout)): If not NULL: On error, it gets assigned an error message
errmsg_len (intent(in)): The string length of errmsg - or 0.

Calling SYNC IMAGES

The SYNC IMAGES statement has the following prototype:

void
_gfortran_caf_sync_images (int count, int images[],
                           int *stat, char *errmsg, int errmsg_len);

count (intent(in)): Number of images passed by the next argument. -1 indicates SYNC IMAGES(*), 0 is a zero-sized array.
images (intent(in)): Array of image numbers to be used for synchronizing.
stat (intent(out)): If not NULL: Set to 0 on success, set to STAT_STOPPED_IMAGE or other nonzero value on error
errmsg (intent(inout)): If not NULL: On error, it gets assigned an error message - otherwise neither read nor set
errmsg_len (intent(in)): The string length of errmsg - or 0

=== Calling SYNC MEMORY === $/!\$ The implementation is still in a flux such that the ABI might still change; hence, the implementation support in different libraries may vary and the documentation tends to be incomplete or already outdated.

This memory barrier is implemented using __sync_synchronize.

The STOP statement

The STOP statement is handled as in serial programs; if the file has been compiled with -fcoarray=lib, before the stop statement _gfortran_caf_finalize is invoked.

The ERROR STOP statement

For the ERROR STOP statement, the library is called and a graceful stop is tried before ending the program forcefully. The functions do not return. Two library functions are implemented, their prototype are

void
_gfortran_caf_error_stop_str (const char *string, int32_t len);

void
_gfortran_caf_error_stop (int32_t error)

string (intent(in)): an string with an error message; it is written in the ERROR STOP string to stderr
len (intent(in)): the length of string.
error (intent(in)): an error number used, if possible, as exit status code and in ERROR STOP string, written to stderr.

Implementation Details

Argument handling

If a coarray is passed as actual argument to a coarray dummy, additional information needs to be transferred.

For descriptor-free arrays and for assumed-shape coarray dummies: There are two hidden arguments, the "token" and the offset (of type ptrdiff_t). The latter contains the offset between the address of the (first element of) object which was passed and the address which has been saved in the token. Think of "call sub(caf(2:8))" or "call sub(caf%comp2)" for examples where this can occur.
For allocatable coarrays, which use a descriptor: The descriptor simply contain besides rank-elements of the array-bound dimension triplets also corank-elements (dim[0] ...dim[rank-1] and dim[rank]...dim[rank+corank-1]); additionally the "token" is stored after the dimension triplets.

Front-end internal representation

Descriptor-free coarrays are arrays of the type GFC_ARRAY_TYPE_P; scalar types are normal scalars with the language-specific node attached. In this lang-specific node, the corank (GFC_TYPE_ARRAY_CORANK), the cobounds (GFC_TYPE_ARRAY_LBOUND and GFC_TYPE_ARRAY_UBOUND), token (GFC_TYPE_ARRAY_CAF_TOKEN) and offset (GFC_TYPE_ARRAY_CAF_OFFSET) is saved.

Allocatable coarrays use a descriptor - contrary to normal allocatables, also scalar coarrays have one. The descriptor contains additional elements as outlined above.

The passed token and offset arguments for assumed-shape coarrays are stored in the language-specific declaration (DECL_LANG_SPECIFIC) as GFC_DECL_TOKEN and DECL_LANG_SPECIFIC.

The token is an opaque object (of type "void*"), which contains in some direct or indirect way the base address of the coarray. The details are left to the library implementation. Possible choices would be the base address itself (e.g. for libcaf_single.a) or the base address of the coarray on all images (e.g. for libcaf_mpi.a).