Coarrays via GNU Fortran's Coarray Communication Library


GNU Fortran currently supports three coarray modes, which can be selected via the -fcoarray= flag:

Planned is also a shared-memory version.

Currently, three GNU Fortran coarray communication libraries exist:

/!\ The whole implementation is in an embryonic state, most features do not yet work! Both the implementation in the compiler itself (front end) as also in the MPI and ARMCI libraries still require quite some work.

Obtaining and Compiling the Coarray Communication Libraries

The single library version is automatically installed with GCC. Thus it can be directly linked with -lcaf_single. For MPI or if you want to have a special version of libcaf_single, follow the instructions below.

The libraries are located in the GCC trunk (4.7) source code in the libgfortran/caf directory. (CAF = Co-Array Fortran, the initial name of the feature.) Note that the MPI and ARMCI versions will currently not be automatically compiled and installed

  1. Obtain the files: by checking out the subversion or GIT repository, using the snapshot tar balls or directly by clicking on the links below. The following files are needed:

  2. Compiling the files
    gcc -c -O2 single.c
    mpicc -c -O2 mpi.c
    mpicc -c -O2 armci.c
    • Note 1: The files might offer different versions, depending on some pre-processor flag. For instance, via GFC_CAF_CHECK (-DGFC_CAF_CHECK) some run-time checks can be enabled.

      Note 2: Hereby, it has been assumed that both versions, single and MPI, should be compiled and that you have an MPI compiler installed, where mpicc invokes the proper compiler. The created object files (.o) can directly be linked to the coarray programs - or one proceeds to step 3. Note 3: It has been assumed that ARMCI version uses MPI as backend and that the header files are in the standard include path.

  3. Creating a library. Run now:
    ar rcv libcaf_single.a single.o
    ranlib libcaf_single.a
    ar rcv libcaf_mpi.a mpi.o
    ranlib libcaf_mpi.a
    ar rcv libcaf_armci.a armci.o
    ranlib libcaf_armci.a
  4. Creating a shared library
    mpicc -O2 -fPIC -shared -Wl,-soname, -o mpi.c
    mpicc -O2 -fPIC -shared -Wl,-soname, -o armci.c
    Usually a static library will produce faster code. However, if you want to use your program on several systems without recompiling (e.g. because it is closed source) or if you regularly test several MPI versions or ARMCI with different backends, using a shared library might be better.

    If you want to switch between different CAF communication libraries, you can also generate a "" - and then use the LD_LIBRARY_PATH to switch between, e.g., the single.c and, e.g., the mpi.c version. However, be prepared to get confused ...

Compiling Coarray Programs

Simply compile the Fortran files as usually using the -fcoarray=lib flag.

Running Coarray Programs

Combining Coarray Parallelization with OpenMP, MPI, pthreads, etc.

In principle, combining coarrays with other means of parallelization should work, but it is the users responsibility to avoid race conditions and other issues.

For the MPI version of the coarray library: Make sure that the user code does not call MPI_Init if the library is already intialized; e.g.

integer :: ierror
logical :: init
call MPI_Initialized (init, ierror)
if (.not. init) call MPI_Init (ierror)

Coarray Fortran with a Non-Fortran Main Program (for -fcoarray=lib)

If the main program is not written in Fortran, the initialization and finalization of the coarray communication library needs to be called manually. The other prototypes are listed for references and should usually not be needed. The value of the named constants and function prototypes can be found in the C header file libcaf.h.

/!\ Note that this documentation is still in a flux and incomplete. The source code of the compiler and the libraries is the authoritative source; however, the API is not yet finished and will keep changing for a while.

Initializing the Coarray Communication Library

The initialization happens automatically if the main program is written in Fortran and its file is compiled with -fcoarray=lib. The initialization function has the following prototype

_gfortran_caf_init (int *argc, char ***argv,
                    int *this_image, int *num_images);

The image numbers returned by _gfortran_caf_init need to be stored in global variables with the following type and name

const int _gfortran_caf_this_image;
const int _gfortran_caf_num_images;

Note: As static coarrays are registered at start up time (via constructor functions [__attribute__((constructor))]), the _gfortran_caf_init calls comes after the actual library initialization; nevertheless, it should happen early and should be followed by a SYNC ALL call to make sure the initialization and coarray registration has happened on all images. Nevertheless, the _gfortran_caf_init is required and should include the command-line arguments.

Closing down the Coarray Communication Library

The following command should be called at the end of the program. It is automatically invoked at the end of the program, if the main program is written in Fortran and compiled with -fcoarray=lib. It is also invoked before the STOP statement, if the file is compiled with -fcoarray=lib. The prototype is

_gfortran_caf_finalize (void);

Calling SYNC ALL

The SYNC ALL statement has the following prototype:

_gfortran_caf_sync_all (int *stat, char *errmsg, int errmsg_len);


The SYNC IMAGES statement has the following prototype:

_gfortran_caf_sync_images (int count, int images[],
                           int *stat, char *errmsg, int errmsg_len);


This memory barrier is implemented using __sync_synchronize.

The STOP statement

The STOP statement is handled as in serial programs; if the file has been compiled with -fcoarray=lib, before the stop statement _gfortran_caf_finalize is invoked.

The ERROR STOP statement

For the ERROR STOP statement, the library is called and a graceful stop is tried before ending the program forcefully. The functions do not return. Two library functions are implemented, their prototype are

_gfortran_caf_error_stop_str (const char *string, int32_t len);

_gfortran_caf_error_stop (int32_t error)

Registering coarrays

void *
_gfortran_caf_register (ptrdiff_t size, caf_register_t type, caf_token_t ***token,
                        int *stat, char *errmsg, int errmsg_len);

Note 1: For nonallocatable coarrays, this function needs to be called at startup of the program - or at least before any of the other images uses it. The registering must happen in the same order on all images. With gfortran, the registering happens via constructor functions (__attribute__((constructor))) such that usually a _gfortran_caf_register call precedes the  _gfortran_caf_init call.

Note 2: The deregistering for allocatable coarrays is done via an explicit call to _gfortran_caf_deregister, for nonallocatable ones, this is done in _gfortran_caf_finalize.

Deregistering coarrays

_gfortran_caf_deregister (const caf_token_t ***token, int *stat, char *errmsg, int errmsg_len);

Note: This function only applies to allocatable coarrays, the others are automatically deregistered in _gfortran_caf_finalize.

Sending data to a remote image

/!\ The following is still rather incomplete and not completely thought through. Expect changes and additions for missed cases. Recall that communication is possible from local to remote, from remote to local and remote to remote; there can be scalars and (simply) contiguous arrays but also arrays with strides and vectors. Asynchronous communication is possible for defining remote coarrays (be careful of references to the same coarray later in the same segment) and referencing multiple coarrays in the same statement.

Note that especially broadcasts (as special support is lacking) should be optimized by doing asynchronous communication - with a wait after the loop. Requires that RHS may not be modified during the loop and does not go out of scope (temporary variable!)

  do i = 1, num_images()
    ! if (i /= this_image())  ! (Optionally)
    settings[i] = settings
  end do

Sending to a scalar or to a contiguous array section

_gfortran_caf_send (const caf_token_t ***token, size_t offset, int image_index, const void *data, size_t size, bool asynchonous);


Waiting for asynchronous transactions

_gfortran_caf_send_wait (const caf_token_t ***token)


Implementation Details

Argument handling

If a coarray is passed as actual argument to a coarray dummy, additional information needs to be transferred.

Front-end internal representation

Descriptor-free coarrays are arrays of the type GFC_ARRAY_TYPE_P; scalar types are normal scalars with the language-specific node attached. In this lang-specific node, the corank (GFC_TYPE_ARRAY_CORANK), the cobounds (GFC_TYPE_ARRAY_LBOUND and GFC_TYPE_ARRAY_UBOUND), token (GFC_TYPE_ARRAY_CAF_TOKEN) and offset (GFC_TYPE_ARRAY_CAF_OFFSET) is saved.

Allocatable coarrays use a descriptor - contrary to normal allocatables, also scalar coarrays have one. The descriptor contains additional elements as outlined above.

The passed token and offset arguments for assumed-shape coarrays are stored in the language-specific declaration (DECL_LANG_SPECIFIC) as GFC_DECL_TOKEN and DECL_LANG_SPECIFIC.

The token is an opaque object (of type "void*"), which contains in some direct or indirect way the base address of the coarray. The details are left to the library implementation. Possible choices would be the base address itself (e.g. for libcaf_single.a) or the base address of the coarray on all images (e.g. for libcaf_mpi.a).

None: CoarrayLib (last edited 2013-05-07 15:22:04 by TobiasBurnus)