gfortran Coarray status – compiler, library & GSoC

[Tobias Burnus]

Dear all,

I though it would be useful to give the current status of 
GCC's/gfortran's corray support – what has been recently done, what's 
planed for the immediate future and what has still to be done. That's to 
give a general status but also
to highlight what Alessandro did during his GSoC.

Coarrays are a PGAS (Partitioned Global Address Space) extension to 
Fortran, existing in since several years as vendor extension implemented 
in the Cray compiler and since Fortran 2008 in (modified form) part of 
the Fortran standard (ISO/IEC 1539-1). Some additional features have 
been postponed to a later Technical Specification (TS 18508), which is 
currently being drafted (e.g. at this weeks J3/WG5 meeting). Most parts 
of the TS are rather settled, other are still a bit in a flux. Coarrays 
work on separate processes called images and all communication is 
explicit; data transfer happens by specifying a remote image index when 
accessing a coarray variable.

On the GCC side, some initial support has been added with GCC 4.6 – and 
version by version a bit extended. GCC comes with two modes, 
-fcoarray=single which in the compiler simplifies all coarray constructs 
for a single image. And -fcoarray=lib, which inserts library calls to a 
communication library.

Until early spring this year, coindexed communication was not supported 
– and the only existing library (libcaf_single) handled only a single 
process/image, which is kind of boring.

This spring, it changed a bit by Alessandro starting to implement a 
multi-image communication library and exploring different strategies and 
communication libraries to hook on. (That work started before GSoC and 
continued as GSoC project.) And me adding the necessary calls to the 
compiler itself. The library work started on a nonpublic git repository, 
the compiler work on the fortran-caf branch.


Current status on the compiler side (GCC 4.10 trunk): Full support of 
Fortran 2008 and of the TS18508 collectives co_sum/co_min/co_max except for:
– No support yet for allocatable/pointer components of coarrays (i.e. no 
remote access to those via explicit component reference or via intrinsic 
assignment of the coarray, which should copy over allocatable components)
– No locking, critical and atomics support
– A few known bugs, e.g. with polymorphic coarrays in corner cases. I 
assume more corner cases will be found once there are more users.

Current status for libcaf_single as shipping (and automatically build) 
with GCC 4.10: Besides the points above, the following is not yet working:
– Coarrays with Vector subscripts.


But now to the multi-image work: Alessandro has started to implement 
versions based on MPI v2.0 (Message Passing Interface), ARMCI and 
GASNet. MPI implementations are the most widely used means to do 
communication on distributed-memory systems; MPI is well defined and 
every high-performance computation vendor has an optimized version. 
MPICH and Open MPI are also shipping with every Linux distribution. 
ARMCI and GASNet are libraries designed for PGAS application; they have 
hence the potential to be better optimized. ARMCI is quite simple to 
install and to use, however, we quickly encountered nonworking functions 
(either wrong use from our side [the forum is not very responsive] – or 
functionality either not implemented or simply buggy); thus, the 
development of that version has stalled from our side. GASNet is more 
complicated to use as every communication means ("conduit") comes with a 
different header and link flags; in addition, GASNet seems to require 
that one knows at start up how much memory the coarrays need. But on the 
other hand, it often was a tad faster than MPI. Thus, MPI is the 
work-horse library both for the laptop and for systems with a couple of 
thousands of CPUs while GASNet is an alternative for special cases. And 
the ARMCI version will no longer be developed by us, but stays in case 
someone wants to pick up the task.

Library status for libcaf_MPI: All what GCC 4.10 supports, except:
– Coarrays with Vector subscripts.
– Character kind conversion (unicode <-> 8byte during assignment)
– Numeric type conversion (e.g. assignment of an integer array to a 
complex array)
– co_sum/co_min/co_max: Some types are not yet supported, e.g. 128bit 
reals and character strings, which both do not map to an MPI type (which 
some stub implementation to handle also those).
– Optimization for strided access still has room for improvement [btw: 
strides are important for halo exchange]. The basic support handles them 
element wise. There is a draft implementation which handles them more 
efficiently, but is still has some issues (excessive memory use) either 
it has to be refined or a different approach to be used.

Library status for ARMCI/GASNet: As I broke the API when moving the 
patches from GCC's fortran-caf branch to GCC 4.10, the API for those has 
still to be updated. Additionally and besides the items of MPI above, 
collectives and string padding/trimming are not yet implemented.

The MPI/GASNet/ARMCI libraries are supposed to get publicly released 
after Alessandro has returned from the this week's J3/WG5 meeting. By 
then, the API for ARMCI/GASNet should have been updated. Afterward, the 
idea is that the development is done in the public, which is admittedly 
overdue, but got delayed due to him and me having been busy to get the 
code running – and as the initial project didn't start as GSoC project.

On the pro side, most coarray programs now work and the performance of 
code used as benchmark was very encouraging, including providing better 
performances in some cases than one competitor; I think in very few 
special cases even better than two competitors. (If one ignores some 
academic compilers, never released compilers and stalled compilers, only 
two other compilers support coarrays.) I think Alessandro will share 
some numbers next week.

On the library side, the next steps are to do the API update for 
ARMCI/GASNet, to finish the missing bits for single and MPI and also add 
missing bits to GASNet. And to explore ways to speed up strided access 
without exploding memory requirements.

On the compiler side, I hope that I get soon my pending patches 
reviewed. Next is planned to iron out some wrinkles in the polymorphism 
support, write some numeric-type-conversion test cases and to 
update/write the API documentation. And then comes the next big project: 
Supporting allocatable/pointer components of coarrays (nonpolymophic and 
polymorphic) – in a way that also TS18508's co_broadcast and co_reduce 
will be supported.


Final remark: The coarray implementation is such that the compiler just 
calls some library functions and leaves the rest to the library. That 
way, one can relatively simple add support for a different library, 
starting from an existing one. Or one can modify the library to test new 
strategies or have a debug version of the library, which provides some 
run-time checks. I hope that this opportunity will be used by others to 
try different strategies and to contribute to the library and/or to the 
compiler itself.

Tobias