Re: gfortran Coarray status – compiler, library & GSoC

[Maxim Kuvyrkov]

Tobias,

Thanks for a great update!

--
Maxim Kuvyrkov
www.linaro.org



On Jun 26, 2014, at 6:50 AM, Tobias Burnus <burnus@net-b.de> wrote:

> Dear all,
> 
> I though it would be useful to give the current status of GCC's/gfortran's corray support – what has been recently done, what's planed for the immediate future and what has still to be done. That's to give a general status but also
> to highlight what Alessandro did during his GSoC.
> 
> Coarrays are a PGAS (Partitioned Global Address Space) extension to Fortran, existing in since several years as vendor extension implemented in the Cray compiler and since Fortran 2008 in (modified form) part of the Fortran standard (ISO/IEC 1539-1). Some additional features have been postponed to a later Technical Specification (TS 18508), which is currently being drafted (e.g. at this weeks J3/WG5 meeting). Most parts of the TS are rather settled, other are still a bit in a flux. Coarrays work on separate processes called images and all communication is explicit; data transfer happens by specifying a remote image index when accessing a coarray variable.
> 
> On the GCC side, some initial support has been added with GCC 4.6 – and version by version a bit extended. GCC comes with two modes, -fcoarray=single which in the compiler simplifies all coarray constructs for a single image. And -fcoarray=lib, which inserts library calls to a communication library.
> 
> Until early spring this year, coindexed communication was not supported – and the only existing library (libcaf_single) handled only a single process/image, which is kind of boring.
> 
> This spring, it changed a bit by Alessandro starting to implement a multi-image communication library and exploring different strategies and communication libraries to hook on. (That work started before GSoC and continued as GSoC project.) And me adding the necessary calls to the compiler itself. The library work started on a nonpublic git repository, the compiler work on the fortran-caf branch.
> 
> 
> Current status on the compiler side (GCC 4.10 trunk): Full support of Fortran 2008 and of the TS18508 collectives co_sum/co_min/co_max except for:
> – No support yet for allocatable/pointer components of coarrays (i.e. no remote access to those via explicit component reference or via intrinsic assignment of the coarray, which should copy over allocatable components)
> – No locking, critical and atomics support
> – A few known bugs, e.g. with polymorphic coarrays in corner cases. I assume more corner cases will be found once there are more users.
> 
> Current status for libcaf_single as shipping (and automatically build) with GCC 4.10: Besides the points above, the following is not yet working:
> – Coarrays with Vector subscripts.
> 
> 
> But now to the multi-image work: Alessandro has started to implement versions based on MPI v2.0 (Message Passing Interface), ARMCI and GASNet. MPI implementations are the most widely used means to do communication on distributed-memory systems; MPI is well defined and every high-performance computation vendor has an optimized version. MPICH and Open MPI are also shipping with every Linux distribution. ARMCI and GASNet are libraries designed for PGAS application; they have hence the potential to be better optimized. ARMCI is quite simple to install and to use, however, we quickly encountered nonworking functions (either wrong use from our side [the forum is not very responsive] – or functionality either not implemented or simply buggy); thus, the development of that version has stalled from our side. GASNet is more complicated to use as every communication means ("conduit") comes with a different header and link flags; in addition, GASNet seems to require that one knows at start up how much memory the coarrays need. But on the other hand, it often was a tad faster than MPI. Thus, MPI is the work-horse library both for the laptop and for systems with a couple of thousands of CPUs while GASNet is an alternative for special cases. And the ARMCI version will no longer be developed by us, but stays in case someone wants to pick up the task.
> 
> Library status for libcaf_MPI: All what GCC 4.10 supports, except:
> – Coarrays with Vector subscripts.
> – Character kind conversion (unicode <-> 8byte during assignment)
> – Numeric type conversion (e.g. assignment of an integer array to a complex array)
> – co_sum/co_min/co_max: Some types are not yet supported, e.g. 128bit reals and character strings, which both do not map to an MPI type (which some stub implementation to handle also those).
> – Optimization for strided access still has room for improvement [btw: strides are important for halo exchange]. The basic support handles them element wise. There is a draft implementation which handles them more efficiently, but is still has some issues (excessive memory use) either it has to be refined or a different approach to be used.
> 
> Library status for ARMCI/GASNet: As I broke the API when moving the patches from GCC's fortran-caf branch to GCC 4.10, the API for those has still to be updated. Additionally and besides the items of MPI above, collectives and string padding/trimming are not yet implemented.
> 
> The MPI/GASNet/ARMCI libraries are supposed to get publicly released after Alessandro has returned from the this week's J3/WG5 meeting. By then, the API for ARMCI/GASNet should have been updated. Afterward, the idea is that the development is done in the public, which is admittedly overdue, but got delayed due to him and me having been busy to get the code running – and as the initial project didn't start as GSoC project.
> 
> On the pro side, most coarray programs now work and the performance of code used as benchmark was very encouraging, including providing better performances in some cases than one competitor; I think in very few special cases even better than two competitors. (If one ignores some academic compilers, never released compilers and stalled compilers, only two other compilers support coarrays.) I think Alessandro will share some numbers next week.
> 
> On the library side, the next steps are to do the API update for ARMCI/GASNet, to finish the missing bits for single and MPI and also add missing bits to GASNet. And to explore ways to speed up strided access without exploding memory requirements.
> 
> On the compiler side, I hope that I get soon my pending patches reviewed. Next is planned to iron out some wrinkles in the polymorphism support, write some numeric-type-conversion test cases and to update/write the API documentation. And then comes the next big project: Supporting allocatable/pointer components of coarrays (nonpolymophic and polymorphic) – in a way that also TS18508's co_broadcast and co_reduce will be supported.
> 
> 
> Final remark: The coarray implementation is such that the compiler just calls some library functions and leaves the rest to the library. That way, one can relatively simple add support for a different library, starting from an existing one. Or one can modify the library to test new strategies or have a debug version of the library, which provides some run-time checks. I hope that this opportunity will be used by others to try different strategies and to contribute to the library and/or to the compiler itself.
> 
> Tobias