Re: memory ownership for descriptors

[Tobias Burnus <burnus at net-b dot de>]

Hi Tom,

Thomas Breuel wrote:
> I am writing a ctypes-based interface to gfortran; this allows
> GFortran functions and subroutines to be called with NumPy
> multidimensional arrays.  Passing arrays in already works.
>   

Just for completeness:
"you can use the Chasm Language Interoperability Tools,
http://chasm-interop.sourceforge.net/, which provide an interface to GNU
Fortran's array descriptor." (GNU Fortran manual, 7.1.5)

> The part I'm not sure about is how allocation is handled.  As far as I
> can tell, the first word in the array descriptor is a pointer to
> malloc'ed data.  GFortran must have conventions for when that pointer
> gets allocated and freed.  Are those documented anywhere?
>   

Well, in case of a POINTER it is easy: ALLOCATE and DEALLOCATE are used
explicitly. (Ignoring the case of pointer assignment or nullifying whjch
do not free memory, but just change the object to which the pointer points).

For ALLOCATABLEs, it is the same, except that such objects get
automatically deallocated when they leave scope (or, if they are an
allocatable dummy argument with INTENT(OUT), when they enter the scope).
This is documented in the Fortran standard, cf.
http://gcc.gnu.org/wiki/GFortranStandards

Automatic arrays of the form
  subroutine sub(n)
    integer :: n
    real :: AutomaticArray(n)
are allocated when the subroutine (or function) is entered and
automatically freed if the subroutine/function is left.


> Is there some way of telling the GFortran runtime that the pointer is
> shared with some other data structure and shouldn't get freed?
>   

Well, POINTER are not automatically freed, ALLOCATABLEs are. It should
never be possible to loose memory with ALLOCATABLEs - contrary to
POINTERS. ALLOCATABLEs (and other non-pointer variables) are also
"restricted" pointers (in the C sense) - unless you use TARGET.


> The third word in the descriptor is an integer indicating the type,
> kind, and rank of the array?  I've figured out the common cases, but
> is the precise meaning documented somewhere?
>   

Documentation: Not yet, there is only a bugreport for it.

If you use  -fdump-tree-original, you get a dump file, which shows the
internal tree generated by gfortran, which helps to see what happens.
(Unfortunately, it does not show most declarations.)

Look at libgfortran/libgfortran.h for some more insight:

typedef struct descriptor_dimension
{
  index_type _stride;
  index_type _lbound;
  index_type _ubound;
}
descriptor_dimension;

#define GFC_ARRAY_DESCRIPTOR(r, type) \
struct {\
  type *data;\
  size_t offset;\
  index_type dtype;\
  descriptor_dimension dim[r];\
}

typedef GFC_ARRAY_DESCRIPTOR (GFC_MAX_DIMENSIONS, GFC_INTEGER_4)
gfc_array_i4;
...


> Two comments:
> (1) For the future evolution of the GFortran array descriptor, ideal
> would be if there were hooks in the array descriptor for allocation
> and deallocation of its memory, as well as an extra void * for a
> hook-related context; that way, it would actually be possible to have
> Fortran interoperate with arbitrary arrays from other languages:
> Fortran could be passed a C++ or NumPy numerical array and
> allocate/deallocate/resize it without any data needing to be copied.
>   

It is also a bit unclear what you want to achieve. In principle, in to
deallocate Fortran memory, you just  "free (array->data); array->data =
NULL;" it - for reallocation, you also need to update the bounds. Thus,
I think you are mostly talking about preserving extra information of
non-Fortran allocated memory. But then, I do not see why you want to
have a hook and not just some space to store this extra information
(which can also contain some hook information). I don't think one will
ever use a hook to for deallocating memory via Fortran. If one needs to
do something special, one can still call the external function from Fortran.

> (2) I also looked at TR 29113 as a possible future solution, since it
> defines standard C access to Fortran array descriptors, but it isn't
> really a good solution since it doesn't allow ctypes-style transparent
> access to existing Fortran libraries; access with TR 29113 always
> seems to involve invoking the compiler

Well, the reason is that TR 29113 does not want to require the compiler
to change the Fortran implementation and just adds wrapper functions.
For gfortran's array descriptor update (see wiki link above), the idea
is to follow the structure of TR 29113, but add some trailing fields -
though this is not yet set in stone. Thus, some of the Fortran<->C
descriptor operations of TR 29113 will become no ops.

Note additionally that TR 29113 will still see some changes. If you have
some good suggestions, feel free to send them the Interop committee (->
http://j3-fortran.org/mailman/listinfo/interop-tr ); the suggestion
should follow the requirements outlined at
ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1820.txt and should not require
the compiler to change its internal representation. (Many compiler will
use a somewhat different internal representation, though gfortran will
probably follow TR 29113 with some added fields, which is allowed, cf.
ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1818.txt)

Tobias