RFC: [GUPC] UPC-related front-end changes

[Gary Funck <gary at intrepid dot com>]

This email is a follow-up to an email with a similar title
(posted a year ago).  During that time period, we have worked
on making the changes suggested by Joseph Myers, Tom Tromey,
and other reviewers.  We have also implemented various bug fixes
and improvements.

This RFC focuses mainly on front-end and infrastructure changes.

Our goal with this RFC is to acquaint the reviewers with UPC and
the impact of UPC changes on the GCC front-end, and to gain consensus
that the changes are acceptable for incorporation into the GCC trunk.

Once we make further suggested changes, and have a consensus on this
batch of changes, I will send out RFC's for the "middle end"
(the lowering pass), "debugging" (UPC-specific DWARF extensions),
"runtime" (libupc) and "testing" RFC's.  Those additional RFC's
are likely to be more modular and will have less impact on
the GCC infrastructure.

The main review-related changes are:

* FIXMEs and TODOs were either fixed or cleaned up.

* The copyright and license notices were updated.

* The code was reviewed for conformance to coding standards and updated.

* Diagnostics now use appropriate format strings rather than building
up the strings with sprintf().

* Files in c-family/ no longer include c-tree.h to conform with modularization
improvements.

* Most of the #ifdef conditionals have been removed.  Some target macros
have been defined and documented in tm.texi.  We still have some questions
in this area due to some configuration issues that are unique to UPC
and will discuss those changes in a follow-up email.

* The code was reviewed to verify that it conforms with
current GCC coding practices and that it incorporates cleanups
done in the past several years.

* Comments were added to most new functions, and typos and
spelling errors in comments were fixed.

* Changes that appeared in the diffs that were unrelated to UPC
were removed or incorporated into the trunk.

* Tree node flags that record UPC qualifiers were moved from
their own flag byte into tree_base's "spare" bits.

* The linkage to the libupc library was changed to use the newly
defined method (used in libgomp/libgo for example) of including
library 'spec' files.  This led to a simplification where we no
longer needed to add UPC-specific spec. files in various target-specfic
config. dirs.

thanks,

- Gary

RFC: UPC-related Front-End Changes
----------------------------------

This document describes the UPC-related front-end
changes, and is provided as background for
review of the UPC changes implemented in
the GUPC branch.

The current GUPC branch is based upon a recent
version of the GCC trunk (175584), and has been
bootstrapped on x86_64/i686 Linux and IA64/Altix Linux.
Bootstraps on other platforms are in progress.

The GUPC branch is described here:
  http://gcc.gnu.org/projects/gupc.html

The UPC-related source code differences
are summarized here:
  http://gccupc.org/gupc-changes

In the discussion below, the changes are
excerpted in order to highlight important
aspects of the changes.

UPC's Shared Qualifier and Layout Qualifier
-------------------------------------------

The UPC language specification describes
the language syntax and semantics:
  http://upc.gwu.edu/docs/upc_specs_1.2.pdf

UPC introduces a new qualifier, "shared"
that indicates that the qualified object
is located in a global shared address space
that is accessible by all UPC threads.
Additional qualifiers ("strict" and "relaxed")
further specify the semantics of accesses to
UPC shared objects.

In UPC, a shared qualified array can further
specify a "layout qualifier" that indicates
how the shared data is blocked and distributed.

There are two language pre-defined identifiers
that indicate the number of threads that
will be created when the program starts (THREADS)
and the current (zero-based) thread number
(MYTHREAD).  Typically, a UPC thread is implemented
as an operating system process.  Access to UPC
shared memory may be implemented locally via
OS provided facilities (for example, mmap),
or across nodes via a high speed network
inter-connect (for example, Infiniband).

GUPC provides a runtime (libupc) that targets
an SMP-based system and uses mmap() to implement
global shared memory.  

Optionally, GUPC can use the more general and
more capable Berkeley UPCR runtime:
  http://upc.lbl.gov/download/source.shtml#runtime
The UPCR runtime supports a number of network
topologies, and has been ported to most of the
current High Performance Computing (HPC) systems.

The following example illustrates
the use of the UPC "shared" qualifier
combined with a layout qualifier.

    #define BLKSIZE 5
    #define N_PER_THREAD (4 * BLKSIZE)
    shared [BLKSIZE] double A[N_PER_THREAD*THREADS];

Above the "[BLKSIZE]" construct is the UPC
layout factor; this specifies that the shared
array, A, distributes its elements across
each thread in blocks of 5 elements.  If the
program is run with two threads, then A is
distributed as shown below:

    Thread 0	Thread 1
    --------	---------
    A[ 0.. 4]	A[ 5.. 9]
    A[10..14]	A[15..19]
    A[20..24]	A[25..29]
    A[30..34]	A[35..39]

Above, the elements shown for thread 0
are defined as having "affinity" to thread 0.
Similarly, those elements shown for thread 1
have affinity to thread 1.  In UPC, a pointer
to a shared object can be cast to a thread
local pointer (a "C" pointer), when the
designated shared object has affinity
to the referencing thread.

A UPC "pointer-to-shared" (PTS) is a pointer
that references a UPC shared object.
A UPC pointer-to-shared is a "fat" pointer
with the following logical fields:
   (virt_addr, thread, offset)

The virtual address (virt_addr) field is combined with
the thread number (thread) and offset within the
block (offset), to derive the location of the
referenced object within the UPC shared address space.

GUPC implements pointer-to-shared objects using
either a "packed" representation or a "struct"
representation.  The user can select the
pointer-to-shared representation with a "configure"
parameter.  The packed representation is the default.

The "packed" pointer-to-shared representation
limits the range of the various fields within
the pointer-to-shared in order to gain efficiency.
Packed pointer-to-shared values encode the three
part shared address (described above) as a 64-bit
value (on both 64-bit and 32-bit platforms).

The "struct" representation provides a wider
addressing range at the expense of requiring
twice the number of bits (128) needed to encode
the pointer-to-shared value.

UPC-Related Front-End Changes
-----------------------------

GCC's internal tree representation is
extended to record the UPC "shared",
"strict", "relaxed" qualifiers,
and the layout qualifier.

--- gcc/tree.h	(.../trunk)	(revision 175584)
+++ gcc/tree.h	(.../branches/gupc)	(revision 175683)
@@ -462,8 +462,11 @@ struct GTY(()) tree_base {
   unsigned packed_flag : 1;
   unsigned user_align : 1;
   unsigned nameless_flag : 1;
+  unsigned upc_shared_flag : 1;
+  unsigned upc_strict_flag : 1;
+  unsigned upc_relaxed_flag : 1;
 
-  unsigned spare : 12;
+  unsigned spare : 9;

@@ -2405,6 +2469,9 @@ struct GTY(()) tree_type_common {
   alias_set_type alias_set;
   tree pointer_to;
   tree reference_to;
+  /* UPC: for block-distributed arrays */
+  tree block_factor;

UPC defines a few additional tree node types:

--- gcc/upc/upc-tree.def	(.../trunk)	(revision 0)
+++ gcc/upc/upc-tree.def	(.../branches/gupc)	(revision 175683)
+/* UPC statements */
+
+/* Used to represent a `upc_forall' statement. The operands are
+   UPC_FORALL_INIT_STMT, UPC_FORALL_COND, UPC_FORALL_EXPR,
+   UPC_FORALL_BODY, and UPC_FORALL_AFFINITY respectively. */
+
+DEFTREECODE (UPC_FORALL_STMT, "upc_forall_stmt", tcc_statement, 5)
+
+/* Used to represent a UPC synchronization statement. The first
+   operand is the synchronization operation, UPC_SYNC_OP:
+   UPC_SYNC_NOTIFY_OP	1	Notify operation
+   UPC_SYNC_WAIT_OP	2	Wait operation
+   UPC_SYNC_BARRIER_OP	3	Barrier operation
+
+   The second operand, UPC_SYNC_ID is the (optional) expression
+   whose value specifies the barrier identifier which is checked
+   by the various synchronization operations. */
+
+DEFTREECODE (UPC_SYNC_STMT, "upc_sync_stmt", tcc_statement, 2)

The "C" parser is extended to recognize UPC's syntactic
extensions.

--- gcc/c-family/c-common.c	(.../trunk)	(revision 175584)
+++ gcc/c-family/c-common.c	(.../branches/gupc)	(revision 175683)
+   UPC is like C except that D_UPC is not set
+   C++ --std=c98: D_CONLY | D_CXXOX | D_OBJC | D_UPC
+   C++ --std=c0x: D_CONLY | D_OBJC | D_UPC
+
[...]
+  /* UPC keywords */
+  { "shared",		RID_SHARED,		D_UPC },
+  { "relaxed",		RID_RELAXED,		D_UPC },
+  { "strict",		RID_STRICT,		D_UPC },
+  { "upc_barrier",	RID_UPC_BARRIER,	D_UPC },
+  { "upc_blocksizeof",	RID_UPC_BLOCKSIZEOF,	D_UPC },
+  { "upc_elemsizeof",	RID_UPC_ELEMSIZEOF,	D_UPC },
+  { "upc_forall",	RID_UPC_FORALL,		D_UPC },
+  { "upc_localsizeof",	RID_UPC_LOCALSIZEOF,	D_UPC },
+  { "upc_notify",	RID_UPC_NOTIFY,		D_UPC },
+  { "upc_wait",		RID_UPC_WAIT,		D_UPC },


-- gcc/c-parser.c	(.../trunk)	(revision 161517)
+++ gcc/c-parser.c	(.../branches/gupc)	(revision 161914)
[...]
+/* These UPC parser functions are only ever called when
+   compiling UPC.  */
+static void c_parser_upc_forall_statement (c_parser *);
+static void c_parser_upc_sync_statement (c_parser *, int);
+static void c_parser_upc_shared_qual (c_parser *, struct c_declspecs *);
[...]
+        /* UPC qualifiers */
+	case RID_SHARED:
+	  attrs_ok = true;
+          c_parser_upc_shared_qual (parser, specs);
+	  break;
+	case RID_STRICT:
+	case RID_RELAXED:
+	  attrs_ok = true;
+	  declspecs_add_qual (specs, c_parser_peek_token (parser)->value);
+	  c_parser_consume_token (parser);
+	  break;
[...]
+  /* Process all #pragma's just after the opening brace.  This
+     handles #pragma upc, which can only appear just after
+     the opening brace, when it appears within a function body.  */
+  push_upc_consistency_mode ();
+  permit_pragma_upc ();
+  while (c_parser_next_token_is (parser, CPP_PRAGMA))
+    {
+      location_t loc ATTRIBUTE_UNUSED = c_parser_peek_token (parser)->location;
+      if (c_parser_pragma (parser, pragma_compound))
+        last_label = false, last_stmt = true;
+      parser->error = false;
+    }
+  deny_pragma_upc ();
[...]
+	case RID_UPC_FORALL:
+          gcc_assert (c_dialect_upc ());
+	  c_parser_upc_forall_statement (parser);
+	  break;
+        case RID_UPC_NOTIFY:
+          gcc_assert (c_dialect_upc ());
+	  c_parser_upc_sync_statement (parser, UPC_SYNC_NOTIFY_OP);
+	  goto expect_semicolon;
+        case RID_UPC_WAIT:
+          gcc_assert (c_dialect_upc ());
+	  c_parser_upc_sync_statement (parser, UPC_SYNC_WAIT_OP);
+	  goto expect_semicolon;
+        case RID_UPC_BARRIER:
+          gcc_assert (c_dialect_upc ());
+	  c_parser_upc_sync_statement (parser, UPC_SYNC_BARRIER_OP);
+	  goto expect_semicolon;
[...]
+	case RID_UPC_BLOCKSIZEOF:
+	case RID_UPC_ELEMSIZEOF:
+	case RID_UPC_LOCALSIZEOF:
+          gcc_assert (c_dialect_upc ());
+	  return c_parser_sizeof_expression (parser);
[...]

--- gcc/c-family/c-pragma.c	(.../trunk)	(revision 175584)
+++ gcc/c-family/c-pragma.c	(.../branches/gupc)	(revision 175683)
[...]
+/*
+ *  #pragma upc strict
+ *  #pragma upc relaxed
+ *  #pragma upc upc_code
+ *  #pragma upc c_code
+ */
+static void
+handle_pragma_upc (cpp_reader * ARG_UNUSED (dummy))
[...]

c-decl.c handles the additional UPC qualifiers
and declspecs.  The layout qualifier is handled here:

--- gcc/c-decl.c	(.../trunk)	(revision 175584)
+++ gcc/c-decl.c	(.../branches/gupc)	(revision 175683)
[...]
+  /* A UPC layout qualifier is encoded as an ARRAY_REF,
+     further, it implies the presence of the 'shared' keyword. */
+  if (TREE_CODE (qual) == ARRAY_REF)
+    {
+      if (specs->upc_layout_qualifier)
+        {
+          error ("two or more layout qualifiers specified");
+	  return specs;
+        }
+      else
+        {
+          specs->upc_layout_qualifier = qual;
+          qual = ridpointers[RID_SHARED];
+        }
+    }

In UPC, a qualifier includes both the traditional
"C" qualifier flags and the UPC "layout qualifier".
Thus, the pointer_quals field of a declarator node
is defined as a struct including both qualifier
flags and the UPC type qualifier, as shown below.
 
 	    /* Process type qualifiers (such as const or volatile)
 	       that were given inside the `*'.  */
-	    type_quals = declarator->u.pointer_quals;
+	    type_quals = declarator->u.pointer.quals;
+	    layout_qualifier = declarator->u.pointer.upc_layout_qual;
+	    sharedp = ((type_quals & TYPE_QUAL_SHARED) != 0);

UPC shared variables are allocated at runtime in the global
memory that is allocated and managed by the UPC runtime.
A separate link section is used as a method of assigning
virtual addresses to UPC shared variables.  The UPC
shared variable section is designated as a "no load"
section on systems that support that facility; in that
case, the linkage section begins at virtual address zero.
The logic below assigns UPC shared variables to
their own linkage section.

+    /* Shared variables are given their own link section on
+       most target platforms, and if compiling in pthreads mode
+       regular local file scope variables are made thread local. */
+    if ((TREE_CODE(decl) == VAR_DECL)
+        && !threadp && (TREE_SHARED (decl) || flag_upc_pthreads))
+      upc_set_decl_section (decl);

Various UPC language related checks and operations
are called in the "C" front-end and middle-end.
To insure that these operations are defined,
when linked with the other language front-ends
and compilers, these functions are stub-ed,
in a fashion similar to Objective C:

--- gcc/c-family/c-upc.h	(.../trunk)	(revision 0)
+++ gcc/c-family/c-upc.h	(.../branches/gupc)	(revision 175683)
[...]
+extern int count_upc_threads_refs (tree);
+extern void deny_pragma_upc (void);
+extern int get_upc_consistency_mode (void);
+extern int get_upc_pupc_mode(void);
+extern int is_multiple_of_upc_threads (tree);
+extern void permit_pragma_upc (void);
+extern void pop_upc_consistency_mode (void);
+extern int pragma_upc_permitted_p (void);
+extern void push_upc_consistency_mode (void);
+extern void set_upc_consistency_mode (int);
+extern void set_upc_threads_refs_to_one (tree *);
+extern tree upc_affinity_test (location_t, tree);
+extern tree upc_blocksizeof (location_t, tree);
+extern tree upc_build_shared_var_addr (location_t, tree, tree);
+extern tree upc_build_sync_stmt (location_t, tree, tree);
+extern int upc_check_decl_init (tree, tree);
+extern void upc_check_decl (tree);
+extern void upc_cpp_builtins (cpp_reader *);
+extern void upc_decl_init (tree, tree);
+extern int upc_diagnose_deprecated_stmt (location_t, tree);
+extern tree upc_elemsizeof (location_t, tree);
+extern tree upc_get_block_factor (const tree);
+extern tree upc_instrument_forall (location_t, int);
+extern int upc_is_null_pts_p (tree);
+extern tree upc_localsizeof (location_t, tree);
+extern tree upc_num_threads (void);
+extern tree upc_pts_diff (tree, tree);
+extern tree upc_pts_increment (location_t, enum tree_code, tree);
+extern tree upc_pts_int_sum (location_t, enum tree_code, tree, tree);
+extern tree upc_rts_forall_depth_var (void);
+extern tree upc_set_block_factor (enum tree_code, tree, tree);
+extern void upc_set_decl_section (tree);
+extern void upc_write_global_declarations (void);

A few command line option flags must also be
stub'ed out in order to link the other
language front-ends.

+++ gcc/c-family/stub-upc.c	(.../branches/gupc)	(revision 175683)
+++ gcc/c-family/stub-upc.c	(.../branches/gupc)	(revision 161914)
[...]
+int compiling_upc;
+int flag_upc;
+int use_upc_dwarf2_extensions;
[...]

The UPC-related front-end "diff's" are attached
for review.  All feedback and suggestions
are appreciated.  The goal is to work
these changes (and the others that will
be posted to this list) into a form that
the GUPC branch can be merged into the GCC trunk.

                                  -- end --

Attachment: gupc-trunk-front-end-diffs.txt
Description: Text document