• MELT tutorial

MELT tutorial

OBSOLETE

Please read http://gcc-melt.org/docum.html for more up to date documentation (feb. 2013).

old stuff::

This page is for people wanting to experiment the MiddleEndLispTranslator branch, which follows quite strongly GCC trunk, but also provides a Lisp like dialect to write GCC passes (at least for experimentation and prototyping). This page also contains instructions to build MELT as a GCC plugin and an URL for downloading a MELT plugin snapshot. An old version of the GCC MELT internal documentation is attached as GCC-MELT--gcc-internals-snapshot.pdf but you really should build it from the MELT branch. Some of it is generated from the MELT source files using MELT. This documentation is incomplete.

prerequisites

It is supposed that :

• you did read the MELT wikipage, and you are vaguely familiar with basic GCC middle representations (Gimple, Tree, Gimple/SSA) and passes.

• you did build successfully the MELT branch (with MELT enabled at configure time), or the MELT plugin (for a GCC 4.5 at least).. Don't forget to ./contrib/gcc_update it often (twice a week) and rebuild it (and reinstall it)¹

• you are familiar with some Lisp like language, be it Scheme, Common Lisp, Emacs Lisp or other.

• you either did a make install of MELT (and we suppose that it is gcc-melt i.e. because it was configure-d with --program-suffix=-melt (or you made a shell script wrapping GCC MELT invocation), or you know GCC build machinery -the autoconf and Makefile.in mess inside GCC- to be able to run GCC MELT (probably from its build tree) by yourself.

• you know a little bit the internals of GCC (but we try to teach a bit of them here) and notably you have some good idea of what GIMPLE is.

Most of MELT additional files are under gcc libexec directory, which is something like /usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.5.0/ which we note here $gcclibexec for simplicity.

hello world in MELT

As with any language, we start by the traditional hello world program, So you'll edit your first MELT source file, and call it hello.melt. We give that file in several consecutive chunks; MELT source code files have the *.melt file extension.

As with any program, we start with comments explaining it. MELT being a Lisp dialect, comments start with a semi-colon and goes to the end of the line. I am using GNU Emacs to edit MELT code (but you could use any other Lisp friendly editor; lisp friendliness means at least the ability to match opening and closing parenthesis conveniently). But never forget that MELT is a Lisp dialect translated to C code called from inside GCC (more exactly from the cc1 or cc1plus program), and manipulating C

;;-*-mode:Lisp;-*-
;;file hello.melt

;; say hello world when this MELT module is loaded

We then define a function saying hello to someone if some boolean is true. Like in Common or Emacs Lisp, we use the defun keyword for defining functions, followed by the newly defined function name, then the list of formal arguments, and the function body.

;; define the say-hello-if function.
(defun say-hello-if (p :cstring msg)

We didn't finish the definition of function say-hello yet, so the starting parenthesis is not yet matched. We just gave the name of the defined function (and MELT names are like in Lisp: case insensitive, and can contain some extra punctuation, like dash or minus). The formal argument list is here (p :cstring msg) and it shows a very peculiar feature unique to MELT. MELT does not only deal with ordinary LISP-like values (first class citizens like atoms, lists, functions [i.e. closures], objects and many other kind of values, which are garbage collected ...) but also with stuff which is data which is not value (so is not a first class citizen in MELT, but can somehow be more or less conveniently handled by MELT code), but usually a direct stuff immediately represented by some data in C. In particular, The :cstring keyword symbol (starting with a colon ':' sign) declares that the following formal arguments are C string stuff. Such strings are exactly the same as in C (inside the GCC MELT compiler) and are required to be constant strings (of const char[] type in C), so are not garbage collected. So the formal argument list (p :cstring msg) declare one value formal p and another cstring stuff formal msg. For implementation reasons, each non-empty formal argument list (of MELT function) should start with a value formal, not a stuff formal. So the first (if any) formal argument must be a value, it cannot be a stuff (like a C string).

Now, back to our say-hello-if function definition. The initial opening parenthesis (just before defun) is not matched, so our definition is incomplete. We did already gave the formal argument list. We have to test if the formal value p is true. As in Emacs Lisp or Common Lisp (but unlike in Scheme), every non-nil value is considered as true, and nil (coded () as in most Lisps) represent false. So we simply code

   (if p

Of course, as most Lisp dialects, MELT is an expression oriented language. It does not have statements, but some expressions are computed only for their side effects and if their value is retrieved, it is nil. So we expect an expression which is computed conditionally when p is a non-nil value, and it should print something.

To give a more realistic feeling of MELT, we do the actual printing thru another MELT facility, the ability to mix C and MELT code inside a code chunk. Actually, code chunks are rarely needed, but are quite powerful, and this shows artificially another feature of MELT: the translation to C is quite easily tunable by giving parameterized C code chunks. The code_chunk keyword is the lowest level way to give such code chunks.

      (code_chunk hellochunk #{/*$hellochunk#_here*/ printf("hello %s\n", $msg);}#)

Higher level ways of giving code chunks include primitives (MELT operators which are directly translated to C), c-iterators (looping constructs), c-matchers (pattern matching constructs). The characters from #{ to }# are parsed as macro-strings by the MELT lexer and do not follow usual Lisp lexical rules (in particular, the semi-colon is not for comments; it has its obvious C-like meaning). So a macro-string starts with #{ and ends with }# ² all the characters inside a macro-string are taken verbatim, except the dollar sign $ which starts a MELT alphanumerical symbol -ending either by a space or ponctuation character or an ignored number sign # so the #{/*$hellochunk#_here*/ printf("hello %s\n", $msg);}# macro-string is parsed exactly by MELT as the list made of the /* two-character string, followed by the hellochunk symbol, followed by the _here*/ printf("hello %s\n",  string (starting with an underscore and ending with a space preceded by a comma), followed by the msg symbol, and ended by the ); two-character string. So instead of using these macro-strings, we could have typed: (code_chunk hellochunk ("/*" hellochunk "_here*/ printf(\"hello %s\\n\", " msg ");")) which is much less convenient, because we have to be careful of nested backslash escapes.

The code_chunk is a MELT keyword. It should be followed by a state symbol, here hellochunk and a list containing only symbols and strings, and usually input as a macro-string. State symbols are not locally bound, but are only useful at expansion time of a code chunk. Expanding a code chunk means generating a C code fragment (during the MELT to C translation process). This generated code fragment keeps the same strings as in the code chunk -and these strings are hence verbatim C code- and replace the MELT symbols by something appropriate. A state symbol is replaced by a unique (generated) C identifier, which is different inside each expansion ³. So for instance the state symbol hellochunk would be expanded to perhaps HELLOCHUNK__12 on the first expansion, and to HELLOCHUNK__345 on the second expansion. In this case, there is only one code expansion, so the C generated code would perhaps contain the /*HELLOCHUNK__1_here*/ comment.

MELT being a Lisp dialect, opening and closing parenthesis must be strictly balanced.

))

Here, the first right parenthesis matches the (if and the second matches the (defun.

MELT is an expression based language. Once we have defined our function say-hello-if we can call it inside a top-level expression

;; call the function with the symbol YES as first argument 
;; ... and the C string stuff "world" as the second one
(say-hello-if 'yes "world")

The above code shows that symbols can be quoted, and that a quoted symbol 'yes being exactly the same as (quote yes) where quote is a MELT (or lisp) keywords. The computation of a quted symbol evaluates to that symbol, so computing 'yes gives the symbol YES which is considered as a true value (the only false value being nil, noted () but never nil in MELT).

To facilitate the test of this example, here it is again in a single block which you can copy-paste inside your editor

;;-*-mode:Lisp;-*-
;;file hello.melt

;; say hello world when this MELT module is loaded

;; define the say-hello-if function.
(defun say-hello-if (p :cstring msg)
   (if p
      (code_chunk hellochunk
                  #{/*$hellochunk#_here*/ printf("hello %s\n", $msg);}#)
))

;; call the function with the symbol YES as first argument 
;; ... and the C string stuff "world" as the second one
(say-hello-if 'yes "world")

But we didn't explain yet how to run this example. You need some C (or C++ etc...) code to compile to get cc1 or cc1plus called by gcc-melt, so let assume you have a testit.c file (it can be a short C or C++ ... file, or even an empty one).

invoking GCC MELT

You can invoke gcc-melt (the compiler from the MELT branch) like

gcc-melt -fmelt-mode=runfile -fmelt-arg=hello.melt -c testit.c 

If you built MELT as a plugin to a GCC 4.5, the arguments are slightly different, Probably like

gcc-4.5 -fplugin=/some/path/to/melt.so -fplugin-arg-melt-mode=runfile \
        -fplugin-arg-melt-arg=hello.melt -c testit.c

The invocation translates the hello.melt file into a C code inside a temporary directory, then builds the hello.so module, loads and run it, displaying the hello world message and at last cleans the temporary directory. It should probably output something similar to

cc1: note: MELT generated new file /tmp/GCCMeltTmpdir-1a48fb7a/hello.c
cc1: note: MELT has built module /tmp/GCCMeltTmpdir-1a48fb7a/hello.so in 0.664 sec.
hello world
cc1: note: MELT removed 3 temporary files from /tmp/GCCMeltTmpdir-1a48fb7a

The first note message informs you that a hello.c file has been generated. It is then compiled ⁴ to some hello.so shared object which is dlopen-ed by GCC MELT. Afterwards, the hello world message is given. Before exiting, GCC MELT cleans the temporary directory.

Of course, the input file testit.c has been compiled by GCC MELT. Here, our MELT code did not improve or modify this compilation.

The -fmelt-mode=runfile program argument to gcc (or cc1) is essential. The word appearing after -fmelt-mode= is here runfile and is called the MELT mode. If it was missing, no MELT specific processing happens, and gcc-melt behaves exactly as the trunk.

It is possible to have the hello.melt translated to a hello.so which is kept and reused later. This is important in practice, since most of the additional time needed to translate MELT code into an executable shared library is the time spent by the C compiler to compile the generated .c file : generating hello.c from hello.melt is usually a lot faster (typically 10-20 times faster) than compiling hello.c into hello.so, and for real MELT code, the compilation time of the generated C code may be the bottleneck.

To generate the loadable module hello.so from hello.melt just invoke GCC MELT with the translatetomodule mode (under the covers, a make command is launched to build the hello.so module from the generated hello.c), i.e.

gcc-melt -fmelt-mode=translatetomodule -fmelt-arg=hello.melt -c empty.c 

One some rare occasions (in particular when a MELT pass crashes because it was inserted at the wrong place within the many existing GCC passes) you may want to keep the generated C file and to generated a hello.n.so module which does not know about locations in your hello.melt source file, using:

gcc-melt -fmelt-mode=translatedebug -fmelt-arg=hello.melt -c empty.c 

This should generate both hello.c and hello.n.so in your current directory. If you need to run gdb to find a bug, this could be useful, because the lines numbers in the hello.n.so modules refer to lines in the generated hello.c.

If you built MELT as a plugin to a GCC 4.5, the arguments are like

gcc-4.5 -fplugin=/some/path/to/melt.so -fplugin-arg-melt-mode=translatetomodule \
        -fplugin-arg-melt-arg=hello.melt -c empty.c

We won't repeat the variants anymore.

You still need to pass some file e.g. empty.c (which preferably should be an empty file) to be compiled by GCC, otherwise the cc1 program won't be invoked. With -fmelt-mode=translatetomodule the loadable module is generated in the same directory, so after the above command a file hello.so appears.

To get the generated C file hello.c replace the mode translatetomodule by translatefile; you could also specify the generated file name with -fmelt-output= so

gcc-melt -fmelt-mode=translatefile -fmelt-arg=hello.melt -fmelt-output=helloMELT.c -c empty.c 

would translate hello.melt into helloMELT.c⁵

Once you have your module hello.so you can have it running directly by GCC MELT. For that, you have to specify the list of MELT modules to be loaded, and the directory in which your module hello.so can be found, and you still has to give a MELT command, here noop which does nothing (but still have all the MELT modules loaded and their start routines executed).

gcc-melt -fmelt-mode=noop -fmelt-init=@@:hello -fmelt-module-path=. -c testit.c

The -fmelt-init argument (defaulting to @@ which means all the modules listed in file melt-default-modules.modlis) gives the list of modules to be loaded, and the -fmelt-module-path=. sets the path to seek MELT [dynamically loaded] modules (hence finding there your melt.so. You could also use the GCCMELT_MODULE_PATH environment variable for MELT modules *.so and GCCMELT_SOURCE_PATH ⁶ for *.melt & *.c source files.). By giving -fmelt-init=@@:hello wa are requiring to load all the melt default modules⁷ and then your hello module. If you wanted your hello.so module loaded by default (that would be a bad idea for this hello.melt!), put hello.melt inside $gcclibexec/melt-source, the generated hello.c inside $gcclibexec/melt-source and the generated hello.so module inside $gcclibexec/melt-module and add a line containing only hello at the end of file $gcclibexec/melt-dynlib/melt-default-modules.modlis and if you did that your hello module would be loaded at every explicit MELT invocation. Alternatively, set appropriately the GCCMELT_MODULE_PATH & GCCMELT_SOURCE_PATH environment variables.

In practice, a MELT module should do something on the GCC middle end representations so has to:

1. install some additional command[s] specific to your module
2. have this command enabling some additional GCC middle end passes (coded in MELT)
3. define the new GCC pass[es] by MELT code.

This is covered in more details in writing a pass in MELT

building and using MELT as a plugin (for GCC 4.5 ou Trunk)

updated in march 2010

If you have retrieved all MELT related files⁸, you could use the contrib/build-melt-plugin.sh script. Read it carefully first before running. It is not bullet proof⁹.

It is strongly advised to build MELT in the C locale, so do first an export LANG=C LC_ALL=C.

A major annoyance when building MELT as a plugin is that since MELT uses gengtype in plugin mode, it needs both¹⁰ build and source directory of the GCC into which it is plugged in. Read carefully the very end of plugins documentation. See also these GCC, GCC and Debian GCC mailing list threads.

Brave users should in principle be able to build and use MELT as a plugin for an unmodified gcc 4.5 (or gcc trunk, retrieved after july 15th 2009, svn revision 149655). In other words, it is possible (but not easy) to make a melt.so plugin usable on a pristine GCC 4.5 (or trunk) installation. However, you need both the GCC¹¹ source tree and the gcc build tree to do that. It is supposed that you already did build and install a new GCC (ie 4.5 or trunk). Both build and source trees of GCC are needed since MELT generates C files which includes a run-melt.h file which itself includes many header files. Since rev.152455 the trunk's gengtype is able to process MELT files. And your gcc should have been build with PPL enabled (i.e. graphite); PPL is also needed by MELT.

You first should retrieve (into a fresh directory like MeltPlugin), using svn, the following files and directories from the MELT branch: gcc/melt-runtime.h gcc/melt-runtime.c gcc/make-melt-predefh.awk gcc/make-warmelt-predef.awk gcc/melt-predef.list gcc/run-melt.h gcc/melt-make.mk gcc/melt/ contrib/build-melt-plugin.sh and possibly gcc/gengtype.c (to have it generate the gt-melt-runtime.hfile).

As a convenience, you alternatively could retrieve a GCC MELT plugin snapshot tarball. However, this snapshot is probably not the latest source, so retrieving thru subversion is much preferred.

A big issue¹² of GCC MELT plugin source tree is the painful generation of gt-melt-runtime.h using gengtype (because in plugin mode, gengtype needs the build and source tree of the GCC for which the plugin is built). This is a machine generated file (from gcc/melt-runtime.h and gcc/melt-runtime.c files), and it needs both GCC build and source tree to be regenerated in plugin mode. To avoid this inconvenience, this generated file is also available as gcc/melt/generated/gt-melt-runtime-plugin.h which is useful only for users not editing gcc/melt-runtime.h and gcc/melt-runtime.c or other MELT C files.

Then you should build the MELT plugin (for GCC, ie 4.5 or trunk). This can be done thru the build-melt-plugin.sh script. Please read carefully that script before running it, because it is not fail-proof and not well tested. In the common case you just want to build GCC MELT without editing the C code of the runtime, you could try to run in a fresh empty directory using something like

env GCC=gcc-4.5 HOSTADMINCMD=sudo ..../GCC-MELT-plugin/build-melt-plugin.sh -M ..../GCC-MELT-plugin/ -Y ..../GCC-MELT-plugin/melt/generated/gt-melt-runtime-plugin.h

Of course, replace the environment variables with whatever is appropriate for you. The -M argument gives the gcc source sub-tree of GCC MELT, or the directory containing melt-runtime.c and other files and melt/ directory (so also melt/generated/*.c files). The -Y argument tells the script to not use gengtype to regenerate gt-melt-runtime.h but instead to copy it. If you are very lucky, the MELT plugin has been installed (after having bootstrapped MELT translator), but if you are not lucky that script might have spoiled your system. Perhaps trying it with HOSTADMINCMD=echo first is preferable (it won't run under root commands to install MELT using sudo). There are many environment variables which can be set, read the build-melt-plugin.sh shell script to know them.

Unfortunately, if you have like me (Jérémie Salvucci), most of libraries needed by GCC in your $HOME and the MELT branch in your $HOME/gcc-melt-branch, your build command will look like :

$HOME/gcc-melt-branch/contrib/build-melt-plugin.sh -M $HOME/gcc-melt-branch/gcc \
     -Y $HOME/gcc-melt-branch/gcc/melt/generated/gt-melt-runtime-plugin.h \
     -s GCC=gcc-4.5 -C "-I $HOME/include/"

The MELT documentation and the bootstrap and update process still requires the MELT branch, so this branch need to be kept.

All MELT specific program arguments can be passed as plugin arguments. See the code of function melt_argument in melt-runtime.c

On july 15th 2009, the MELT plugin (i.e. melt.so compiled as a plugin above) is able to regenerate its C code from the MELT source, so is able to run as a GCC trunk plugin; however the MELT branch is still needed for MELT documentation and for MELT cold bootstrap¹³.

Of course, the arguments passed to MELT change in plugin mode, to follow the plugin conventions. So -fmelt-mode=compilefile -fmelt-arg=hello.melt becomes -fplugin-arg-melt-mode=compilefile -fplugin-arg-melt-arg=hello.melt (that is, -fmelt-mode= becomes -fplugin-arg-melt-mode= and -fmelt-XXX becomes -fplugin-arg-melt-XXX for various XXX so -fmelt-arg=hello.melt becomes -fplugin-arg-melt-arg=hello.melt in plugin mode.).

(Thanks to Sylvestre Ledru for finding mistakes here).