compiler probe patch (to trunk, rev120801)

[Basile STARYNKEVITCH <basile at starynkevitch dot net>]

Hello All,

As I suggested in the http://gcc.gnu.org/ml/gcc/2006-11/msg00819.html
thread, here is a compiler probe patch.

Ben Elliston kindly told me in
http://gcc.gnu.org/ml/gcc/2006-11/msg00823.html about the VISTA
compilation environment http://citeseer.ist.psu.edu/433065.html

This (rather big) patch is less ambitious. My goal is only to be able
to inspect (but not change) the compiler internal data during a
compilation, and without altering the visible compiler behaviro
(i.e. the output assembly or object code). In a certain sense, this
patch generalize Sebastian Pop's tree browser (and I would like to
thank Seb for his support & patience explaining me GCC internal
stuff).

So in contrast to the VISTA paper, my goal is not to guide or change
the behavior of the compiler. It is simply to be able to query some
part of the internal state of the compiler in an interactive way (and
avoiding dumping lots of debug file and having to scan them, while
needing only a fraction of the information they contain).

Why did I code that? In addition to becoming familiar with GCC
internals, I am starting to work on adding static analysis inside
GCC. This means producing a lot of information inside GCC, and dumping
it to textual files is wasteful (both to the developers working inside
GCC and to those using such GCC compilers). It is much easier to
interactively give to the user the selected pieces of information
he/she wants.

The intended audience of this feature includes:

  GCC hackers which want to inspect GCC internal state without using
  GDB, including GCC newbies.

  GCC advanced users who might want to understand why their code was
  compiled in this or that way, without using the -fdump flag, and
  without only inspecting the generated assembly file.

  GCC advanced users who runs huge compilations - i.e. a single GCC
  invocation lasting more than a few minutes (in particular, those using
  -fwhole-program -combine on medium sized C programs). They could be
  interested in knowing interactively what is happenning inside the
  compiler.

  People adding "slow" analysis into the compiler. I won't argue on
  this again, but there is still a small niche market here - those
  using commercial static analysis tools [ABSINT, PolySpace,
  Astrée...] who want similar features inside GCC.


This patch provides a configurable feature which is by default
disabled. To have a GCC with a compiler probe ability you need to
explicitly configure it with --enable-compiler-probe at build time.
So by default this feature is disabled.

In practical terms, with this patch, you can select some given points
of your source code and ask GCC to display the GIMPLE trees or basic
blocks associated with this point. The points where you can ask for
information (most of the significant lines of your source) are called
infopoints.

Once you've got a GCC with a compiler probe ability (by explicitly
configuring it at build time), you occasionally want to probe the
compiler. To do so, you need to explicitly give to the (specially
configured) GCC compiler a flag, -fcompiler-probe= This flag requires
an argument, which is a probing command to run as a co-process with
GCC. When you give such a command, this probing command communicate
with GCC thru pipes, using a (documented) textual protocol. Of course,
in that rare case (only) the compiler could take an arbitrary amount
of time to handle all the probing requests (from the probe to GCC) -
since the requests can be arbitrarily many (and the compiler waits
till the probe tell him to continue).

In the common case when you use a compiler with the probing feature
configured but unused (i.e. without any -fcompilerp-orbe flag), there
is a very small speed penalty - so small that I cannot measure it
realiably, but it is about 0.1 or 0.2% (yes, one or two thousanth, ie
0.001 - 0.002) of the total compilation time on a big file. This is
the cost relative to the same source with the probing feature
disabled, or to the GCC trunk rev 120801

Now, how does it works? The trick is to be eventually able to call
some probing code quite frequently (typically every few hundredth of a
seconds, eg 30 times per second at least) in GCC, without having that
feature cost to much. To do so, I have a macro comprobe_check in my
file gcc/compiler-probe.h, with the intent that people would add in a
lot of places in their code they want to be probable, an invocation of
comprobe_check with a small descriptive string. For (imaginary)
example, you could add

  comprobe_check("begin dont_simulate_again bb loop")

in line 188 of tree-complex.c, e.g.
static bool
init_dont_simulate_again (void)
{
  basic_block bb;
  block_stmt_iterator bsi;
  tree phi;
  bool saw_a_complex_op = false;

  FOR_EACH_BB (bb)
    {
      comprobe_check("begin dont_simulate_again bb loop") ///// added stuff
      for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
	DONT_SIMULATE_AGAIN (phi) = !is_complex_reg (PHI_RESULT (phi));

//// etc....

in practical terms you can add frequent "calls" to the comprobe_check
macro inside your passes at every point where you want to frequently
be able to probe the compiler (and where your internal data is sane
enough!)

Now, what does this comprobe_check macro do? It just tests a volatile,
which is almost always false. Actually 
comprobe_check("begin dont_simulate_again bb loop") 
gets expanded into

  do{ 
   if (comprobe_interrupted) { 
      comprobe_handle_probe("begin dont_simulate_again bb loop",__FILE__,__LINE__); 
   }
  } while(0)

where comprobe_interrupted is the volatile sigatomic_t which is almost
always zero. So in the common case when you are not probing the
compiler, you just add some tests of a volatile word (e.g. a few
machine instructions) which happens to be always zero.

If the compiler probe was disabled at configure time (this is the
default) the comprobe_check is expanded into an empty do{}while(0)
thing, which GCC treats as a NOP and discards it.

How does it work? When the probe sends a request message, the
comprobe_interrupted flag should be set, and then the
comprobe_handle_probe routine does non trivial things like reading
requests (without blocking) and handling them. To do that, I am using
some Unix specific tricks (which I think are available on Solaris -it
was already in SunOS4!, recent BSD like Net,Free,Open BSD, MacOSX and
perhaps rel:iabe AIX and HPUX). When input is available from the
probe, a SIGIO signal is sent, and its handler sets the
comprobe_interrupted volatile flag. It could happen that some non Unix
systems offer similar abilities, but I don't know them. (I believe a
Windows expert could even patch my compiler-probe.c to make it work on
Windows).

Of course, the probed code should store all the user-accessible
information in some permanent data (lasting the whole
compilation). The compiler probe pass already stores tree-s and
basic_block-s uniquely (in a GGC-ed vector).

So that is the cons of this patch. It only works on some recent
Unixes. But I tried carefully to have the configure.ac test carefully
that some system dependent file compiles, so I hope you won't be able
to configure with --enable-compiler-probe on those systems not
providing necessary hooks. Unfortunately, I only have a Linux
workstation!

The main files added in this patch are
  gcc/compiler-probe.c   gcc/compiler-probe.h
which provides the pass and the machinery for probing

  gcc/doc/compiler-probe.texi
which documents (very incompletely) the protocol and the probing API

  contrib/simple-probe.c
which is a proof of concept probe, which only compiles on GNU/Linux
and uses the GTK2 and GTKSOURCEVIEW1 graphical toolkit libraries.

Of course there are other files patched ...

This patch seems to work on my Linux/AMD64/Debian/Sid machine (I am
bootstrapping it, it is already doing stage 3). Of course it is
imperfect, it may have some bugs, the documentation is quite poor, but
it seems usable .... But your comments are welcome!

I also tested the patch with --disable-compiler-probe to be sure it
compiles ok when disabled. You'll need to regenerate the configure
scripts.

Hoping to read you soon!

Regards

Changelog after my sig

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/ 
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359 
8, rue de la Faïencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***


########################################### Changelog
2007-01-16  Basile Starynkevitch  <basile@starynkevitch.net>

	* gcc/compiler-probe.c: new file

	* gcc/doc/gccint.texi:  added Compiler Probe

	* gcc/doc/compiler-probe.texi: new file

	* gcc/doc/invoke.texi: added -fcompiler-probe option

	* gcc/doc/install.texi: added --enable-compiler-probe option

	* gcc/compiler-probe.h: new file

	* contrib/simple-probe.c: new file

	* gcc/diagnostic.c:  kill the compiler probe when crashing

	* gcc/toplev.c:  added compiler probe

	* gcc/configure.ac: added compiler probe with compile test

	* gcc/common.opt: added compiler probe options

	* gcc/system.h: added for compiler probe

	* gcc/Makefile.in: added compiler-probe

	* gcc/passes.c: compiler probes (& checks)

	* contrib/simple-probe.c: new file for compiler probe

Attachment: patch-gcc-compiler-probe-trunk-rev120801.diff.gz
Description: Binary data