Re: Threading the compiler

[Joe Buck <Joe dot Buck at synopsys dot COM>]

Paul Brook wrote:
> >For other optimisations I'm not convinced there's an easy win compared 
> >with make -j. You have to make sure those passes don't have any global 
> >state, and as other people have pointed out garbage collection gets messy. 
> >The compile server project did something similar, and that seems to have 
> >died.

On Mon, Nov 13, 2006 at 11:35:07AM -0700, Kevin Handy wrote:
> On the -j option:
> 
> What kind of improvements could you get by bypassing the GCC/G++/etc.
> front end, and replace it with a set of rules in the makefile?
> Using a set of rules to break down the compilation into the individual
> steps which can be scheduled in parallel, instead of the GCC front
> end running all the parts sequentially. This would allow the -j to
> schedule things in a finer grain without touching the compiler itself.

Or maybe a hybrid solution might work; allow gcc to talk to the jobserver
in make -j, to control the degree of parallelism.  For example, you might
have gcc do the basic parsing of headers and other declarations serially,
and then fork itself to generate optimized code for each function in
parallel.  The parallel jobs could share the memory devoted to
representing the declarations (using a copy-on-write approach, and
adjusting things so that pages devoted to declarations are distinct
from those used for the processing of individual functions).

If gcc is invoked by make -jn, then gcc's forks would wait
for tokens from make's jobserver before proceeding, so that the number
of jobs can be set to a sane limit.  The implementation would be to give
gcc a -j flag as well, and have gcc and make talk to each other the same
way as make talks to sub-makes.

Of course, going for whole-program optimization messes up this scheme,
which pretty much assumes that the production of code from each function
can be treated as a separate job.