If you had a month to improve gcc build parallelization, where would you begin?

[David Brown]

On 10/04/13 04:51, Geert Bosch wrote:
> 
> On Apr 9, 2013, at 22:19, Segher Boessenkool <segher@kernel.crashing.org> wrote:
> 
>> Some numbers, 16-core 64-thread POWER7, c,c++,fortran bootstrap:
>> -j6:  real    57m32.245s
>> -j60: real    38m18.583s
> 
> Yes, these confirm mine. It doesn't make sense to look at more
> parallelization before we address the serial bottlenecks.
> The the -j6 parallelism level is about where current laptops
> are. Having a big machine doesn't do as much as having fewer,
> but faster cores. 
> 
> We should be able to do far better. I don't know how the Power7
> threads compare in terms of CPU throughput, but going from -j6 to
> -j48 on our 48-core AMD system should easily yield a 6x speed up
> as all are full cores, but we get similar limited improvements to
> yours, and we get almost perfect scaling in many test suite runs
> that are dominated by compilations.
> 
> The two obvious issues:
>   1. large sequential chains of compiling/running genattrtab followied
>      by compiling insn-attrtab.c and linking the compiler
>   2. repeated serial configure steps
> 
> For 1. we need to somehow split the file up in smaller chunks.
> For 2. we need to have efficient caching.
> 
> Neither is easy...
> 
>   -Geert
> 
> 

Efficient caching of configuration steps would have many more benefits
than just improving parallel builds.  It would directly improve builds
on /all/ systems, and have an even bigger effect on re-builds (re-build
compilation speed can be improved with ccache - but you still need to
wait for the ./configure steps).

And of course if someone figures out how to speed up ./configure with
gcc, the same technique could probably be used on thousands of other
programs.

I presume fixing the slow, inefficient serial nature of autotools is a
hard job - otherwise it would have been done already.  But any
improvements there could have big potential benefits.