This is the mail archive of the gcc-help@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: reduce compilation times?


On 2007/11/27, Vladimir Vassilev <vladimir@logicom-bg.com> wrote:
> J.C. Pizarro wrote:
> > It's the same problem of "Nine Woman Can't Have a Baby in One Month"
> >
> > "Nine Cores Can't Reduce a Compilation of 9 Seconds to One Second"
> >
> In case the project is composed of reasonably sized files and the number
> of those files is sufficient the parallel build gives good scalability.
> The "make -j" option is a very powerful tool not only in terms of
> compilation. We have been using this also for rendering and other time
> consuming jobs. It basically allows you to build in parallel any target
> described in a Makefile. Implementation of MPI based plugin module for
> GNU Make (the -j option comes with a well defined interface to spawn the
> "threads" and it is easy to implement that for any MPI or even socket
> interconnected number of hosts) in combination with shared file system
> has allowed us to build various time consuming projects on computer
> clusters gaining almost linear scaling.
>
> Vladimir

About scaling, we can use LiveCDs to test the measures of compile-time
in clusters or virtual clusters of machines with not much effort.

These LiveCDs can be ParallelKnoppix, Knoppix OpenSSI, Clusterix,
ClusterKnoppix, Scientific, Quantian, Rocks Cluster, VMKnoppix, Xenoppix, ..

On 2007/11/27, Sven Eschenberg <eschenb@cs.uni-frankfurt.de> wrote:
> I am not sure about ccache, but I thought it does some file and
> preprocessing caching (not exactly sure, how it works, I thought, it kinda
> gets called instead of the preprocessor or at least before the PP).

Combining distcc & ccache is little complicated because of the square vision
with two triangles in the fourth vertex:

    no distcc  ----------------------------   ccache-only
    no ccache                                     |
         |                                        |
         |                                        |
         |                                     \  1st ccache of the local machi$
   distcc-only   ----------------     1st distcc \   2nd distcc (otherwise)
                                    2nd ccache of  \
                                 the remote machine  \

How to redistribute to the machines the ccache's data?

The quick and dirty strategy is to use "rsync" between every dirs
/ramdisk/tmps/ccache_data/ of the participing remote machines.

It's replicating and merging the local data of ccache of each local
machine to every machines of the cluster.

I suppose that doesn't exist yet a "distributed distccache for GCC".

   J.C.Pizarro


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]