Re: Separate preprocess and compile: hack or feature?

[Boris Kolpackov <boris at codesynthesis dot com>]

Hi Nathan,

Nathan Sidwell <nathan@acm.org> writes:

> How c++ modules fit into a build system is currently an open question.
> Richard Smith & I have talked about it, but with no firm conclusion.
> However, I think that breaking out the preprocessor is not the right
> answer.

Handling modules is one of the least important motivations for my desire
to "break out" the preprocessor. Distributed compilation is probably the
main one. But I am quickly realizing that this is probably not going to
be reliable enough.

But for completeness, let me describe how all the pieces would have fitted
together. I think it is quite elegant, even if I have to say so myself ;-).

Let's say the build system realizes (for example, based on filesystem
mtimes) that hello.o may be out-of-date. This is what it does:

1. In a single pass preprocess hello.cxx and extract header dependencies
   (-E -MD options).

   Note that if you do -M you are essentially running the preprocessor,
   so might as well save the result.

2. If any of the extracted headers are auto-generated and are missing or
   out-of-date, regenerate them.

3. Pre-parse the preprocessed output and extract module dependency
   information. If any are missing/out-of-date, compile them.

   What we need to detect here are (1) module imports and (2) module
   implementation units. All these things are "top level" and it won't
   be hard to recognize.
   
4. If nothing above has indicated that hello.o is indeed out of date,
   hash the preprocessed output to detect and ignore comment-only changes.

5. If we indeed need to update hello.o, pass the preprocessed output
   either to the compiler or ship it to a remote host for compilation.

Boris