Scaling -fmacro-prefix-map= to thousands entries

[Sergei Trofimovich]

On Thu, Oct 05, 2023 at 07:20:35AM -0400, Ben Boeckel wrote:
> On Wed, Oct 04, 2023 at 22:19:32 +0100, Sergei Trofimovich via Gcc wrote:
> > The prototype that creates equivalent of the following commands does
> > work for smaller packages:
> > 
> >     -fmacro-prefix-map=/nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.37-8-dev
> >     -fmacro-prefix-map=/nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-with-cxx-6.3.0-dev
> >     -fmacro-prefix-map=/nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nlohmann_json-3.11.2
> >     ...
> > 
> > The above works for small amount of options (like, 100). But around 1000
> > options we start hitting linux limits on the single environment variable
> > or real-world packages like `qemu` with a ton of input depends.
> 
> Are you trying to pass this through via `CFLAGS` and friends?

Roughly via `CFLAGS`. `nixpkgs` uses it's own private
`NIX_CFLAGS_COMPILE` variable which gets extended in `gcc-wrapper` shel
wrapper as expcit list of arguments to real `gcc-binary`. It's almost
like `CFLAGS` but is expected to be transaparent to most build systems.

> > The command-line limitations are in various places:
> > - `gcc` limitation of lifting all command line options into a single
> >   environment variable: https://gcc.gnu.org/PR111527
> > - `linux` limitation of constraining single environ variable to a value
> >   way below than full available environment space:
> >   https://lkml.org/lkml/2023/9/24/381
> > 
> > `linux` fix would buy us 50x more budged (A Lot) but it will not help
> > much other operating systems like `Darwin` where absolute environment
> > limit is a lot lower than `linux`.
> > 
> > I already implemented [1.] in https://github.com/NixOS/nixpkgs/pull/255192
> > (also attached `mangle-NIX_STORE-in-__FILE__.patch` 3.5K patch against
> > `master` as a proof of concept).
> > 
> > What would be the best way to scale up `-fmacro-prefix-map=` up to NixOS
> > needs for `gcc`? I would like to implement something sensible I could
> > upstream.
> 
> How about `CFLAGS=@macro_prefix_map.args` and writing that file in the
> same codepath where you generate the flags today. It'll work with just
> about every compiler and tools like `ccache` will understand that it is
> an input that affects the build and properly take the file's contents
> into account.

That was my initial attempt. I'll duplicate my response from 
https://gcc.gnu.org/pipermail/gcc/2023-October/242639.html here as is:

"""
Yeah, in theory response files would extend the limit. In practice `gcc`
always extends response files internally into a single
`COLLECT_GCC_OPTIONS` option and hits the environment variable limit
very early:

    https://gcc.gnu.org/PR111527

Example reproducer:

    $ for i in `seq 1 1000`; do printf -- "-fmacro-prefix-map=%0*d=%0*d\n" 200 1 200 2; done > a.rsp
    $ touch a.c; gcc @a.rsp -c a.c
    gcc: fatal error: cannot execute 'cc1': execv: Argument list too long
    compilation terminated.

And if you want to look at the gory details:

    $ strace -f -etrace=execve -s 1000000 -v -v -v  gcc @a.rsp -c a.c
    ...
    [pid    78] execve("cc1", ["cc1", "-quiet", "a.c", "-quiet", "-dumpbase", "a.c", "-dumpbase-ext", ".c", "-mtune=generic", "-march=x86-64",
    "-fmacro-prefix-map=...=...",
    "-fmacro-prefix-map=...=...",
    ...],
    [...,
     "COLLECT_GCC=gcc",
     "COLLECT_GCC_OPTIONS='-fmacro-prefix-map=...=...' '-fmacro-prefix-map=...=...' ... '-c' '-mtune=generic' '-march=x86-64'"]) = -1 E2BIG (Argument list too long)

Note how `gcc` not only expands response file into an argument list
(that is not too bad) but also duplicates the whole list as a single
`COLLECT_GCC_OPTIONS=...` environment variable with added quoting on
top.

Would be nice if `gcc` just passed response files around as is :)
"""

-- 

  Sergei