Scaling -fmacro-prefix-map= to thousands entries

[Richard Biener]

On Thu, Oct 5, 2023 at 2:17 PM Arsen Arsenović <arsen@aarsen.me> wrote:
>
> Hi,
>
> Sergei Trofimovich via Gcc <gcc@gcc.gnu.org> writes:
>
> > Sounds good. Do you have any preference over specific syntax? My
> > suggestions would be:
> >
> > 1. `-fmacro-prefix-map=file-name`: if `file-name` there is not in `key=val`
> >    format then treat it as file
> > 2. `-fmacro-prefix-map=@file-name`: use @ as a signal to use file
> > 3. `fmacro-prefix-map-file=file-name`: use a new option
> >
> >> Btw, I thought we have response files to deal with command-line limits,
> >> why doesn't that work here?  I see the driver expands response files
> >> but IIRC it also builds those when the command-line gets too large
> >> and uses it for the environment and the cc1 invocation?  If it doesn't
> >> do the latter why not fix it that way?
> >
> > Yeah, in theory response files would extend the limit. In practice `gcc`
> > always extends response files internally into a single
> > `COLLECT_GCC_OPTIONS` option and hits the environment variable limit
> > very early:
> >
> >     https://gcc.gnu.org/PR111527
>
> Doesn't it make more sense to fix this?  The issue is more general than
> just this option (even if manifesting like so today).
>
> Can the COLLECT_GCC_OPTIONS consumers deal with argfiles?

No.  The traditional consumer is collect2, nowadays the main consumer
is lto-wrapper and the lto-plugin.  There's
parse_options_from_collect_gcc_options but
for example collect2 does its own parsing.  There might be other tools out
in the wild interpreting COLLECT_GCC_OPTIONS.

We might make life of the tools easy if either all of COLLECT_GCC_OPTIONS
is fully expanded or it is a single @file "argument" (but with
otherwise identical,
quoted content).

Richard.

> > Example reproducer:
> >
> >     $ for i in `seq 1 1000`; do printf -- "-fmacro-prefix-map=%0*d=%0*d\n" 200 1 200 2; done > a.rsp
> >     $ touch a.c; gcc @a.rsp -c a.c
> >     gcc: fatal error: cannot execute 'cc1': execv: Argument list too long
> >     compilation terminated.
> >
> > And if you want to look at the gory details:
> >
> >     $ strace -f -etrace=execve -s 1000000 -v -v -v  gcc @a.rsp -c a.c
> >     ...
> >     [pid    78] execve("cc1", ["cc1", "-quiet", "a.c", "-quiet", "-dumpbase", "a.c", "-dumpbase-ext", ".c", "-mtune=generic", "-march=x86-64",
> >     "-fmacro-prefix-map=...=...",
> >     "-fmacro-prefix-map=...=...",
> >     ...],
> >     [...,
> >      "COLLECT_GCC=gcc",
> >      "COLLECT_GCC_OPTIONS='-fmacro-prefix-map=...=...' '-fmacro-prefix-map=...=...' ... '-c' '-mtune=generic' '-march=x86-64'"]) = -1 E2BIG (Argument list too long)
> >
> > Note how `gcc` not only expands response file into an argument list
> > (that is not too bad) but also duplicates the whole list as a single
> > `COLLECT_GCC_OPTIONS=...` environment variable with added quoting on
> > top.
> >
> > Would be nice if `gcc` just passed response files around as is :)
>
>
> --
> Arsen Arsenović