Bug 70268 - add option -ffile-prefix-map to map one directory name (old) to another (new) in __FILE__, __BASE_FILE__and __builtin_FILE()
Summary: add option -ffile-prefix-map to map one directory name (old) to another (new)...
Status: RESOLVED FIXED
Alias: None
Product: gcc
Classification: Unclassified
Component: other (show other bugs)
Version: 6.0
: P3 enhancement
Target Milestone: 8.0
Assignee: Not yet assigned to anyone
URL:
Keywords: patch
Depends on:
Blocks:
 
Reported: 2016-03-17 07:35 UTC by hongxu jia
Modified: 2018-07-20 06:57 UTC (History)
4 users (show)

See Also:
Host:
Target:
Build:
Known to work:
Known to fail:
Last reconfirmed:


Attachments
Implement patch (4.09 KB, patch)
2016-03-17 09:01 UTC, hongxu jia
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description hongxu jia 2016-03-17 07:35:13 UTC
Hi all,

I come from OpenEmbedded community (https://www.yoctoproject.org/),
and we use the powerful gcc for cross compilation. Previously, with
the help of Richard Biener and Jakub Jelinek, PR69821 was fixed which
do not expose the build path to others while '-g' used.

But there is another similar issue, while '__FILE__' used, this macro
expands to the name of the current input file which has build path.

I know the tradition solution is using relative path to compile.
(Such as instead of 'gcc /full/path/to/test.c',
try 'cd /full/path/to; gcc test.c; cd -;')

But if '__FILE__' exists in included header file and the header
file in standard system directories, the above way doesn't work.
Here is my test case:
-----------------------------------------------
1. Prepare a C source file and a C include file
$ cat << ENDOF > test.c
#include <stdio.h>
#include <test.h>

int main(int argc, char *argv[])
{
  printf("__FILE__ %s\n", __FILE__);
  test();
  return 0;
}

ENDOF

$ cat << ENDOF > test.h
#include <stdio.h>

void test(void)
{
  printf("__FILE__ %s\n", __FILE__);
  return;
}

ENDOF

2. Move include file to standard system directory.
$ sudo mv test.h /usr/include/

3. Compile C source file
$ gcc ./test.c -o test

4. Execute and the output shows full path header file.
$ ./test 
__FILE__ ./test.c
__FILE__ /usr/include/test.h
-----------------------------------------------
Comment 1 hongxu jia 2016-03-17 07:35:32 UTC
So I suggest gcc could provide a option '-ffile-prefix-map=<old>=<new>'
to map <old> directory in __FILE__ and replace it with <new> directory.

It is similar to '-fdebug-prefix-map=<old>=<new>', but the latter is used
for DWARF, and the newly added option only works on '__FILE__', '__BASE_FILE__'
and '__builtin_FILE()'

-----------------------------------------------
5. Compile C source file with '-ffile-prefix-map'
$ gcc ./test.c -ffile-prefix-map=/usr/include= -o test

4. Execute and the header file is not pull path.
$ ./test 
__FILE__ ./test.c
__FILE__ /test.h
-----------------------------------------------
Comment 2 Richard Biener 2016-03-17 08:23:02 UTC
IMHO it doesn't make sense to prune system header paths here and having them 
is desired.
Comment 3 hongxu jia 2016-03-17 08:45:51 UTC
(In reply to Richard Biener from comment #2)
> IMHO it doesn't make sense to prune system header paths here and having them 
> is desired.

Hi Richard,

It is just a simple example to explain the requirements.

In our OpenEmbedded project, we do many cross complication,
and have very complex build path.

Here is the actual use of option '-fdebug-prefix-map' in our project:

DEBUG_FLAGS="-g -feliminate-unused-debug-types -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/work/core2-64-poky-linux/service/0.1-r0/service-0.1=/usr/src/service -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/work/core2-64-poky-linux/service/0.1-r0/service-0.1=/usr/src/service -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/sysroots/x86_64-linux= -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/sysroots/qemux86-64= "

//Hongxu
Comment 4 hongxu jia 2016-03-17 09:01:23 UTC
Created attachment 37995 [details]
Implement patch
Comment 5 rguenther@suse.de 2016-03-17 09:02:21 UTC
On Thu, 17 Mar 2016, hongxu.jia at windriver dot com wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70268
> 
> --- Comment #3 from hongxu jia <hongxu.jia at windriver dot com> ---
> (In reply to Richard Biener from comment #2)
> > IMHO it doesn't make sense to prune system header paths here and having them 
> > is desired.
> 
> Hi Richard,
> 
> It is just a simple example to explain the requirements.
> 
> In our OpenEmbedded project, we do many cross complication,
> and have very complex build path.
> 
> Here is the actual use of option '-fdebug-prefix-map' in our project:
> 
> DEBUG_FLAGS="-g -feliminate-unused-debug-types
> -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/work/core2-64-poky-linux/service/0.1-r0/service-0.1=/usr/src/service
> -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/work/core2-64-poky-linux/service/0.1-r0/service-0.1=/usr/src/service
> -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/sysroots/x86_64-linux=
> -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/sysroots/qemux86-64=
> "

I see.  Care to propose a patch then?
Comment 6 hongxu jia 2016-03-17 09:04:58 UTC
(In reply to rguenther@suse.de from comment #5)
> On Thu, 17 Mar 2016, hongxu.jia at windriver dot com wrote:
> 
> > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70268
> > 
> > --- Comment #3 from hongxu jia <hongxu.jia at windriver dot com> ---
> > (In reply to Richard Biener from comment #2)
> > > IMHO it doesn't make sense to prune system header paths here and having them 
> > > is desired.
> > 
> > Hi Richard,
> > 
> > It is just a simple example to explain the requirements.
> > 
> > In our OpenEmbedded project, we do many cross complication,
> > and have very complex build path.
> > 
> > Here is the actual use of option '-fdebug-prefix-map' in our project:
> > 
> > DEBUG_FLAGS="-g -feliminate-unused-debug-types
> > -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/work/core2-64-poky-linux/service/0.1-r0/service-0.1=/usr/src/service
> > -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/work/core2-64-poky-linux/service/0.1-r0/service-0.1=/usr/src/service
> > -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/sysroots/x86_64-linux=
> > -fdebug-prefix-map=/buildarea/raid0/hjia/build-20160316-yocto-buildpath-2/tmp/sysroots/qemux86-64=
> > "
> 
> I see.  Care to propose a patch then?

Yes, also sent to gcc-patches@gcc.gnu.org for review

//Hongxu
Comment 7 Sascha Steinbiss 2016-09-11 13:17:37 UTC
Is there any progress on this? Actually such a functionality as provided by -ffile-prefix-map would also be highly desirable in the context of the Reproducible Builds effort [1]. Build-specific source paths included via __FILE__ in gcc-built artefacts are quite common and -- at least from my experience -- a problematic source of binary variation.

Cheers
Sascha

[1] https://reproducible-builds.org
Comment 8 Tom Rini 2017-05-22 17:28:09 UTC
This would also be useful in cases where we care about the size of our outputs but may have cases where file paths are included (critical debug type messages for example).
Comment 9 Boris Kolpackov 2017-11-17 10:15:44 UTC
I've proposed a different implementation (and a bit different options names) here:

https://gcc.gnu.org/ml/gcc-patches/2017-11/msg01451.html
Comment 10 Boris Kolpackov 2017-12-07 12:34:13 UTC
Second revision of the patch:

https://gcc.gnu.org/ml/gcc-patches/2017-12/msg00379.html
Comment 11 Boris Kolpackov 2017-12-09 13:40:24 UTC
Third revision of the patch:

https://gcc.gnu.org/ml/gcc-patches/2017-12/msg00544.html
Comment 12 infinity0 2017-12-15 15:14:57 UTC
Are you aware of this patch? https://gcc.gnu.org/ml/gcc-patches/2017-07/msg01315.html

If the overall motivation is to support reproducible builds, then you might be interested in that, which is a unified mechanism that allows future extensions to cover things *other* than __FILE__ and debug paths. (For example, reproducible .o files which, I am told, contain STT_FILE entries that contain the build path.)

The patch is not accepted yet; I had some review comments back in August and I'm supposed to amend it to use a command-line flag instead of an environment variable instead. I was going through my TODO backlog (non-GCC) since that time, but I'll be finally working on this over the next week or so.
Comment 13 Boris Kolpackov 2017-12-15 15:27:16 UTC
No, I was not aware, thanks for the pointer. I skimmed through it and I agree, the environment variable is a bad idea. In fact, if you look at the patch that I've proposed, it has a unified option (-ffile-prefix-map) that I believe does exactly what you want (remap any references to matching paths in the result of the compilation) while still allowing you to do more fine-grained mapping (e.g., only for __FILE__). And it has a provision for future extensions.
Comment 14 infinity0 2017-12-15 16:50:04 UTC
> it has a unified option (-ffile-prefix-map) [..]

Oh, nice. That might save me some work, then.

Could you bounce me the thread? Or failing that, tell me the Message-ID of one of the messages, so I can reply to it.

> [..] that I believe does exactly what you want

Basically yes. However one minor extra thing my patch enables, is the remapping of arbitrary file paths - whereas the current form can't remap paths with "=" in it. This is not a major issue, but I'd recommend that you change:

-  p = strchr (arg, '=');
+  p = strrchr (arg, '=');

This would be a bit more flexible.

> the environment variable is a bad idea.

The problem with doing this on the command-line is that the command-line arguments then contain the build-path. Sometimes, builds like to save the command-line arguments somewhere.

Using an environment variable avoids this issue. I agree envvars are dirty in general, but in this case one would be *removing* information from what GCC is already doing (taking in absolute-path information via the filesystem), as opposed to *adding* information which is what other envvars do.

A more refined way to avoid this issue, is to allow $-based substitution, like `-ffile-prefix-map=$BUILD_PATH=/usr/my/path` where BUILD_PATH is read from the environment. Then we avoid adding $BUILD_PATH to the command-line args, but /usr/my/path is still in there. Daniel Kahn Gillmor implemented this here [1] though it was rejected in favour of a simpler approach for the bug mentioned #69821 [2], however in retrospect this mechanism is actually very useful and flexible. I found out recently that NetBSD still carries this patch to this day, for reproducible builds. [3]

[1] https://gcc.gnu.org/ml/gcc-patches/2015-12/msg01168.html
[2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69821
[3] http://cvsweb.netbsd.org/bsdweb.cgi/src/external/gpl3/gcc/dist/gcc/final.c?rev=1.2&content-type=text/x-cvsweb-markup
Comment 15 infinity0 2017-12-19 17:48:41 UTC
(In reply to infinity0 from comment #14)
> > it has a unified option (-ffile-prefix-map) [..]
> 
> Oh, nice. That might save me some work, then.
> 
> Could you bounce me the thread? Or failing that, tell me the Message-ID of
> one of the messages, so I can reply to it.
> 
> > [..] that I believe does exactly what you want
> 

Thanks for forwarding the mail. I've posted a follow-up patch to be applied on top of your patch, that implements the two things I mentioned.
Comment 16 boris 2018-01-18 13:18:09 UTC
Author: boris
Date: Thu Jan 18 13:17:37 2018
New Revision: 256847

URL: https://gcc.gnu.org/viewcvs?rev=256847&root=gcc&view=rev
Log:
Add ability to remap file names in __FILE__, etc (PR other/70268)

This commit adds the -fmacro-prefix-map option that allows remapping of file
names in __FILE__, __BASE_FILE__, and __builtin_FILE(), similar to how
-fdebug-prefix-map allows to do the same for debug information.

Additionally, it adds -ffile-prefix-map which can be used to specify both
mappings with a single option (and, should we need to add more -f*-prefix-map
options in the future, those as well).

libcpp/ChangeLog:

2018-01-18  Boris Kolpackov  <boris@codesynthesis.com>

        PR other/70268
        * include/cpplib.h (cpp_callbacks::remap_filename): New callback.
        * libcpp/macro.c (_cpp_builtin_macro_text): Call remap_filename for
        __FILE__ and __BASE_FILE__.


gcc/ChangeLog:

2018-01-18  Boris Kolpackov  <boris@codesynthesis.com>

        PR other/70268
        * common.opt: (-ffile-prefix-map): New option.
        * opts.c (common_handle_option): Defer it.
        * opts-global.c (handle_common_deferred_options): Handle it.
        * debug.h (remap_debug_filename, add_debug_prefix_map): Move to...
        * file-prefix-map.h: New file.
        (remap_debug_filename, add_debug_prefix_map): ...here.
        (add_macro_prefix_map, add_file_prefix_map, remap_macro_filename): New.
        * final.c (debug_prefix_map, add_debug_prefix_map
        remap_debug_filename): Move to...
        * file-prefix-map.c: New file.
        (file_prefix_map, add_prefix_map, remap_filename) ...here and rename,
        generalize, get rid of alloca(), use strrchr() instead of strchr().
        (add_macro_prefix_map, add_debug_prefix_map, add_file_prefix_map):
        Implement in terms of add_prefix_map().
        (remap_macro_filename, remap_debug_filename): Implement in term of
        remap_filename().
        * Makefile.in (OBJS, PLUGIN_HEADERS): Add new files.
        * builtins.c (fold_builtin_FILE): Call remap_macro_filename().
        * dbxout.c: Include file-prefix-map.h.
        * varasm.c: Likewise.
        * vmsdbgout.c: Likewise.
        * xcoffout.c: Likewise.
        * dwarf2out.c: Likewise plus omit new options from DW_AT_producer.
        * doc/cppopts.texi (-fmacro-prefix-map): Document.
        * doc/invoke.texi (-ffile-prefix-map): Document.
	(-fdebug-prefix-map): Update description.


gcc/c-family/ChangeLog:

2018-01-18  Boris Kolpackov  <boris@codesynthesis.com>

        PR other/70268
        * c-family/c.opt (-fmacro-prefix-map): New option.
        * c-family/c-opts.c (c_common_handle_option): Handle it.
        * c-family/c-lex.c (init_c_lex): Set remap_filename cpp callback.
        * c-family/c-ppoutput.c (init_pp_output): Likewise.


gcc/testsuite/ChangeLog:

2018-01-18  Boris Kolpackov  <boris@codesynthesis.com>

        PR other/70268
        * c-c++-common/ffile-prefix-map.c: New test.
        * c-c++-common/fmacro-prefix-map.c: New test.
        * c-c++-common/cpp/ffile-prefix-map.c: New test.
        * c-c++-common/cpp/fmacro-prefix-map.c: New test.


Added:
    trunk/gcc/file-prefix-map.c
    trunk/gcc/file-prefix-map.h   (with props)
    trunk/gcc/testsuite/c-c++-common/cpp/ffile-prefix-map.c
    trunk/gcc/testsuite/c-c++-common/cpp/fmacro-prefix-map.c
    trunk/gcc/testsuite/c-c++-common/ffile-prefix-map.c
    trunk/gcc/testsuite/c-c++-common/fmacro-prefix-map.c
Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/Makefile.in
    trunk/gcc/builtins.c
    trunk/gcc/c-family/ChangeLog
    trunk/gcc/c-family/c-lex.c
    trunk/gcc/c-family/c-opts.c
    trunk/gcc/c-family/c-ppoutput.c
    trunk/gcc/c-family/c.opt
    trunk/gcc/common.opt
    trunk/gcc/dbxout.c
    trunk/gcc/debug.h
    trunk/gcc/doc/cppopts.texi
    trunk/gcc/doc/invoke.texi
    trunk/gcc/dwarf2out.c
    trunk/gcc/final.c
    trunk/gcc/opts-global.c
    trunk/gcc/opts.c
    trunk/gcc/testsuite/ChangeLog
    trunk/gcc/varasm.c
    trunk/gcc/vmsdbgout.c
    trunk/gcc/xcoffout.c
    trunk/libcpp/ChangeLog
    trunk/libcpp/include/cpplib.h
    trunk/libcpp/macro.c

Propchange: trunk/gcc/file-prefix-map.h
            ('svn:eol-style' added)

Propchange: trunk/gcc/file-prefix-map.h
            ('svn:keywords' added)
Comment 17 alex_y_xu 2018-07-19 19:23:13 UTC
seems like this should be RESOLVED FIXED?
Comment 18 Richard Biener 2018-07-20 06:57:45 UTC
Fixed in GCC 8.