[EXTERNAL] Re: State of AutoFDO in GCC
Eugene Rozenfeld
Eugene.Rozenfeld@microsoft.com
Tue May 25 03:33:02 GMT 2021
That fixed the error I saw before but the build still fails. The errors start with
eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[2/217] Building CXX object CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
/usr/bin/c++ -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:144:26: error: 'unordered_map' in namespace 'std' does not name a template type
using container = std::unordered_map<key_t, value_t>;
^~~~~~~~~~~~~
From: Wei Mi <wmi@google.com>
Sent: Monday, May 24, 2021 8:12 PM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://github.com/google/autofdo/commit/6ca36cdc30986f13583a3aef3e27746ca4fc5bf6<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312421958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3fWMYoqpMP9jRocxHKRKN5zv5DLIgCuLR%2FgczvA1rJc%3D&reserved=0>.
Thanks,
Wei.
On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com<mailto:Eugene.Rozenfeld@microsoft.com>> wrote:
Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
/usr/bin/c++ -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
from ../sample_reader.h:18,
from ../profile.h:15,
from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: "DISALLOW_COPY_AND_ASSIGN" redefined
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../profile.h:14:0,
from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
from ../sample_reader.h:18,
from ../profile.h:15,
from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: "arraysize" redefined
#define arraysize(x) (sizeof(x) / sizeof(*x))
In file included from ../profile.h:14:0,
from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
#define arraysize(array) (sizeof(ArraySizeHelper(array)))
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
from ../sample_reader.h:18,
from ../profile.h:15,
from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: fatal error: perf_data.pb.h: No such file or directory
#include "perf_data.pb.h"
^~~~~~~~~~~~~~~~
compilation terminated.
[6/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
ninja: build stopped: subcommand failed.
Thanks,
Eugene
From: Wei Mi <wmi@google.com<mailto:wmi@google.com>>
Sent: Saturday, May 22, 2021 9:37 AM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com<mailto:Eugene.Rozenfeld@microsoft.com>>
Cc: Andi Kleen <ak@linux.intel.com<mailto:ak@linux.intel.com>>; Hongtao Yu <hoy@fb.com<mailto:hoy@fb.com>>; Xinliang David Li <davidxl@google.com<mailto:davidxl@google.com>>; Jan Hubicka <hubicka@ucw.cz<mailto:hubicka@ucw.cz>>; gcc@gcc.gnu.org<mailto:gcc@gcc.gnu.org>; Wenlei He <wenlei@fb.com<mailto:wenlei@fb.com>>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342270322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TxMivTXj9w0W5OFLunWcOpqf9wefrrUd0m3RqV8JLaA%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312431954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OdcAYo8mJD7ipadVeQq94T%2BGoBlO0AsJErGmeVKVJ7c%3D&reserved=0>. Please try again.
Thanks,
Wei.
On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com<mailto:Eugene.Rozenfeld@microsoft.com>> wrote:
I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342280318%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KBPIcf5k1v230tbLke9%2FdSt25nbPrHhpBvPb8PiebPY%3D&reserved=0and<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312441950%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Nz6MH%2B2MFwbvgAAi6FcxJretHqF6SiH3P2oLbVqOJ%2B4%3D&reserved=0> got build failures:
eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/228] Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
/usr/bin/c++ -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
from ../sample_reader.h:18,
from ../profile.h:15,
from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: "DISALLOW_COPY_AND_ASSIGN" redefined
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../profile.h:14:0,
from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
from ../sample_reader.h:18,
from ../profile.h:15,
from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: "arraysize" redefined
#define arraysize(x) (sizeof(x) / sizeof(*x))
In file included from ../profile.h:14:0,
from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
#define arraysize(array) (sizeof(ArraySizeHelper(array)))
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
from ../sample_reader.h:18,
from ../profile.h:15,
from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: fatal error: perf_stat.pb.h: No such file or directory
#include "perf_stat.pb.h"
^~~~~~~~~~~~~~~~
compilation terminated.
What is supposed to generate perf_stat.pb.h?
Thanks,
Eugene
From: Wei Mi <mailto:wmi@google.com<mailto:wmi@google.com>>
Sent: Monday, May 10, 2021 4:47 PM
To: Andi Kleen <mailto:ak@linux.intel.com<mailto:ak@linux.intel.com>>
Cc: Hongtao Yu <mailto:hoy@fb.com<mailto:hoy@fb.com>>; Xinliang David Li <mailto:davidxl@google.com<mailto:davidxl@google.com>>; Jan Hubicka <mailto:hubicka@ucw.cz<mailto:hubicka@ucw.cz>>; mailto:gcc@gcc.gnu.org<mailto:gcc@gcc.gnu.org>; Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com<mailto:Eugene.Rozenfeld@microsoft.com>>; Wenlei He <mailto:wenlei@fb.com<mailto:wenlei@fb.com>>
Subject: [EXTERNAL] Re: State of AutoFDO in GCC
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342290320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UWJlvkaOdGT5p8ZLmeZYkZ6lPhSmdmzc0tEnBULcNsY%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312451945%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xGCwhFyre4ipz2xXZnkeIL1a%2Bd6x%2FbWLe58szmVYdO8%3D&reserved=0> has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
Please look at "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342290320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3fXLah0%2B0YHbo%2BHlbcgUFwlFQD9YwCUZApmZHoasiW0%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312461941%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4U%2B6DVDVtRVsfE%2FN41s2IaygGaFchfwVqvbhiC%2FSLTY%3D&reserved=0>
On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen <mailto:ak@linux.intel.com<mailto:ak@linux.intel.com>> wrote:
>
> On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > Andi, thanks for pointing out the perf script issues. Can you please
> > elaborate a bit on the exact issue you have seen? We've been using
> > specific output of perf script such as mmap, LBR and callstack events
> > filtered by process id. It works fine so far but may certainly hit issues
> > in the future with extended uses.
>
> Okay I took a look at the latest autofdo now. It seems to be basically
> a LLVM project now that depends on LLVM to even build with all kinds
> of dependency hell on some old LLVM version and other packages.
>
> I guess gcc will really need a replacement that doesn't pull in
> all of LLVM if it wants to continue supporting autofdo.
>
> I'm myself unable to build now.
>
> I'm using the old version I had a git fork of and that
> was before all of this. I added a patch to make it work
> with the latest perf by ignoring increased perf_attr
> and unknown perf events.
>
> Honza please use
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342300314%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aNe2qZeEjsE2%2Fm7ZbaIFPFAcnF3wtQEkOJr0fG5tHWo%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312471940%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4JAfSVhReYE9ITLr5IKzLucH5ckdFtvGG5AKtnFjBZQ%3D&reserved=0> -b perf-future
>
> for testing.
>
> -Andi
>
More information about the Gcc
mailing list