[EXTERNAL] Re: State of AutoFDO in GCC
Wei Mi
wmi@google.com
Wed May 26 03:06:51 GMT 2021
I checked the source of protobuf 3.0.0 and it didn't contain the
operator[] in RepeatedField. Need to install a newer version of
protobuf.
Thanks,
Wei.
On Tue, May 25, 2021 at 1:49 PM Eugene Rozenfeld
<Eugene.Rozenfeld@microsoft.com> wrote:
>
> Both are 3.0.0-9.1ubuntu1:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list protobuf-compiler
> Listing... Done
> protobuf-compiler/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed]
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list libprotobuf-dev
> Listing... Done
> libprotobuf-dev/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed]
>
> -----Original Message-----
> From: Wei Mi <wmi@google.com>
> Sent: Tuesday, May 25, 2021 9:17 AM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It looks like some version problem about protobuf-compiler and libprotobuf-dev. Could you check what is the installed version on your end for those two packages and see if they are consistent?
>
> On my platform, they are both 3.12.4.
>
> On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > That eliminates the previous error but there is a new one:
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] Building
> > CXX object
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> > r/perf_reader.cc.o
> > FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> > /usr/bin/c++ -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: no match for 'operator[]' (operand types are 'const google::protobuf::RepeatedField<unsigned int>' and 'int')
> > nrcpus = proto_uint32_metadata.data()[0];
> >
> > -----Original Message-----
> > From: Wei Mi <wmi@google.com>
> > Sent: Monday, May 24, 2021 8:54 PM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang
> > David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>;
> > gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > It isn't exposed on my platform either. Looks like a bug in perf_data_converter (i.e., quipper). Could you try adding #include <unordered_map> in third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if it fixes the problem? If it works, I will need to file a bug against perf_data_converter.
> >
> > Thanks,
> > Wei.
> >
> > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > >
> > > That fixed the error I saw before but the build still fails. The
> > > errors start with
> > >
> > >
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> > >
> > > [2/217] Building CXX object
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > > pe
> > > r/huge_page_deducer.cc.o
> > >
> > > FAILED:
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > > pe
> > > r/huge_page_deducer.cc.o
> > >
> > > /usr/bin/c++ -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> > >
> > > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:
> > > 14
> > > 4:26: error: 'unordered_map' in namespace 'std' does not name a
> > > template type
> > >
> > > using container = std::unordered_map<key_t, value_t>;
> > >
> > > ^~~~~~~~~~~~~
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: Wei Mi <wmi@google.com>
> > > Sent: Monday, May 24, 2021 8:12 PM
> > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > >
> > >
> > > Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315397315%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9G9uGhG%2BmK35bdV4QL9BOTVg6bFFQrVP46YdLOi3Ak4%3D&reserved=0.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Wei.
> > >
> > >
> > >
> > > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > >
> > > Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241]
> > > Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > /usr/bin/c++ -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > from ../sample_reader.h:18,
> > > from ../profile.h:15,
> > > from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > > DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > >
> > > In file included from ../profile.h:14:0,
> > > from ../profile.cc:5:
> > > ../base/macros.h:114:0: note: this is the location of the previous
> > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > >
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > from ../sample_reader.h:18,
> > > from ../profile.h:15,
> > > from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > > sizeof(*x))
> > >
> > > In file included from ../profile.h:14:0,
> > > from ../profile.cc:5:
> > > ../base/macros.h:162:0: note: this is the location of the previous
> > > definition #define arraysize(array)
> > > (sizeof(ArraySizeHelper(array)))
> > >
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > > from ../sample_reader.h:18,
> > > from ../profile.h:15,
> > > from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10:
> > > fatal error: perf_data.pb.h: No such file or directory #include "perf_data.pb.h"
> > > ^~~~~~~~~~~~~~~~
> > > compilation terminated.
> > > [6/241] Building CXX object
> > > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> > > ninja: build stopped: subcommand failed.
> > >
> > > Thanks,
> > >
> > > Eugene
> > >
> > > From: Wei Mi <wmi@google.com>
> > > Sent: Saturday, May 22, 2021 9:37 AM
> > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > > It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SL5mLhelrkeB26Mg7tNaml0qjZ2OklYPwQR9btVhsqk%3D&reserved=0. Please try again.
> > >
> > > Thanks,
> > > Wei.
> > >
> > > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
> > > I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PYVm%2BHsRk%2Bfw7%2BaNgYNWn8zMnBsqcqqVSpyXQktd7sA%3D&reserved=0 got build failures:
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228]
> > > Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > /usr/bin/c++ -I../ -I../third_party/glog/src
> > > -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > from ../sample_reader.h:18,
> > > from ../profile.h:15,
> > > from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile.h:14:0,
> > > from ../profile.cc:5:
> > > ../base/macros.h:114:0: note: this is the location of the previous
> > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file
> > > included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > from ../sample_reader.h:18,
> > > from ../profile.h:15,
> > > from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > > sizeof(*x)) In file included from ../profile.h:14:0,
> > > from ../profile.cc:5:
> > > ../base/macros.h:162:0: note: this is the location of the previous
> > > definition #define arraysize(array) (sizeof(ArraySizeHelper(array)))
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > > from ../sample_reader.h:18,
> > > from ../profile.h:15,
> > > from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10:
> > > fatal error: perf_stat.pb.h: No such file or directory #include "perf_stat.pb.h"
> > > ^~~~~~~~~~~~~~~~
> > > compilation terminated.
> > >
> > > What is supposed to generate perf_stat.pb.h?
> > >
> > > Thanks,
> > >
> > > Eugene
> > >
> > > From: Wei Mi <mailto:wmi@google.com>
> > > Sent: Monday, May 10, 2021 4:47 PM
> > > To: Andi Kleen <mailto:ak@linux.intel.com>
> > > Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li
> > > <mailto:davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>;
> > > mailto:gcc@gcc.gnu.org; Eugene Rozenfeld
> > > <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He
> > > <mailto:wenlei@fb.com>
> > > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uwQwWdjE%2FL%2F2r0P%2Bdn7r5ulE8kRmPPw8A2XOQuZnUhM%3D&reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
> > >
> > > Please look at "2.2 Build autofdo tool for gcc" in
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > th
> > > ub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfe
> > > ld
> > > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141
> > > af
> > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3
> > > d8
> > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7
> > > C1
> > > 000&sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&r
> > > es
> > > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen
> > > <mailto:ak@linux.intel.com> wrote:
> > > >
> > > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > > > > Andi, thanks for pointing out the perf script issues. Can you please
> > > > > elaborate a bit on the exact issue you have seen? We've been using
> > > > > specific output of perf script such as mmap, LBR and callstack events
> > > > > filtered by process id. It works fine so far but may certainly hit issues
> > > > > in the future with extended uses.
> > > >
> > > > Okay I took a look at the latest autofdo now. It seems to be
> > > > basically a LLVM project now that depends on LLVM to even build
> > > > with all kinds of dependency hell on some old LLVM version and other packages.
> > > >
> > > > I guess gcc will really need a replacement that doesn't pull in
> > > > all of LLVM if it wants to continue supporting autofdo.
> > > >
> > > > I'm myself unable to build now.
> > > >
> > > > I'm using the old version I had a git fork of and that was before
> > > > all of this. I added a patch to make it work with the latest perf
> > > > by ignoring increased perf_attr and unknown perf events.
> > > >
> > > > Honza please use
> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> > > > gi
> > > > thub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld
> > > > %4
> > > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141
> > > > af
> > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZs
> > > > b3
> > > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > > > 3D
> > > > %7C1000&sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%3D
> > > > &a
> > > > mp;reserved=0 -b perf-future
> > > >
> > > > for testing.
> > > >
> > > > -Andi
> > > >
More information about the Gcc
mailing list