State of AutoFDO in GCC

[Jan Hubicka]

David,
> 
> The text format is documented here:
> https://clang.llvm.org/docs/UsersManual.html
> The binary format is not documented. The binary format is not guaranteed to
> be backward compatible, so sharing the same format may not be the best way
> as changes for clang may break GCC.
> 
> Since linux perf format does not change, the tool should be relatively
> stable with low maintenance cost. Changes are needed only when some new
> AutoFDO features are added to the compiler side.

I was under impression that it is indeed problem with the tool requiring
old format of linux perf. At least with opensuse distro the shipped tool
fails for me:
jan@skylake:~> create_llvm_prof --binary=./code --out=code.prof
E0425 21:01:55.038128 17977 perf_reader.cc:996] Unsupported event type
79
F0425 21:01:55.038295 17977 perf_parser.cc:240] Check failed:
reader_.ReadPerfSampleInfo(*parsed_event.raw_event, &sample_info) 
*** Check failure stack trace: ***
    @     0x55e6deb6058e  (unknown)
    @     0x55e6deb94a49  (unknown)
    ..
    Aborted (core dumped)

I collect data as intstructed here:
https://clang.llvm.org/docs/UsersManual.html

It is from package autofdo-0.18-4.4.x86_64 and perf 5.11.15.

Is there a way to get this working w/o using older perf?
Honza
> 
> Does LLVM's auto-FDO support non-Intel CPUs these days?
> >
> 
> It supports LBR like events, so it is CPU vendor dependent. For ARM, using
> ETM can achieve the goal, but I don't have detailed knowledge of it.
> 
> David
> 
> >
> > Honza
> > >
> > > David
> > > >
> > > >
> > > >> Honza
> > > >> >
> > > >> > David
> > > >> >
> > > >> >
> > > >> > > Honza
> > > >> > > >
> > > >> > > > David
> > > >> > > >
> > > >> > > > >
> > > >> > > > > Thoughts?
> > > >> > > > > Martin
> > > >> > > > >
> > > >> > > > > >
> > > >> > > > > > Having the tool third-party makes keeping the whole chain
> > > >> working
> > > >> > > more
> > > >> > > > > > difficult.
> > > >> > > > > >
> > > >> > > > > > Richard.
> > > >> > > > > >
> > > >> > > > > >> Thanks,
> > > >> > > > > >>
> > > >> > > > > >> David
> > > >> > > > > >>
> > > >> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <
> > hubicka@ucw.cz>
> > > >> wrote:
> > > >> > > > > >>
> > > >> > > > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > >> > > > > >>>>> GCC documentation for AutoFDO points to create_gcov tool
> > > >> that
> > > >> > > > > converts
> > > >> > > > > >>> perf.data file into gcov format that can be consumed by
> > gcc
> > > >> with
> > > >> > > > > >>> -fauto-profile (
> > > >> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> > > >> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > > >> > > > > >>>>>
> > > >> > > > > >>>>> I noticed that the source code for create_gcov has been
> > > >> deleted
> > > >> > > from
> > > >> > > > > >>> https://github.com/google/autofdo on April 7. I asked
> > about
> > > >> that
> > > >> > > > > change
> > > >> > > > > >>> in that repo and got the following reply:
> > > >> > > > > >>>>>
> > > >> > > > > >>>>>
> > > >> > > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > > >> > > > > >>>>>
> > > >> > > > > >>>>> "Actually we didn't use create_gcov and havn't updated
> > > >> > > create_gcov
> > > >> > > > > for
> > > >> > > > > >>> years, and we also didn't have enough tests to guarantee
> > it
> > > >> works
> > > >> > > (It
> > > >> > > > > was
> > > >> > > > > >>> gcc-4.8 when we used and verified create_gcov). If you
> > need
> > > >> it, it
> > > >> > > is
> > > >> > > > > >>> welcomed to update create_gcov and add it to the
> > respository."
> > > >> > > > > >>>>>
> > > >> > > > > >>>>> Does this mean that AutoFDO is currently dead in gcc?
> > > >> > > > > >>>>
> > > >> > > > > >>>> Hello.
> > > >> > > > > >>>>
> > > >> > > > > >>>> Yes. I know that even basic test cases have been broken
> > for
> > > >> years
> > > >> > > in
> > > >> > > > > the
> > > >> > > > > >>> GCC.
> > > >> > > > > >>>> It's new to me that create_gcov was removed.
> > > >> > > > > >>>>
> > > >> > > > > >>>> I tend to send patch to GCC that will remove AutoFDO from
> > > >> GCC.
> > > >> > > > > >>>> I known Bin spent some time working on AutoFDO, has he
> > came
> > > >> up to
> > > >> > > > > >>> something?
> > > >> > > > > >>>
> > > >> > > > > >>> The GCC side of auto-FDO is not that hard.  We have most
> > of
> > > >> > > > > >>> infrastructure in place, but stopping point for me was
> > always
> > > >> > > > > difficulty
> > > >> > > > > >>> to get gcov-tool working.  If some maintainer steps up, I
> > > >> think I
> > > >> > > can
> > > >> > > > > >>> fix GCC side.
> > > >> > > > > >>>
> > > >> > > > > >>> I am bit unsure how important feature it is - we have FDO
> > that
> > > >> > > works
> > > >> > > > > >>> quite well for most users but I know there are some users
> > of
> > > >> the
> > > >> > > LLVM
> > > >> > > > > >>> implementation and there is potential to tie this with
> > other
> > > >> > > hardware
> > > >> > > > > >>> events to asist i.e. if conversion (where one wants to
> > know
> > > >> how
> > > >> > > well
> > > >> > > > > CPU
> > > >> > > > > >>> predicts the jump rather than just the jump probability)
> > > >> which I
> > > >> > > always
> > > >> > > > > >>> found potentially interesting.
> > > >> > > > > >>>
> > > >> > > > > >>> Honza
> > > >> > > > > >>>>
> > > >> > > > > >>>> Martin
> > > >> > > > > >>>>
> > > >> > > > > >>>>>
> > > >> > > > > >>>>> Thanks,
> > > >> > > > > >>>>>
> > > >> > > > > >>>>> Eugene
> > > >> > > > > >>>>>
> > > >> > > > > >>>>
> > > >> > > > > >>>
> > > >> > > > >
> > > >> > > > >
> > > >> > >
> > > >>
> > > >
> >