[PING^2] Re: [PATCH 1/4] Add gcc-auto-profile script

[Andi Kleen <andi at firstfloor dot org>]

Andi Kleen <andi@firstfloor.org> writes:

Ping^2 for the patch series!

> Andi Kleen <andi@firstfloor.org> writes:
>
> Ping for the patch series!
>
>> From: Andi Kleen <ak@linux.intel.com>
>>
>> Using autofdo is currently something difficult. It requires using the
>> model specific branches taken event, which differs on different CPUs.
>> The example shown in the manual requires a special patched version of
>> perf that is non standard, and also will likely not work everywhere.
>>
>> This patch adds a new gcc-auto-profile script that figures out the
>> correct event and runs perf. The script is installed with on Linux systems.
>>
>> Since maintaining the script would be somewhat tedious (needs changes
>> every time a new CPU comes out) I auto generated it from the online
>> Intel event database. The script to do that is in contrib and can be
>> rerun.
>>
>> Right now there is no test if perf works in configure. This
>> would vary depending on the build and target system, and since
>> it currently doesn't work in virtualization and needs uptodate
>> kernel it may often fail in common distribution build setups.
>>
>> So Linux just hardcodes installing the script, but it may fail at runtime.
>>
>> This is needed to actually make use of autofdo in a generic way
>> in the build system and in the test suite.
>>
>> So far the script is not installed.
>>
>> gcc/:
>> 2016-03-27  Andi Kleen  <ak@linux.intel.com>
>>
>> 	* doc/invoke.texi: Document gcc-auto-profile
>> 	* gcc-auto-profile: Create.
>>
>> contrib/:
>>
>> 2016-03-27  Andi Kleen  <ak@linux.intel.com>
>>
>> 	* gen_autofdo_event.py: New file to regenerate
>> 	gcc-auto-profile.
>> ---
>>  contrib/gen_autofdo_event.py | 155 +++++++++++++++++++++++++++++++++++++++++++
>>  gcc/doc/invoke.texi          |  31 +++++++--
>>  gcc/gcc-auto-profile         |  70 +++++++++++++++++++
>>  3 files changed, 251 insertions(+), 5 deletions(-)
>>  create mode 100755 contrib/gen_autofdo_event.py
>>  create mode 100755 gcc/gcc-auto-profile
>>
>> diff --git a/contrib/gen_autofdo_event.py b/contrib/gen_autofdo_event.py
>> new file mode 100755
>> index 0000000..db4db33
>> --- /dev/null
>> +++ b/contrib/gen_autofdo_event.py
>> @@ -0,0 +1,155 @@
>> +#!/usr/bin/python
>> +# generate Intel taken branches Linux perf event script for autofdo profiling
>> +
>> +# Copyright (C) 2016 Free Software Foundation, Inc.
>> +#
>> +# GCC is free software; you can redistribute it and/or modify it under
>> +# the terms of the GNU General Public License as published by the Free
>> +# Software Foundation; either version 3, or (at your option) any later
>> +# version.
>> +#
>> +# GCC is distributed in the hope that it will be useful, but WITHOUT ANY
>> +# WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> +# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
>> +# for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with GCC; see the file COPYING3.  If not see
>> +# <http://www.gnu.org/licenses/>.  */
>> +
>> +# run it with perf record -b -e EVENT program ...
>> +# The Linux Kernel needs to support the PMU of the current CPU, and
>> +# it will likely not work in VMs.
>> +# add --all to print for all cpus, otherwise for current cpu
>> +# add --script to generate shell script to run correct event
>> +#
>> +# requires internet (https) access. this may require setting up a proxy
>> +# with export https_proxy=...
>> +#
>> +import urllib2
>> +import sys
>> +import json
>> +import argparse
>> +import collections
>> +
>> +baseurl = "https://download.01.org/perfmon";
>> +
>> +target_events = (u'BR_INST_RETIRED.NEAR_TAKEN',
>> +                 u'BR_INST_EXEC.TAKEN',
>> +                 u'BR_INST_RETIRED.TAKEN_JCC',
>> +                 u'BR_INST_TYPE_RETIRED.COND_TAKEN')
>> +
>> +ap = argparse.ArgumentParser()
>> +ap.add_argument('--all', '-a', help='Print for all CPUs', action='store_true')
>> +ap.add_argument('--script', help='Generate shell script', action='store_true')
>> +args = ap.parse_args()
>> +
>> +eventmap = collections.defaultdict(list)
>> +
>> +def get_cpu_str():
>> +    with open('/proc/cpuinfo', 'r') as c:
>> +        vendor, fam, model = None, None, None
>> +        for j in c:
>> +            n = j.split()
>> +            if n[0] == 'vendor_id':
>> +                vendor = n[2]
>> +            elif n[0] == 'model' and n[1] == ':':
>> +                model = int(n[2])
>> +            elif n[0] == 'cpu' and n[1] == 'family':
>> +                fam = int(n[3])
>> +            if vendor and fam and model:
>> +                return "%s-%d-%X" % (vendor, fam, model), model
>> +    return None, None
>> +
>> +def find_event(eventurl, model):
>> +    print >>sys.stderr, "Downloading", eventurl
>> +    u = urllib2.urlopen(eventurl)
>> +    events = json.loads(u.read())
>> +    u.close()
>> +
>> +    found = 0
>> +    for j in events:
>> +        if j[u'EventName'] in target_events:
>> +            event = "cpu/event=%s,umask=%s/" % (j[u'EventCode'], j[u'UMask'])
>> +            if u'PEBS' in j and j[u'PEBS'] > 0:
>> +                event += "p"
>> +            if args.script:
>> +                eventmap[event].append(model)
>> +            else:
>> +                print j[u'EventName'], "event for model", model, "is", event
>> +            found += 1
>> +    return found
>> +
>> +if not args.all:
>> +    cpu, model = get_cpu_str()
>> +    if not cpu:
>> +        sys.exit("Unknown CPU type")
>> +
>> +url = baseurl + "/mapfile.csv"
>> +print >>sys.stderr, "Downloading", url
>> +u = urllib2.urlopen(url)
>> +found = 0
>> +cpufound = 0
>> +for j in u:
>> +    n = j.rstrip().split(',')
>> +    if len(n) >= 4 and (args.all or n[0] == cpu) and n[3] == "core":
>> +        if args.all:
>> +            vendor, fam, model = n[0].split("-")
>> +            model = int(model, 16)
>> +        cpufound += 1
>> +        found += find_event(baseurl + n[2], model)
>> +u.close()
>> +
>> +if args.script:
>> +    print '''#!/bin/sh
>> +# profile workload for gcc profile feedback (autofdo) using Linux perf
>> +# auto generated. to regenerate for new CPUs run
>> +# contrib/gen_autofdo_event.py --shell --all in gcc source
>> +
>> +# usages:
>> +# gcc-auto-profile program             (profile program and children)
>> +# gcc-auto-profile -a sleep X          (profile all for X secs, may need root)
>> +# gcc-auto-profile -p PID sleep X      (profile PID)
>> +# gcc-auto-profile --kernel -a sleep X (profile kernel)
>> +# gcc-auto-profile --all -a sleep X    (profile kernel and user space)
>> +
>> +# identify branches taken event for CPU
>> +#
>> +
>> +FLAGS=u
>> +
>> +if [ "$1" = "--kernel" ] ; then
>> +  FLAGS=k
>> +  shift
>> +fi
>> +if [ "$1" == "--all" ] ; then
>> +  FLAGS=uk
>> +  shift
>> +fi
>> +
>> +if ! grep -q Intel /proc/cpuinfo ] ; then
>> +  echo >&2 "Only Intel CPUs supported"
>> +  exit 1
>> +fi
>> +
>> +if grep -q hypervisor /proc/cpuinfo ; then
>> +  echo >&2 "Warning: branch profiling may not be functional in VMs"
>> +fi
>> +
>> +case `egrep -q "^cpu family\s*: 6" /proc/cpuinfo &&
>> +  egrep "^model\s*:" /proc/cpuinfo | head -1` in'''
>> +    for event, mod in eventmap.iteritems():
>> +        for m in mod[:-1]:
>> +            print "model*:\ %s|\\" % m
>> +        print 'model*:\ %s) E="%s$FLAGS" ;;' % (mod[-1], event)
>> +    print '''*)
>> +echo >&2 "Unknown CPU. Run contrib/gen_autofdo_event.py --all --script to update script."
>> +	exit 1 ;;'''
>> +    print "esac"
>> +    print 'exec perf record -e $E -b "$@"'
>> +
>> +if cpufound == 0 and not args.all:
>> +    sys.exit('CPU %s not found' % cpu)
>> +
>> +if found == 0:
>> +    sys.exit('Branch event not found')
>> diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
>> index 9e54bb7..427d89a 100644
>> --- a/gcc/doc/invoke.texi
>> +++ b/gcc/doc/invoke.texi
>> @@ -8249,13 +8249,34 @@ which are generally profitable only with profile feedback available:
>>  If omitted, it defaults to @file{fbdata.afdo} in the current directory.
>>  
>>  Producing an AutoFDO profile data file requires running your program
>> -with the @command{perf} utility on a supported GNU/Linux target system.
>> +with the @command{gcc-auto-profile} utility on a supported GNU/Linux target system. @command{gcc-auto-profile} calls the @command{perf} utility.
>> +It also requires Last-Branch-Record support, which typically requires
>> +a new enough kernel not running virtualized.
>> +@command{gcc-auto-profile} accepts the same arguments as @command{perf record}.
>>  For more information, see @uref{https://perf.wiki.kernel.org/}.
>>  
>> -E.g.
>>  @smallexample
>> -perf record -e br_inst_retired:near_taken -b -o perf.data \
>> -    -- your_program
>> +gcc-auto-profile your_program
>> +@end smallexample
>> +
>> +On larger programs the resulting perf.data file may be very large.
>> +In this case it can be better to reduce the sampling rate.
>> +Collect samples every million taken branches:
>> +
>> +@smallexample
>> +gcc-auto-profile -c 1000000 program
>> +@end smallexample
>> +
>> +Or only profile representative run intervals of the program:
>> +
>> +@smallexample
>> +gcc-auto-profile -p PID-OF-PROGRAM sleep 5
>> +@end smallexample
>> +
>> +Profile complete system for 10 seconds (may require root)
>> +
>> +@smallexample
>> +gcc-auto-profile -a sleep 10
>>  @end smallexample
>>  
>>  Then use the @command{create_gcov} tool to convert the raw profile data
>> @@ -8266,7 +8287,7 @@ See @uref{https://github.com/google/autofdo}.
>>  E.g.
>>  @smallexample
>>  create_gcov --binary=your_program.unstripped --profile=perf.data \
>> -    --gcov=profile.afdo
>> +    --gcov=profile.afdo -gcov_version 1
>>  @end smallexample
>>  @end table
>>  
>> diff --git a/gcc/gcc-auto-profile b/gcc/gcc-auto-profile
>> new file mode 100755
>> index 0000000..c6712b2
>> --- /dev/null
>> +++ b/gcc/gcc-auto-profile
>> @@ -0,0 +1,70 @@
>> +#!/bin/sh
>> +# profile workload for gcc profile feedback (autofdo) using Linux perf
>> +# auto generated. to regenerate for new CPUs run
>> +# contrib/gen_autofdo_event.py --shell --all in gcc source
>> +
>> +# usages:
>> +# gcc-auto-profile program             (profile program and children)
>> +# gcc-auto-profile -a sleep X          (profile all for X secs, may need root)
>> +# gcc-auto-profile -p PID sleep X      (profile PID)
>> +# gcc-auto-profile --kernel -a sleep X (profile kernel)
>> +# gcc-auto-profile --all -a sleep X    (profile kernel and user space)
>> +
>> +# identify branches taken event for CPU
>> +#
>> +
>> +FLAGS=u
>> +
>> +if [ "$1" = "--kernel" ] ; then
>> +  FLAGS=k
>> +  shift
>> +fi
>> +if [ "$1" == "--all" ] ; then
>> +  FLAGS=uk
>> +  shift
>> +fi
>> +
>> +if ! grep -q Intel /proc/cpuinfo ] ; then
>> +  echo >&2 "Only Intel CPUs supported"
>> +  exit 1
>> +fi
>> +
>> +if grep -q hypervisor /proc/cpuinfo ; then
>> +  echo >&2 "Warning: branch profiling may not be functional in VMs"
>> +fi
>> +
>> +case `egrep -q "^cpu family\s*: 6" /proc/cpuinfo &&
>> +  egrep "^model\s*:" /proc/cpuinfo | head -1` in
>> +model*:\ 55|\
>> +model*:\ 77|\
>> +model*:\ 76) E="cpu/event=0xC4,umask=0xFE/p$FLAGS" ;;
>> +model*:\ 42|\
>> +model*:\ 45|\
>> +model*:\ 58|\
>> +model*:\ 62|\
>> +model*:\ 60|\
>> +model*:\ 69|\
>> +model*:\ 70|\
>> +model*:\ 63|\
>> +model*:\ 61|\
>> +model*:\ 71|\
>> +model*:\ 86|\
>> +model*:\ 78|\
>> +model*:\ 94) E="cpu/event=0xC4,umask=0x20/p$FLAGS" ;;
>> +model*:\ 46|\
>> +model*:\ 30|\
>> +model*:\ 31|\
>> +model*:\ 26|\
>> +model*:\ 47|\
>> +model*:\ 37|\
>> +model*:\ 44) E="cpu/event=0x88,umask=0x40/p$FLAGS" ;;
>> +model*:\ 28|\
>> +model*:\ 38|\
>> +model*:\ 39|\
>> +model*:\ 54|\
>> +model*:\ 53) E="cpu/event=0x88,umask=0x41/p$FLAGS" ;;
>> +*)
>> +echo >&2 "Unknown CPU. Run contrib/gen_autofdo_event.py --all --script to update script."
>> +	exit 1 ;;
>> +esac
>> +exec perf record -e $E -b "$@"

-- 
ak@linux.intel.com -- Speaking for myself only