This is the mail archive of the
gcc-patches@gcc.gnu.org
mailing list for the GCC project.
contrib: Gcov Combined Coverage Report (GCCR)
- From: Charlie Schluting <charlie at schluting dot com>
- To: gcc-patches at gcc dot gnu dot org
- Cc: gcov-capstone at cs dot pdx dot edu, Janis Johnson <janis187 at us dot ibm dot com>
- Date: Thu, 08 Sep 2005 22:47:59 -0700
- Subject: contrib: Gcov Combined Coverage Report (GCCR)
Hello,
For a recent class project four students at Portland State University
worked on creating a tool for combining gcov reports from multiple
targets. This project was proposed to us by the folks at the IBM Linux
Technology Center, and supervised by Janis Johnson.
The manpage is attached as gccr.pod, and gccr.pl itself is also attached.
The manpage pretty much sums it up.. gccr is run by listing <filename
tag> on the command line, with as many .gcov files as you want to
combine. The "tag" is just an identifier used later to identify which
.gcov file we're talking about (see below). The -c option will generate
output that looks like:
<combined count> : <line number> : <src>
And without, it looks like:
<line number> : <src>
<count 1>: <line number>: <tag 1>
<count 2>: <line number>: <tag 2>
Most exciting, is also produces a useful summary:
44.44% of 9 lines executed on target <tag 1>
84.62% of 13 lines executed on target <tag 2>
100.00% of 13 lines executed across all files
We'd be grateful for any feedback, and please consider commiting this to
gcc/contrib.
Regards,
-Charlie
Content-Type: text/enriched
Text-Width: 60
d1 NAME
gccr - gcov combined coverage reports
=head1 SYNOPSIS
B<gccr filespec tag filespec tag [filespec tag]>
=head1 DESCRIPTION
The gcov combined coverage report (GCCR) merges multiple
.gcov files into a single .gccr report. Use it to compare
the coverage of different test cases, or to compare the
coverage of the same test on different targets. gccr can
help you gage the coverage of your test suite, identify
redundant test cases, and verify cross compiles.
Software developers also use coverage testing in concert
with testsuites, to make sure software is actually good
enough for a release. Testsuites can verify that a program
works as expected; a coverage program tests to see how much
of the program is exercised by the testsuite. gccr measures
the performance of a testsuite, permitting users to:
. adapt their testsuite to different compilation targets
. remove cases made redundant by other sets of cases
. tailor their suite to maximum coverage with reasonable
run times
=head2 To perform coverage testing:
1. Compile your source code with -fprofile-arcs and
-ftest-coverage.
2. Run your first test set. This creates a file with
the .gcda or .da suffix,depending upon the target.
3. Run gcov, supplying your source name. This creates a
file with the .gcov suffix.
4. Move the .gcov file to a new name.
5. Delete the .gcda file.
6. Repeat steps 2 - 5, generating a uniquely named .gcov
for each test set.
7. Run gccr on the re-named .gcov files produced in step 4.
8. Analyze your report.
=head2 Output Choices
gccr prints either detailed (the default) or combined (-c) coverage
reports. The respective formats are:
With -c option:
<####> or <combined count> : <line number> : <src>
Without -c option:
<line number> : <src>
<####> or <count 1>: <line number>: <tag 1>
<####> or <count 2>: <line number>: <tag 2>
where:
line number : the line number in the source file
src : the line text from the source file
combined count: the number of times all tests executed the line, a total
count : the number of times this test executed the line
tag : the name of a test that executed it
Special indicators on the report are:
B< ####> in the execution count means that the line never executed.
B< -:> in the line number means that that line is a comment.
=head2 To run gccr:
Type gccr followed by <tag filename> pairs, where "tag" is
an ID you make up and "filename" is the file spec of a .gcov
file. gccr will tag the lines that a test set executes with
the ID that you supply here. You can keep the [ tag
filename ] pairs in a file supplied with the -f option or
re-directed as standard in.
=head2 To analyze the gccr report:
The report fragment below shows a subroutine as it was executed
by the tests tagged "=A=," "=B=," and "DEFAULT."
-: 0:Source:simple.c
-: 0:Object:simple.bb
-: 1:/* Test Gcov basics. */
-: 2:
-: 3:/* { dg-options "-fprofile-arcs -ftest-coverage" } */
-: 4:/* { dg-do run { target native } } */
-: 5:
-: 6:void noop ()
-: 7:{
10: 7:=A=
30: 7:=B=
30: 7:DEFAULT
-: 8:}
-: 9:
-: 10:int main ()
-: 11:{
1: 11:=A=
3: 11:=B=
3: 11:DEFAULT
-: 12: int i;
1: 12:=A=
3: 12:=B=
3: 12:DEFAULT
-: 13:
-: 14: for (i = 0; i << 10; i++) /* count(11) */
11: 14:=A=
33: 14:=B=
33: 14:DEFAULT
-: 15: noop (); /* count(10) */
10: 15:=A=
30: 15:=B=
30: 15:DEFAULT
-: 16:
-: 17: return 0; /* count(1) */
1: 17:=A=
3: 17:=B=
3: 17:DEFAULT
-: 18:}
-: 19:
-: 20:/* { dg-final { run-gcov gcov-1.c } } */
=head1 OPTIONS
B<-c>, B<--combined>
prints one line for every source line. The execution count shows
the number of executions by any test, e.g. if test =A= executed
line #14 once, test =B= executed it twice and =DEFAULT= skipped
that line, the report would show:
.
.
3: 14: for (i = 0; i << 10; i++)
.
.
B<-h>, B<--help>
Display help about using gccr (on the standard output), and exit
without doing any further processing.
B<-n>, B<--nosummary>
removes the % lines from the report's end, e.g. leaves off
50.00% of 14 lines executed on target =A=
92.86% of 14 lines executed on target =B=
21.43% of 14 lines executed on target DEFAULT
B<-t> I<filespec>, B<--tagfile>=I<filespec>
reads the tag-filespec assignments from filespec. Enter the
file-tag pairs one pair to a line, separating the file from
the tag by white space. GCCR treats anything after the tag
as a comment. You can enter more than one tagfile on the
command line, and mix tagfiles with directly entered file-tags.
B<-v>, B<--version>
Display the gccr version number (on the standard output), and exit
without doing any further processing.
=head1 EXAMPLES
Run gccr on three test sets, whose files you
saved under the names "run1," "run2," and "run3." You
instruct gccr to tag lines exercised by run 1 with
"=A=," those exercised by run 2 with "=B=," and those
exercised by run 3 with the word "DEFAULT."
gccr run1 =A= run2 =B= run3 DEFAULT
Avoid typos by capturing the previous file-tag setup
in the file "testAM," then run gccr on it:
# cat - > testAM
> run1 =A=
> run2 =B=
> run3 DEFAULT
^D
# gccr -f testAM
=head1 EXIT STATUS
gccr returns a zero exist status if it succeeds in creating
a report. It returns nonzero in case of failure, with an
explanation on STDOUT.
=head1 AUTHOR
Nick Groesz <groesz@cs.pdx.edu>
Jesse Burkett <ransom@cs.pdx.edu>
Charlie Schluting <manos@cat.pdx.edu>
Dickson Patton <pattond@cs.pdx.edu>
=head1 SEE ALSO
gcov(1) gcc(1) gprof(1)
#!/usr/bin/perl -w
#
# Copyright (C) 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002,
# 2003, 2004, 2005 Free Software Foundation, Inc.
# This Makefile.in is free software; the Free Software Foundation
# gives unlimited permission to copy and/or distribute it,
# with or without modifications, as long as this notice is preserved.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY, to the extent permitted by law; without
# even the implied warranty of MERCHANTABILITY or FITNESS FOR A
# PARTICULAR PURPOSE.
#
# version 0.4.11 : Nick Groesz : fix potential divide by zero
# version 0.4.10 : Nick Groesz : fixed summary in combined coverage, ignore function data, added copyright
# version 0.4.9 : Nick Groesz : added combined reporting in print_summary(), changed usage text
# version 0.4.8 : Nick Groesz : added -c option (combined coverage)
# version 0.4.7 : Nick Groesz : fixed formatting, added comments
# version 0.4.6 : Dickson Patton : fixed tagfile option, right justify counts
# version 0.4.5 : Nick Groesz : list code generated with #define macros
# version 0.4.4 : Dickson Patton : added tagfile option
# version 0.4.3 : Nick Groesz : changed around internal data structures, start of version history
use strict;
use Getopt::Long;
# prototypes
sub read_args(); # read in command line arguments
sub process_files(); # run through each file
sub parse_execution_data($$); # parse the data from each file
sub print_results(); # print gcov data
sub print_summary(); # print summary (similary to gcov's summary)
sub print_usage(); # print gccr usage text
our $tool_name = 'gccr'; # name of script
our $version = 'gccr (GCC) 0.4.10'; # version of script
our $copyright = 'Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
'; # copyright notice
# internal data
our @files; # gcov file data
#
# File information:
# file_number - ranges from 0 to NUMBER OF FILES SPECIFIED - 1
# $file[file_number]{'tag'} - user specified file tag
# $file[file_number]{'name'} - name of file to be read and parsed
#
# $file[file_number[{'data'} contains parsed gcov data
#
# Line information:
# line_number - corresponds to the line number of the GCOV file
# and ranges from 1 to NUMBER OF LINES
# $data{'line'}[line_number]{'type'} - type of line
# Can be set to:
# no_ex - non-executing code (gcov meta-data, header code, code that has been #ifdef'd out)
# line - an executable line of code
# branch - branch execution information
# call - call execution information
#
# $data{'line'}[line_number]{'count'} - number of times an executable line, branch, or call was executed
# $data{'line'}[line_number]{'raw'} - raw data straight from gcov file
# this is only populated in the first file's data structure
# $data{'line'}[line_number]{'line_num'} - line number in original source code
# Execution information:
# $data{'line_count'} - number of executable lines
# $data{'execution_count'} - number of executed lines
our $file_count = 0; # number of files read
our $line_count = 0; # number of lines in gcov data files (should be the same for each file)
our $executable_total = 0; # number of different executable lines across all files
our $executed_total = 0; # number of different executable lines across all files that were executed
# options
our $opt_combined = 0;
our $opt_help = 0; # help option
our $opt_version = 0; # version option
our @tag_files = (); # tagfile(s) option
our $opt_nosummary = 0; # do not print summary
read_args();
process_files();
print_results();
unless($opt_nosummary) {
print_summary();
}
# Summary: read command line arguments:
# Parameters: none
# Return: none
sub read_args()
{
unless( GetOptions(
'combined' => \$opt_combined,
'help' => \$opt_help,
'no-summary' => \$opt_nosummary,
'tagfile|file=s' => \@tag_files,
'version' => \$opt_version,
)
) {
# if GetOptions returns FALSE, then incorrect options were specified
# exit with error
print_usage();
exit(1);
}
if($opt_help) {
print_usage();
exit(0);
}
if($opt_version) {
print "$version\n";
print "$copyright\n";
exit(0);
}
# if any number of tag files were specified, then we read tagfiles instead of command line arugments
if(scalar(@tag_files)) {
@files = &read_tagfiles(@tag_files);
}else {
# number of file arguments + number of tag arguments should equal an even number
if(@ARGV % 2 == 1) {
print("ERROR: file count does not match tag count\n");
exit(1);
}
my $i = 0;
while($ARGV[$i]) {
push @files, {name => $ARGV[$i], tag => $ARGV[$i+1]};
$i+=2;
}
}
$file_count = scalar(@files);
if($file_count < 2 ) {
print("ERROR: at least two files must be specified\n");
exit(1);
}
}
# Summary: read tagfiles specifie dwith the -t option
# Parameters: array of names of tagfiles
# Return: array of name/tag hashes
sub read_tagfiles($)
{
my @tag_files = @_;
my @files;
my $l = 0; # count of --tagfile=___ options
my $m = 0; # count of line in the current tagfile
foreach my $file(@tag_files) {
$l++;
open(TAGFILE, $file) || die "ERROR: on open of tagfile $l, $file: ($!)\n";
while(<TAGFILE>) {
$m++;
chomp $_;
if ( /(^([^ ]+) *(.*)$)/ ) {
push @files, {name => $2, tag => $3};
}
else {
die "ERROR: invalid file-tag pair on line $m of tagfile $l\n";
}
}
close(TAGFILE);
}
return(@files);
}
# Summary: run through all the gcov files and call the parsing function
# Parameters: none
# Return: none
sub process_files()
{
# the first file is used to gather raw data
$files[0]{'data'} = parse_execution_data($files[0]{'name'},1);
for(my $i=1;$i<$file_count;$i++) {
$files[$i]{'data'} = parse_execution_data($files[$i]{'name'},0);
}
}
# Summary: parse the gcov file, populating the %data structure
# Parameters: name of file to parse | boolean indicating whether raw (original gcov) data should be saved
# save_raw is set to 1 on the first file parse and set to zero thereafter
# Returns: reference to data hash
sub parse_execution_data($$)
{
my($file,$save_raw) = @_;
my %data;
$data{'line_count'} = 0; # number of executable lines in file
$data{'execution_count'} = 0; # number of lines that were executed in file
stat($file);
if(!(-r _)) {
die("ERROR: cannot read file: $file\n");
}
if(!(-f _)) {
die("ERROR: not a plain file: $file\n");
}
open(FILE_HANDLE,$file) || die("ERROR: cannot open file $file: $!");
my $file_line_num = 0;
while(<FILE_HANDLE>) {
my $line = $_;
$file_line_num++;
chomp $line;
if($line =~ /^\s+-:\s+(\d+):(.*)/) {
# line is gcov preamble or non-executing code
my $line_num = $1;
my $raw = $2;
$data{'line'}[$file_line_num]{'type'} = 'no_ex';
$data{'line'}[$file_line_num]{'line_num'} = $line_num;
if($save_raw) {
$data{'line'}[$file_line_num]{'raw'} = $2;
}
}elsif($line =~ /^\s+#####:\s+(\d+):(.*)/) {
# line was not executed
my $line_num = $1;
my $raw = $2;
$data{'line'}[$file_line_num]{'count'} = 0;
$data{'line'}[$file_line_num]{'type'} = 'code';
$data{'line'}[$file_line_num]{'line_num'} = $line_num;
$data{'line_count'}++;
if($save_raw) {
$data{'line'}[$file_line_num]{'raw'} = $raw;
}
}elsif($line =~ /^\s+(\d+):\s+(\d+):(.*)/) {
# line was executed
my $count = $1;
my $line_num = $2;
my $raw = $3;
$data{'line'}[$file_line_num]{'count'} = $count;
$data{'line'}[$file_line_num]{'type'} = 'code';
$data{'line'}[$file_line_num]{'line_num'} = $line_num;
$data{'line_count'}++;
$data{'execution_count'}++;
if($save_raw) {
$data{'line'}[$file_line_num]{'raw'} = $raw;
}
}elsif($line =~ /^branch\s+(\d+)/) {
# line contains branch execution information
my $branch_num = $1;
$data{'line'}[$file_line_num]{'num'} = $branch_num;
if($line =~ /^branch\s+\d+\s+never executed/) {
$data{'line'}[$file_line_num]{'count'} = 0;
$data{'line'}[$file_line_num]{'type'} = 'branch';
}elsif($line =~ /^branch\s+\d+\s+taken (\d+)%/) {
$data{'line'}[$file_line_num]{'count'} = $1;
$data{'line'}[$file_line_num]{'type'} = 'branch';
}
if($save_raw) {
$data{'line'}[$file_line_num]{'raw'} = $line;
}
}elsif($line =~ /^call\s+(\d+)/) {
# line contains call execution information
my $call_num = $1;
$data{'line'}[$file_line_num]{'num'} = $call_num;
if($line =~ /^call\s+\d+\s+never executed/) {
$data{'line'}[$file_line_num]{'count'} = 0;
$data{'line'}[$file_line_num]{'type'} = 'call';
}elsif($line =~ /^call\s+\d+\s+returns (\d+)%/) {
$data{'line'}[$file_line_num]{'count'} = $1;
$data{'line'}[$file_line_num]{'type'} = 'call';
}
if($save_raw) {
$data{'line'}[$file_line_num]{'raw'} = $line;
}
}elsif($line =~ /^function/i) {
# function data is ignored
}else {
# line could not be parsed
print("ERROR: cannot parse line $file_line_num in file $file\n Is this a valid gcov file?\n");
exit(1);
}
}
close(FILE_HANDLE);
# check to see if we should save an overall line count (common to all gcov files)
if($save_raw) {
$line_count = $file_line_num;
}
return(\%data);
}
# Summary: print interpolated gcov information
# Parameters: none
# Return: none
sub print_results()
{
for(my $line_i=1;$line_i<=$line_count;$line_i++) {
my $raw_printed = 0; # boolean flag to print out the line slurped in from the gcov file
my $count_sum = 0; # sum of executions across all files for this one line- used for combined coverage reporting
my $never_exec = 1; # boolean flag that is set to 0 when the current line is executed or executable
# in any file. used in combined coverage reporting.
my $first_code_line_executed = 0; # boolean flag that indicates whether this is the first unique
# executed line of code to be found among the gcov files
for(my $file_i=0;$file_i<$file_count;$file_i++) {
# Note that each file is cycled through for every line even if just the raw
# data from the first file that ends up being printed. This is because the same
# line may be non-executing in one file and executable in another file (because code
# may be ifdef'd out).
my $type = $files[$file_i]{'data'}{'line'}[$line_i]{'type'};
if($type eq 'no_ex') {
# non-executing code
unless($raw_printed || $opt_combined) {
print "\t-:\t$files[$file_i]{'data'}{'line'}[$line_i]{'line_num'}:$files[0]{'data'}{'line'}[$line_i]{'raw'}\n";
$raw_printed = 1;
}
# nothing additional is printed for non-executing code
}elsif($type eq 'code') {
# code that is executable
if($never_exec) {
# this code is exectuable, so we indicate that in the never_exec flag
$never_exec = 0;
# we only want the number of UNIQUE lines across files that are executable
# in the executable_total flag, so this is only incremented once for all
# identical lines across each file
$executable_total++;
}
unless($raw_printed || $opt_combined) {
print "\t\t$files[0]{'data'}{'line'}[$line_i]{'line_num'}:\t$files[0]{'data'}{'line'}[$line_i]{'raw'}\n";
$raw_printed = 1;
}
my $count = $files[$file_i]{'data'}{'line'}[$line_i]{'count'};
if($opt_combined) {
$count_sum += $count;
if($first_code_line_executed == 0 && $count > 0) {
$executed_total++;
$first_code_line_executed = 1;
}
}else {
if($count == 0) {
print ' ####';
}else {
unless($first_code_line_executed) {
# update the unique count of code lines executed
# across all gcov files
$executed_total++;
$first_code_line_executed = 1;
}
my $padding = 9;
$padding -= length($count);
printf("%*s%d",$padding,' ',$count);
}
print ":\t$files[$file_i]{'data'}{'line'}[$line_i]{'line_num'}: ";
print "$files[$file_i]{'tag'}\n";
}
}elsif($type eq 'branch') {
# branch information
my $count = $files[$file_i]{'data'}{'line'}[$line_i]{'count'};
if($opt_combined) {
$count_sum += $count;
}else {
if($count == 0) {
print "branch $files[$file_i]{'data'}{'line'}[$line_i]{'num'} never executed:$files[$file_i]{'tag'}\n";
}else {
print "branch $files[$file_i]{'data'}{'line'}[$line_i]{'num'} taken $count%:$files[$file_i]{'tag'}\n";
}
}
}elsif($type eq 'call') {
# call information
my $count = $files[$file_i]{'data'}{'line'}[$line_i]{'count'};
if($opt_combined) {
$count_sum += $count;
}else {
if($count == 0) {
print "call $files[$file_i]{'data'}{'line'}[$line_i]{'num'} never executed:$files[$file_i]{'tag'}\n";
}else {
print "call $files[$file_i]{'data'}{'line'}[$line_i]{'num'} returns $count%:$files[$file_i]{'tag'}\n";
}
}
}
}
if($opt_combined) {
# if the combined coverage flag is set then no information is printed in the above for loop
# count information is summed into $count_sum and printed on a single line
my $type = $files[0]{'data'}{'line'}[$line_i]{'type'};
if($type eq 'no_ex' || $type eq 'code') {
# line is either non-executable or executable code
if($never_exec) {
# code line is not executable in any file
print "\t-:\t$files[0]{'data'}{'line'}[$line_i]{'line_num'}:$files[0]{'data'}{'line'}[$line_i]{'raw'}\n";
}else {
# line is executable in at least one file
if($count_sum == 0) {
print ' ####';
}else {
my $padding = 9;
$padding -= length($count_sum);
printf("%*s%d",$padding,' ',$count_sum);
}
print ":\t$files[0]{'data'}{'line'}[$line_i]{'line_num'}:\t$files[0]{'data'}{'line'}[$line_i]{'raw'}\n";
}
}elsif($type eq 'branch') {
# branch information
if($count_sum == 0) {
print "branch $files[0]{'data'}{'line'}[$line_i]{'num'} never executed\n";
}else {
my $percentage = $count_sum / $file_count;
print "branch $files[0]{'data'}{'line'}[$line_i]{'num'} taken $percentage%\n";
}
}elsif($type eq 'call') {
# call information
if($count_sum == 0) {
print "call $files[0]{'data'}{'line'}[$line_i]{'num'} never executed\n";
}else {
my $percentage = $count_sum / $file_count;
print "call $files[0]{'data'}{'line'}[$line_i]{'num'} returns $percentage%\n";
}
}
}
}
}
# Summary: prints the line execution percentages for each file and a percentage for all files combined
# Parameters: none
# Return: none
sub print_summary()
{
for(my $file_i=0;$file_i<$file_count;$file_i++) {
my $file_line_count = $files[$file_i]{'data'}{'line_count'};
my $percentage;
if($file_line_count) {
$percentage = ($files[$file_i]{'data'}{'execution_count'} / $file_line_count) * 100;
}else {
$percentage = 0;
}
$percentage = sprintf('%.2f',$percentage);
print "$percentage% of $file_line_count lines executed on target $files[$file_i]{'tag'}\n";
}
my $overall_percentage;
if($executable_total) {
$overall_percentage = ($executed_total / $executable_total) * 100;
}else {
$overall_percentage = 0;
}
$overall_percentage = sprintf('%.2f',$overall_percentage);
print "$overall_percentage% of $executable_total lines executed across all files\n";
}
# Summary: print tool usage information
# Parameters: none
# Return: none
sub print_usage()
{
print <<END_USAGE;
Usage: ./$tool_name [options] <file name> <target id> <file name> <target id> [file name] [target id]...
Use $tool_name to compare gcov files generated on different platforms or targets.
-h, --help Print this help, then exit
-v, --version Print version number, then exit
Input Options:
-t, --tagfile Take file-tag assignments from a file, not from command-line
Ouput Options:
-c, --combined Print combined coverage
-n, --no-summmary Do not print summary
END_USAGE
}