Re: VTA guality assessment: better than -O0 ;-)

[Richard Guenther <richard dot guenther at gmail dot com>]

On Sat, Jun 13, 2009 at 9:29 AM, Alexandre Oliva<aoliva@redhat.com> wrote:
> So, after I tested and installed this patch
> http://gcc.gnu.org/ml/gcc-patches/2009-06/msg00903.html I started
> looking closely at the guality (debug info quality) test results.
>
> So far, I have only added two very simple tests to the guality
> testsuite, but they already show very promising results.
>
> GUALCHKXPRVAL("expr", value, maybe_dead_p) checks whether expr,
> evaluated by the debugger, matches value, evaluated at run time.
> maybe_dead_p indicates whether, if the debugger fails to compute expr
> (say, optimized away or missing debug info), we get an UNRESOLVED or a
> FAIL. ?If expr is evaluated successfully but it doesn't match the
> expected value, we get a FAIL, otherwise a PASS.
>
> GUALCHKXPR(expr) is the same as GUALCHKXPRVAL("expr",(expr),1)
>
> GUALCHKFLA(expr) is nearly equivalent to GUALCHKXPRVAL("expr",(expr),0),
> except that (expr) is saved in a temporary and stored in a volatile
> memory location after the check, so expr *must* be live at the point of
> check. ?FLA stands for Forced Live After.
>
> Ok, on to some test results (on x86_64-linux-gnu):
>
> guality.c:
>
> int
> main (int argc, char *argv[])
> {
> ?int i = argc+1;
> ?int j = argc-2;
> ?int k = 5;
>
> ?GUALCHKXPR (argc);
> ?GUALCHKXPR (i);
> ?GUALCHKXPR (j);
> ?GUALCHKXPR (k);
> ?GUALCHKXPR (&i);
> ?GUALCHKFLA (argc);
> ?GUALCHKFLA (i);
> ?GUALCHKFLA (j);
> ?GUALCHKXPR (i);
> ?GUALCHKXPR (j);
> ?GUALCHKXPRVAL ("k", 5, 1);
> ?GUALCHKXPRVAL ("0x40", 64, 0);
> }
>
> -O0 (implied -fno-var-tracking-assignments)
>
> PASS: argc is 1
> PASS: i is 2
> FAIL: j is 32767, not -1
> FAIL: k is 2028276576, not 5
> PASS: &i is 140735221664248
> PASS: argc is 1
> PASS: i is 2
> FAIL: j is 32767, not -1
> PASS: i is 2
> FAIL: j is 32767, not -1
> FAIL: k is 2028276576, not 5
> PASS: 0x40 is 64
> FAIL: 7 PASS, 5 FAIL, 0 UNRESOLVED
>
> -O1 -fno-var-tracking-assignments
>
> PASS: argc is 1
> FAIL: i is 0, not 2
> PASS: j is -1
> UNRESOLVED: k is not computable, expected 5
> UNRESOLVED: &i is not computable, expected 140733781777644
> PASS: argc is 1
> PASS: i is 2
> PASS: j is -1
> FAIL: i is 0, not 2
> PASS: j is -1
> UNRESOLVED: k is not computable, expected 5
> PASS: 0x40 is 64
> FAIL: 7 PASS, 2 FAIL, 3 UNRESOLVED
>
> We see that debug info got better for j, and k is no longer wrong: it is
> completely dropped from debug information, even though it could have
> been encoded in standard debug info, for its value is the same constant
> throughout its entire lifetime. ?For some reason, although i is
> addressable, it's not uniformly computable: taking its address doesn't
> work in between two sucessful evaluations of i, and &i actually works in
> the debugger at those surrounding execution points.
>
> -O1 -fvar-tracking-assignments
>
> PASS: argc is 1
> UNRESOLVED: i is optimized away, expected 2
> PASS: j is -1
> UNRESOLVED: k is not computable, expected 5
> UNRESOLVED: &i is not computable, expected 140735863643740
> PASS: argc is 1
> PASS: i is 2
> PASS: j is -1
> UNRESOLVED: i is optimized away, expected 2
> PASS: j is -1
> UNRESOLVED: k is not computable, expected 5
> PASS: 0x40 is 64
> PASS: 7 PASS, 0 FAIL, 5 UNRESOLVED
>
> Yay, PASS! ?With Jakub's patch to encode constants in location lists,
> the two UNRESOLVED tests for k will become PASS. ?I haven't looked into
> why i is taken as optimized away. ?What I do know is that i isn't
> tracked by VTA, for VTA only tracks variables that aren't addressable.
>
> Not convinced yet? ?Why, sure, VTA (+ DW_OP_implicit_value) has *only*
> gone from totally broken j and k at -O0 to totally correct debug info,
> while somehow *fixing* additional errors for i that is not even tracked
> by VTA. ?From 0% to 100% correctness, and 100% completeness for all
> VTA-tracked variables.
>
> But it gets better.
>
> Remember those examples from the slides in the VTA presentation in last
> year's GCC Summit? ?http://people.redhat.com/~aoliva/papers/vta/
>
> I've turned them into another guality test, that runs with GUALCHK(expr)
> defined GUALCHKXPRVAL("expr", expr, 0), i.e., it requires variables to
> be available and computable.
>
> typedef struct list {
> ?struct list *n;
> ?int v;
> } elt, *node;
>
> node
> find_val (node c, int v, node e)
> {
> ?while (c < e)
> ? ?{
> ? ? ?GUALCHK (c);
> ? ? ?GUALCHK (v);
> ? ? ?GUALCHK (e);
> ? ? ?if (c->v == v)
> ? ? ? ?return c;
> ? ? ?GUALCHK (c);
> ? ? ?GUALCHK (v);
> ? ? ?GUALCHK (e);
> ? ? ?c++;
> ? ?}
> ?return NULL;
> }
>
> node
> find_prev (node c, node w)
> {
> ?while (c)
> ? ?{
> ? ? ?node o = c;
> ? ? ?c = c->n;
> ? ? ?GUALCHK (c);
> ? ? ?GUALCHK (o);
> ? ? ?GUALCHK (w);
> ? ? ?if (c == w)
> ? ? ? ?return o;
> ? ? ?GUALCHK (c);
> ? ? ?GUALCHK (o);
> ? ? ?GUALCHK (w);
> ? ?}
> ?return NULL;
> }
>
> node
> check_arr (node c, node e)
> {
> ?if (c == e)
> ? ?return NULL;
> ?e--;
> ?while (c < e)
> ? ?{
> ? ? ?GUALCHK (c);
> ? ? ?GUALCHK (e);
> ? ? ?if (c->v > (c+1)->v)
> ? ? ? ?return c;
> ? ? ?GUALCHK (c);
> ? ? ?GUALCHK (e);
> ? ? ?c++;
> ? ?}
> ?return NULL;
> }
>
> node
> check_list (node c, node t)
> {
> ?while (c != t)
> ? ?{
> ? ? ?node n = c->n;
> ? ? ?GUALCHK (c);
> ? ? ?GUALCHK (n);
> ? ? ?GUALCHK (t);
> ? ? ?if (c->v > n->v)
> ? ? ? ?return c;
> ? ? ?GUALCHK (c);
> ? ? ?GUALCHK (n);
> ? ? ?GUALCHK (t);
> ? ? ?c = n;
> ? ?}
> ?return NULL;
> }
>
> struct list testme[] = {
> ?{ &testme[1], ?2 },
> ?{ &testme[2], ?3 },
> ?{ &testme[3], ?5 },
> ?{ &testme[4], ?7 },
> ?{ &testme[5], 11 },
> ?{ NULL, 13 },
> };
>
> int
> main (int argc, char *argv[])
> {
> ?int n = sizeof (testme) / sizeof (*testme);
> ?node first, last, begin, end, ret;
>
> ?GUALCHKXPR (n);
>
> ?begin = first = &testme[0];
> ?last = &testme[n-1];
> ?end = &testme[n];
>
> ?GUALCHKXPR (first);
> ?GUALCHKXPR (last);
> ?GUALCHKXPR (begin);
> ?GUALCHKXPR (end);
>
> ?ret = find_val (begin, 13, end);
> ?GUALCHK (ret);
> ?assert (ret == last);
>
> ?ret = find_prev (first, last);
> ?GUALCHK (ret);
> ?assert (ret == &testme[n-2]);
>
> ?ret = check_arr (begin, end);
> ?GUALCHK (ret);
> ?assert (!ret);
>
> ?ret = check_list (first, last);
> ?GUALCHK (ret);
> ?assert (!ret);
> }
>
>
> -O0 (-fno-var-tracking-assignments implied)
>
> FAIL: n is 0, not 6
> FAIL: first is 16187408, not 6297888
> FAIL: last is 249832090421, not 6297968
> FAIL: begin is 140735655442656, not 6297888
> FAIL: end is 139872232800256, not 6297984
> PASS: c is 6297888
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297888
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297904
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297904
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297920
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297920
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297936
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297936
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297952
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297952
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297968
> PASS: v is 13
> PASS: e is 6297984
> FAIL: ret is 66, not 6297968
> PASS: c is 6297904
> FAIL: o is 55840872688, not 6297888
> PASS: w is 6297968
> PASS: c is 6297904
> FAIL: o is 55840872688, not 6297888
> PASS: w is 6297968
> PASS: c is 6297920
> FAIL: o is 55840872688, not 6297904
> PASS: w is 6297968
> PASS: c is 6297920
> FAIL: o is 55840872688, not 6297904
> PASS: w is 6297968
> PASS: c is 6297936
> FAIL: o is 55840872688, not 6297920
> PASS: w is 6297968
> PASS: c is 6297936
> FAIL: o is 55840872688, not 6297920
> PASS: w is 6297968
> PASS: c is 6297952
> FAIL: o is 55840872688, not 6297936
> PASS: w is 6297968
> PASS: c is 6297952
> FAIL: o is 55840872688, not 6297936
> PASS: w is 6297968
> PASS: c is 6297968
> FAIL: o is 55840872688, not 6297952
> PASS: w is 6297968
> FAIL: ret is 66, not 6297952
> PASS: c is 6297888
> PASS: e is 6297968
> PASS: c is 6297888
> PASS: e is 6297968
> PASS: c is 6297904
> PASS: e is 6297968
> PASS: c is 6297904
> PASS: e is 6297968
> PASS: c is 6297920
> PASS: e is 6297968
> PASS: c is 6297920
> PASS: e is 6297968
> PASS: c is 6297936
> PASS: e is 6297968
> PASS: c is 6297936
> PASS: e is 6297968
> PASS: c is 6297952
> PASS: e is 6297968
> PASS: c is 6297952
> PASS: e is 6297968
> FAIL: ret is 66, not 0
> PASS: c is 6297888
> FAIL: n is 6297968, not 6297904
> PASS: t is 6297968
> PASS: c is 6297888
> FAIL: n is 6297968, not 6297904
> PASS: t is 6297968
> PASS: c is 6297904
> FAIL: n is 6297968, not 6297920
> PASS: t is 6297968
> PASS: c is 6297904
> FAIL: n is 6297968, not 6297920
> PASS: t is 6297968
> PASS: c is 6297920
> FAIL: n is 6297968, not 6297936
> PASS: t is 6297968
> PASS: c is 6297920
> FAIL: n is 6297968, not 6297936
> PASS: t is 6297968
> PASS: c is 6297936
> FAIL: n is 6297968, not 6297952
> PASS: t is 6297968
> PASS: c is 6297936
> FAIL: n is 6297968, not 6297952
> PASS: t is 6297968
> PASS: c is 6297952
> PASS: n is 6297968
> PASS: t is 6297968
> PASS: c is 6297952
> PASS: n is 6297968
> PASS: t is 6297968
> FAIL: ret is 66, not 0
> FAIL: 93 PASS, 26 FAIL, 0 UNRESOLVED
>
> -O2 -fno-var-tracking-assignments:
>
> UNRESOLVED: n is not computable, expected 6
> UNRESOLVED: first is not computable, expected 6297408
> UNRESOLVED: last is not computable, expected 6297488
> UNRESOLVED: begin is not computable, expected 6297408
> UNRESOLVED: end is not computable, expected 6297504
> PASS: c is 6297408
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297408
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297424
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297424
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297440
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297440
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297456
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297456
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297472
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297472
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297488
> PASS: v is 13
> PASS: e is 6297504
> FAIL: ret is not computable, expected 6297488
> PASS: c is 6297424
> FAIL: o is optimized away, expected 6297408
> PASS: w is 6297488
> PASS: c is 6297424
> FAIL: o is optimized away, expected 6297408
> PASS: w is 6297488
> PASS: c is 6297440
> FAIL: o is optimized away, expected 6297424
> PASS: w is 6297488
> PASS: c is 6297440
> FAIL: o is optimized away, expected 6297424
> PASS: w is 6297488
> PASS: c is 6297456
> FAIL: o is optimized away, expected 6297440
> PASS: w is 6297488
> PASS: c is 6297456
> FAIL: o is optimized away, expected 6297440
> PASS: w is 6297488
> PASS: c is 6297472
> FAIL: o is optimized away, expected 6297456
> PASS: w is 6297488
> PASS: c is 6297472
> FAIL: o is optimized away, expected 6297456
> PASS: w is 6297488
> PASS: c is 6297488
> FAIL: o is optimized away, expected 6297472
> PASS: w is 6297488
> FAIL: ret is not computable, expected 6297472
> FAIL: c is 6297424, not 6297408
> PASS: e is 6297488
> FAIL: c is 6297424, not 6297408
> PASS: e is 6297488
> FAIL: c is 6297440, not 6297424
> PASS: e is 6297488
> FAIL: c is 6297440, not 6297424
> PASS: e is 6297488
> FAIL: c is 6297456, not 6297440
> PASS: e is 6297488
> FAIL: c is 6297456, not 6297440
> PASS: e is 6297488
> FAIL: c is 6297472, not 6297456
> PASS: e is 6297488
> FAIL: c is 6297472, not 6297456
> PASS: e is 6297488
> FAIL: c is 6297488, not 6297472
> PASS: e is 6297488
> FAIL: c is 6297488, not 6297472
> PASS: e is 6297488
> FAIL: ret is not computable, expected 0
> PASS: c is 6297408
> PASS: n is 6297424
> PASS: t is 6297488
> PASS: c is 6297408
> PASS: n is 6297424
> PASS: t is 6297488
> PASS: c is 6297424
> PASS: n is 6297440
> PASS: t is 6297488
> PASS: c is 6297424
> PASS: n is 6297440
> PASS: t is 6297488
> PASS: c is 6297440
> PASS: n is 6297456
> PASS: t is 6297488
> PASS: c is 6297440
> PASS: n is 6297456
> PASS: t is 6297488
> PASS: c is 6297456
> PASS: n is 6297472
> PASS: t is 6297488
> PASS: c is 6297456
> PASS: n is 6297472
> PASS: t is 6297488
> PASS: c is 6297472
> PASS: n is 6297488
> PASS: t is 6297488
> PASS: c is 6297472
> PASS: n is 6297488
> PASS: t is 6297488
> FAIL: ret is not computable, expected 0
> FAIL: 91 PASS, 23 FAIL, 5 UNRESOLVED
>
> Not much worse than -O0... ?Not only because some variables got
> optimized away, but also because they got optimized in ways that I
> “predicted” in the Summit, causing debug info to map to incorrect
> locations.
>
> -O2 -fvar-tracking-assignments:
>
> UNRESOLVED: n is not computable, expected 6
> UNRESOLVED: first is not computable, expected 6297408
> UNRESOLVED: last is not computable, expected 6297488
> UNRESOLVED: begin is not computable, expected 6297408
> UNRESOLVED: end is not computable, expected 6297504
> PASS: c is 6297408
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297408
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297424
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297424
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297440
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297440
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297456
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297456
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297472
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297472
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297488
> PASS: v is 13
> PASS: e is 6297504
> FAIL: ret is 1, not 6297488
> PASS: c is 6297424
> PASS: o is 6297408
> PASS: w is 6297488
> PASS: c is 6297424
> PASS: o is 6297408
> PASS: w is 6297488
> PASS: c is 6297440
> PASS: o is 6297424
> PASS: w is 6297488
> PASS: c is 6297440
> PASS: o is 6297424
> PASS: w is 6297488
> PASS: c is 6297456
> PASS: o is 6297440
> PASS: w is 6297488
> PASS: c is 6297456
> PASS: o is 6297440
> PASS: w is 6297488
> PASS: c is 6297472
> PASS: o is 6297456
> PASS: w is 6297488
> PASS: c is 6297472
> PASS: o is 6297456
> PASS: w is 6297488
> PASS: c is 6297488
> PASS: o is 6297472
> PASS: w is 6297488
> FAIL: ret is 1, not 6297472
> PASS: c is 6297408
> PASS: e is 6297488
> PASS: c is 6297408
> PASS: e is 6297488
> PASS: c is 6297424
> PASS: e is 6297488
> PASS: c is 6297424
> PASS: e is 6297488
> PASS: c is 6297440
> PASS: e is 6297488
> PASS: c is 6297440
> PASS: e is 6297488
> PASS: c is 6297456
> PASS: e is 6297488
> PASS: c is 6297456
> PASS: e is 6297488
> PASS: c is 6297472
> PASS: e is 6297488
> PASS: c is 6297472
> PASS: e is 6297488
> FAIL: ret is 1, not 0
> PASS: c is 6297408
> PASS: n is 6297424
> PASS: t is 6297488
> PASS: c is 6297408
> PASS: n is 6297424
> PASS: t is 6297488
> PASS: c is 6297424
> PASS: n is 6297440
> PASS: t is 6297488
> PASS: c is 6297424
> PASS: n is 6297440
> PASS: t is 6297488
> PASS: c is 6297440
> PASS: n is 6297456
> PASS: t is 6297488
> PASS: c is 6297440
> PASS: n is 6297456
> PASS: t is 6297488
> PASS: c is 6297456
> PASS: n is 6297472
> PASS: t is 6297488
> PASS: c is 6297456
> PASS: n is 6297472
> PASS: t is 6297488
> PASS: c is 6297472
> PASS: n is 6297488
> PASS: t is 6297488
> PASS: c is 6297472
> PASS: n is 6297488
> PASS: t is 6297488
> FAIL: ret is 1, not 0
> FAIL: 110 PASS, 4 FAIL, 5 UNRESOLVED
>
> Pretty good, eh? ?And the 4 failures are easy to fix: ret is noted in
> debug info as being held in a call-clobbered register at the time of the
> call to the function that implements GUALCHK. ?It was saved in another
> call-saved register, but the annotation that says so, right after the
> call, only takes effect after the call.
>
> Although this could be fixed by changing the logic that emits debug info
> before calls, so as to note the change in location earlier, this would
> be wrong on two accounts: if you're at the pc of the call, you want to
> be able to inspect the variable, and you want all locations in which
> it's live, so that they can all be modified.
>
> So the proper fix would be to emit annotations right after the call that
> take effect after we perform the call, but before it returns, i.e.,
> annotations at next_pc-1, rather than at next_pc. ?This would fix the
> only 4 errors above.
>
> As for the UNRESOLVED bits, n would be fixed with DW_OP_implicit_value,
> but the other four would require DW_OP_stack_value and some additional
> fix in cfgexpand.c, to expand to (plus (symbol_ref) (const_int)) the
> debug stmts that bind first, last, begin and end to &testme[<index>] at
> the tree optimized dump. ?Then it would be another 100% pass rate,
> without any UNRESOLVED whatsoever (all variables are tracked by VTA in
> this testcase).
>
> Could it get any better?

Yes, I don't like -O0 producing worse debug info - what does
the -O0 -fvar-tracking-assignments results look like?

Richard.

> --
> Alexandre Oliva, freedom fighter ? ?http://FSFLA.org/~lxoliva/
> You must be the change you wish to see in the world. -- Gandhi
> Be Free! -- http://FSFLA.org/ ? FSF Latin America board member
> Free Software Evangelist ? ? ?Red Hat Brazil Compiler Engineer
>