[committed] analyzer: tweak priority of callstrings in worklist::key_t::cmp

[David Malcolm]

While debugging another issue I noticed that the analyzer could fail to
merge nodes for control flow in which one path had called a function
and another path hadn't:

        BB
       /  \
      /    \
 fn call   no fn call
      \    /
       \  /
     join BB

The root cause was that the worklist sort function wasn't prioritizing
call strings, and thus it was fully exploring the "no function called"
path to the exit BB, and only then exploring the "within the function call"
parts of the "funcion called" path.

This patch prioritizes call strings when sorting the worklist so that
the nodes with deeper call strings are processed before those with shallower
call strings, thus allowing such nodes to be merged at the joinpoint.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Pushed to trunk as 9d20ec97475b1102d6ca005ad165056d34615a3d.

gcc/analyzer/ChangeLog:
	* engine.cc (worklist::key_t::cmp): Move sort by call_string to
	before SCC.

gcc/testsuite/ChangeLog:
	* gcc.dg/analyzer/loop-0-up-to-n-by-1-with-iter-obj.c: Update
	expected number of enodes after the loop.
	* gcc.dg/analyzer/paths-8.c: New test.

Signed-off-by: David Malcolm <dmalcolm@redhat.com>
---
 gcc/analyzer/engine.cc                        | 25 ++++++++++++++-----
 .../loop-0-up-to-n-by-1-with-iter-obj.c       |  3 +--
 gcc/testsuite/gcc.dg/analyzer/paths-8.c       | 17 +++++++++++++
 3 files changed, 37 insertions(+), 8 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/analyzer/paths-8.c

diff --git a/gcc/analyzer/engine.cc b/gcc/analyzer/engine.cc
index 5b519fdf385..48320bc062e 100644
--- a/gcc/analyzer/engine.cc
+++ b/gcc/analyzer/engine.cc
@@ -2004,7 +2004,25 @@ worklist::key_t::cmp (const worklist::key_t &ka, const worklist::key_t &kb)
 	return cmp;
     }
 
-  /* First, order by SCC.  */
+  /* Sort by callstring, so that nodes with deeper call strings are processed
+     before those with shallower call strings.
+     If we have
+         splitting BB
+             /  \
+            /    \
+       fn call   no fn call
+            \    /
+             \  /
+            join BB
+     then we want the path inside the function call to be fully explored up
+     to the return to the join BB before we explore on the "no fn call" path,
+     so that both enodes at the join BB reach the front of the worklist at
+     the same time and thus have a chance of being merged.  */
+  int cs_cmp = call_string::cmp (call_string_a, call_string_b);
+  if (cs_cmp)
+    return cs_cmp;
+
+  /* Order by SCC.  */
   int scc_id_a = ka.get_scc_id (ka.m_enode);
   int scc_id_b = kb.get_scc_id (kb.m_enode);
   if (scc_id_a != scc_id_b)
@@ -2033,11 +2051,6 @@ worklist::key_t::cmp (const worklist::key_t &ka, const worklist::key_t &kb)
 
   gcc_assert (snode_a == snode_b);
 
-  /* The points might vary by callstring; try sorting by callstring.  */
-  int cs_cmp = call_string::cmp (call_string_a, call_string_b);
-  if (cs_cmp)
-    return cs_cmp;
-
   /* Order within supernode via program point.  */
   int within_snode_cmp
     = function_point::cmp_within_supernode (point_a.get_function_point (),
diff --git a/gcc/testsuite/gcc.dg/analyzer/loop-0-up-to-n-by-1-with-iter-obj.c b/gcc/testsuite/gcc.dg/analyzer/loop-0-up-to-n-by-1-with-iter-obj.c
index 2b0352711ae..0172c9b324c 100644
--- a/gcc/testsuite/gcc.dg/analyzer/loop-0-up-to-n-by-1-with-iter-obj.c
+++ b/gcc/testsuite/gcc.dg/analyzer/loop-0-up-to-n-by-1-with-iter-obj.c
@@ -69,6 +69,5 @@ void test(int n)
 
   free (it);
 
-  __analyzer_dump_exploded_nodes (0); /* { dg-warning "2 processed enodes" } */
-  // TODO: why 2 enodes here, rather than 1
+  __analyzer_dump_exploded_nodes (0); /* { dg-warning "1 processed enode" } */
 }
diff --git a/gcc/testsuite/gcc.dg/analyzer/paths-8.c b/gcc/testsuite/gcc.dg/analyzer/paths-8.c
new file mode 100644
index 00000000000..b350d4d7dbd
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/analyzer/paths-8.c
@@ -0,0 +1,17 @@
+#include "analyzer-decls.h"
+
+static void __attribute__((noinline))
+__analyzer_callee_1 (void)
+{
+  /* empty.  */
+}
+
+void
+test_1 (int flag)
+{
+  if (flag)
+    __analyzer_callee_1 ();
+
+  /* Verify that we merge state, whether or not the call happens.  */
+  __analyzer_dump_exploded_nodes (0); /* { dg-warning "1 processed enode" } */
+}
-- 
2.26.3