Re: possible red flag for new C++ parser

[Joe Buck <jbuck at synopsys dot com>]

On Fri, Feb 07, 2003 at 03:43:18PM -0800, Tom Lord wrote:
> Wow.  (I've just been googling around to catch up on the state of C++
> in the world.)   Quite a pickle.

The free software world is mostly over this hump, in that projects that
are still being maintained fixed their code.  That is, most of the breakage
from "C++" that wasn't really C++ at all, but just code that
happened to compile with old g++ versions, has been fixed.

> Ok, so -- start with some large body of real world code and reduce it
> to the C++ packages that compile with the old parser.

Debian packages requiring g++ would be a good collection, and it's
easy to determine which ones they are.  The Debian folks have already
done most of the required cleanups as they are most of the way through a
transition to 3.2.

> Then try with the new parser.
> 
> It would be interesting, first, to measure the number of possible
> regressions, then to sample them to get some idea of how they line up.
> 
> Statistical sampling might reveal that:
> 
> 	1) Most apparent regressions aren't regressions at all, but
>            are bugs in the C++ code that the old parser didn't catch. 
>            (false regressions)
> 
> 	2) Most apparent regresssions are regresssions -- cases of
>            good code that the new parser fails on.  (true 
>            regressions)
> 
> 	3) Somehere in the middle between (1) and (2).

The answer will be 3).  Even if it leans toward 1), we might possibly
conclude that the new parser is overly pedantic by default (e.g.
objections to stray semicolons that don't bother most other C++
compilers).

> If (1) or (3) is the case, and the number of possible regressions is
> large, then the vendors and the GNU project have a big problem.

The problem will sort itself out, but it will take work.  Fortunately, the
work tends to distribute: maintainers of a particular free software
package are the experts at fixing that package, or can figure out what
to scream to the GCC team about.

> If (3) is the case, then the nature of the "false regressions" has to
> be studied.  It may be the case that there are just a few kinds of
> package bug that explain most of the false regressions.  If so,
> perhaps they can be easilly eliminated from the test suite.  If not,
> then this isn't much different from (1).

C++ is a wildly complex language with a nearly infinite number of corner
cases.  This means that all compilers for C++ have some bugs.  Experienced
C++ programmers, especially those who have to deal with more than one C++
compiler, have learned to deal with this.  Most C++ front-end bugs
are fairly easy to work around by slightly modifying the code.  Where this
is not an easy thing to do, we then have a high-priority bug.

> My understanding is that the new parser is much faster and more
> compliant.

Nevertheless, it is new, which suggests that it will handle some of the
corner cases wrong.  The only way to proceed is to just run a lot of code
through it and analyze the results.  The analysis, unfortunately, is
labor-intensive.

> The costs of picking some body of code and doing the
> sampling are pretty moderate in the grand scheme of things -- a few
> man months, I'd think.  The payoff is large.  This seems to me like
> the kind of thing that the SC should (regularly) be able to ask the
> vendors to pony up for -- that kind of spending path and decision
> making path should be optimized.

If we had some kind of automated build farm to build large bodies of
code regularly and report on problems, that would be cool.  It would
still be labor-intensive to set the thing up and to analyze results
from it.

> And in any event, there's (ultimately, if the tools and processes
> were better) no need to go through this kind of thing on the trunk.

Certainly there is.  If you don't test the trunk, the trunk will be
broken.  You can't merge code, even perfectly working code, from muliple
branches and expect that breakage due to interaction will never occur.