Bytecode verifier

[Tom Tromey]

>>>>> "Andrew" == Andrew Haley <aph@redhat.com> writes:

Andrew> I know that there is a somewhat decent verifier in libgcj, so
Andrew> perhaps we could replace the gcj verifer with that one.

FWIW, I started work on putting the libgcj verifier into gcj last
year.  I haven't had time to finish yet, and in the meantime I made
some major changes to the libgcj verifier (fixing the subroutine
bugs), so my changes won't apply cleanly any more.

Anyway, I had a mostly-working prototype.  I was able to get the
verifier into gcc, bridge the type differences, and have it actually
verify code.  gcc still crashed since I hadn't yet wrote the code to
let the libgcj verifier tell gcc about type maps.  Parts of this were
pretty ugly, so some polishing would also be required...

One nice benefit of these changes is that they would make it easier
for other VMs to use the libgcj verifier as well.

Andrew> That's better than trying to maintain two buggy verifiers.

We can just maintain one buggy verifier :-).  I won't claim that the
libgcj verifier is bug-free, just that it uses a model that lets us
avoid the most common bugs.  In particular it does type unions, not
type merges; and it does subroutine splitting, avoiding the common
pitfalls and misconceptions around subroutine verification (the
subroutine problems particularly affect the gcj verifier).

I have run a lot of code through this verifier.  Unfortunately there
are several orders of magnitude fewer bad bytecode examples than there
are good ones.  So it is hard to trust any verifier's capacity to
detect errors.

Tom