Bytecode verifier

[Andrew Haley]

Tom Tromey writes:
 > >>>>> "Andrew" == Andrew Haley <aph@redhat.com> writes:
 > 
 > Andrew> I know that there is a somewhat decent verifier in libgcj, so
 > Andrew> perhaps we could replace the gcj verifer with that one.
 > 
 > FWIW, I started work on putting the libgcj verifier into gcj last
 > year.  I haven't had time to finish yet, and in the meantime I made
 > some major changes to the libgcj verifier (fixing the subroutine
 > bugs), so my changes won't apply cleanly any more.
 > 
 > Anyway, I had a mostly-working prototype.  I was able to get the
 > verifier into gcc, bridge the type differences, and have it actually
 > verify code.  gcc still crashed since I hadn't yet wrote the code to
 > let the libgcj verifier tell gcc about type maps.  Parts of this were
 > pretty ugly, so some polishing would also be required...
 > 
 > One nice benefit of these changes is that they would make it easier
 > for other VMs to use the libgcj verifier as well.

Excellent news.  Well, I want to raise the priority of this work.
AFAICS it's a show stopper for the new ABI.  Also, the changes to the
bytecode compiler for the new ABI will conflict with verifier changes.

 > Andrew> That's better than trying to maintain two buggy verifiers.
 > 
 > We can just maintain one buggy verifier :-).  I won't claim that the
 > libgcj verifier is bug-free, just that it uses a model that lets us
 > avoid the most common bugs.  In particular it does type unions, not
 > type merges; and it does subroutine splitting, avoiding the common
 > pitfalls and misconceptions around subroutine verification (the
 > subroutine problems particularly affect the gcj verifier).
 > 
 > I have run a lot of code through this verifier.  Unfortunately there
 > are several orders of magnitude fewer bad bytecode examples than there
 > are good ones.  So it is hard to trust any verifier's capacity to
 > detect errors.

I have quite a few failing cases.  It'd be interesting to try them.

Andrew.