Caching one-time startup (and GNU Octave JIT)

[David Malcolm]

On Wed, 2014-01-22 at 10:44 -0500, David Malcolm wrote:
> I've been experimenting with adding libgccjit support to GNU Octave and
> I think I've found a flaw with the current libgccjit API - the lack of a
> way to cache one-time startup.
> 
> GNU Octave already has a JIT compiler, using LLVM.  Octave has a
> singleton class "jit_typeinfo" which performs one-time creation of the
> various Octave types and callbacks, for example creating tables of
> overloaded functions, mapping from (input type, function) pairs to
> optimized handlers.
> (See e.g.:
> http://octave.sourceforge.net/doxygen/html/d2/da9/jit-typeinfo_8cc_source.html#l01072
> the jit_typeinfo constructor tor, which performs this one-time
> startup).
> 
> Then, when a function or loop becomes "hot", it gets JIT-compiled from a
> parse-tree representation to a "jit_ir" representation, which then gets
> turned into LLVM IR, and thence machine code.
> 
> Visually:
> 
>   once:
>      create types, function wrappers, overload tables, etc (the
>      jit_typeinfo singleton)
> 
>   when a function becomes "hot":
>     parse tree -> jit_ir -> llvm -> machine code
> 
> In my experiment, I have a semi-working:
> 
>   when a function becomes "hot":
>     parse tree -> jit_ir -> libgccjit -> machine code
> 
> but I'm running up against the lack of a way of doing one-time startup
> that's shared between all instances of a compile.
> 
> Currently we have an idea of a context, and a compilation callback, and
> all of the "contextual" entities have a lifetime that's only within the
> compilation callback - you can't have e.g. a (gcc_jit_type *) that gets
> stored at startup for later use by other compilations within the same
> process.
> 
> This split into one-time startup vs per-hot function seems like a
> natural thing for a real-world JIT compiler to want to do.
> 
> How should libgccjit support this?   Some ideas:
> 
> (A) Nested contexts: change lifetimes so that entities have the same
> lifetime as the context they were created in, but can be used within
> "child contexts" (with nesting).
> 
> So you can do:
>    root_context: create the one-time stuff
>      child_context (for one function): create stuff for just
>         this function, using the stuff created in
>         "root_context".
> We could then perhaps eliminate result objects, so that e.g. you could
> actually compile stuff on the root_context, and use it in a
> child_context.
> 
> This would be one more entrypoint:
>   gcc_jit_context_new_child_context
> and it would make sense for anything that creates something that's
> returned to take an explicit context ptr as an arg, since that affects
> the lifetime of the returned thing.
> 
> [But how to implement? Perhaps change so that things become
> heap-allocated "memento" objects, with a "free" of them all happening in
> the release of the context (perhaps using obstack?).   Play it all back
> during the compilation hook, with 2-way mappings between the mememtos
> and the tree objects. Eventually we can (I hope) remove the mementos and
> work directly with gcc objects].
> 
> (B) Explicit separation between one-time init vs per-compile: perhaps
> have: gcc_jit_context (one-time init) vs gcc_jit_module
> (an instance of a compile on a context, roughly corresponding to a
> "Translation Unit").
> 
> (C) Something else?  Ideas?
> 
> I think typically you'd want to create types and imported functions
> *once*, and then create exported functions as a per-compile thing.  But
> I think people may want flexibility here e.g. to create some one-time
> functions at startup, and then use them in the per-hot-function code.
> So (A) may be superior to (B), perhaps.
> 
> Thoughts?  I'm most interested in what the API ought to look like [1]

As pointed out on IRC, if we go to a memento-based internal
implementation, we can kill off the need to have a callback, and the
client code can simply call the API directly.   This will solve another
issue I ran into with libgccjit in Octave, which is that liboctave is in
C++ with exception-handling, whereas libgccjit isn't exception-aware,
and thus if an C++ exception is thrown within the callback, there isn't
a good way to propagate that exception back out through all the layers.
So getting rid of the callback from the API seems worthwhile in itself,
and I think I'll attempt that next.

I thought of another place where I'm unhappy with the *lifetime* of
entities in the existing API: some JITs will want to have code-patching,
where you can place a "patch point" within code, which can be updated
later.  So e.g. you have code in a dynamic lanugage that's hot when
called on, say, a list of int objects, but you later getting
non-heterogeneous lists e.g. a mixture of int and float.  Then you can
have a JIT-compiled handler for lists with the type-guards to check for
int-ness, and the failure-handling of the type-guard can have a
patch-point, so that you can later dynamically patch in code to better
handle the new types that the code is seeing (and indeed, the
failure-handler typically actually patches itself).

So we'd want something like:
  gcc_jit_patchpoint *
  gcc_jit_function_add_patchpoint (some args);
though what the args looks like, and what you do with a patchpoint
object, I'm not sure at this time.

But to be useful the gcc_jit_patchpoint will need to have a lifetime
that exceeds the duration of the callback, hence I mention it now.

Dave