Incremental compilation

[Basile Starynkevitch]

On 12/30/18 12:54 PM, Marc Nieper-Wißkirchen wrote:

> My use case is compiling a Scheme library. It contains macro
> definitions whose transformation procedures have to be compiled at
> first to expand subsequent parts of the library body. Furthermore, the
> compiled transformation procedures of global macro defintions have to
> appear in the final dynamic library as well.

I am quite interested in your work, since I did (in the past) GCC MELT 
http://starynkevitch.net/Basile/gcc-melt/ (which now is a dead project; 
it was a Lisp dialect compiled to C or C++ for GCC extensions); I am 
working on http://github.com/bstarynk/bismon right now, and you might be 
interested in reading 
http://starynkevitch.net/Basile/bismon-chariot-doc.pdf

Is your project some open-source stuff? Could you give us the URL of its 
source code?

In MELT, I also had macros similar to yours. So I believe I understand 
your work quite well (since it is similar to what I did in GCC MELT, 
which was a Lisp -but not Scheme- dialect with macros). In practice, you 
still can compile every function in a separate *.so plugin.

You could group several (unrelated) functions together by JIT compiling 
them at once, but that is an optimization.

You could, once all macros have been compiled, regenerate a single 
"shared object" for all your library, but that is an optimization. The 
main reason to do so would be to avoid wasting virtual address space by 
having many thousands of shared objects. I am not sure it is worthwhile.

Since you are implementing a Scheme, all your functions are in fact used 
in closures (which contains a function pointer). So updating all the 
required closures (to put there a function pointer to a newer code) 
might be quite easy in practice.

The potential issue is to "garbage collect" useless code. In your case, 
that would mean doing dlclose in your GC. Do you want to do that?

Since you can in practice have many hundred thousands of useless 
generated plugins, you might decide to never call dlclose (and accept 
the "code segment leak" that follows).

Again, I really am interested by your work. If you have any additional 
references (draft reports, source code, etc...) please share them with 
us if possible.


Cheers

-- 
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France