[Patch, fortran, RFC] PR 40958 Reduce size of module files

[Janne Blomqvist]

On Sat, Nov 26, 2011 at 09:29, Salvatore Filippone
<filippone.salvatore@gmail.com> wrote:
> Unfortunately, Mikael is right. The effect on the module sizes is
> spectacular, but so is the runtime.

Thanks for testing!

> An unpatched 4.7 has completed its build in about 37 minutes; with the
> patched compiler I had to stop the build after 10 hours, and it still
> was at about at 8% or less of the total work to be done.

Ouch!

> My (provisional) conclusions:
> 1. The bloat in the current module file format is definitely very bad,
> but is the kind of bloat that's handled reasonably well by zlib;
> 2.  Even in the case of the current 4.7 compiler 37 minutes is way too
> much, and from Mikael's description I suspect a large part of this is
> due to the seeks.

I did some tests with the aermod benchmark from polyhedron (with 4.4
which was what I happened to have at the machine in question, but IIRC
no major changes to module loading has happened since). Running the
-fsyntax-only compilation under "strace -c -f" showed that the parsing
and module generation/loading requires 21 million (!!!) lseek()
syscalls. Running the same briefly without the "-c" option (i.e. print
syscalls to stderr) shows long runs of repeatedly calling lseek with
exactly the same parameters with no other syscalls inbetween. So
something is definitely quite fishy here.

> So, I am afraid that the module handling code needs a radical rewrite:
> going down the zlib path requires getting rid of the seeks (something
> that would be useful even in the plain text format!). I am not at all
> sure whether this is any easier than redesigning the module format
> from the ground up.

I think it's possible to fix the current system without redesigning it
from the ground up, though one can of course argue whether starting
from a clean sheet would be easier. What I think would be needed, at
least, is roughly:

1. Parse a .mod file in one go without any seeking.

2. My patch to compress the contents (as your tests show, this step
clearly depends on step #1).

3. The "module namespace" thing, that is, parsing and storing a
representation of a module in memory once per translation unit rather
than redoing it every time we encounter a USE statement.
Unfortunately, this thing seems harder than what one would expect at
first glance; it has been talked about for years but nobody has found
the time to do it.

I think that if these three issues were solved, gfortran module
handling would be good enough for all practical purposes.

PS: Heck, what about a simple and stupid approach for #1: Just read
the mod file into a big string, and then do the parsing from there?
Then a rudimentary implementation of #3 would be to just store
pointers to all the strings we have so far read, and the mod file name
in order to look it up.

-- 
Janne Blomqvist