[Patch, fortran, RFC] PR 40958 Reduce size of module files

[Tobias Schlüter]

On 2011-11-29 16:51, Janne Blomqvist wrote:
> On Mon, Nov 28, 2011 at 17:00, Paul Richard Thomas
> <paul.richard.thomas@gmail.com>  wrote:
>> If the quoted ratio between gfortran and NAG module file sizes are
>> typical, this would be a very high return exercise.
>
> I suspect this is not worth doing. As I've mentioned before, the Go
> developers claim that one major reason why go compilation is so fast
> is that transitive dependencies are included (go stores the "module"
> information in the .o file, but the principle is similar to Fortran
> modules). See pages 9-10 on
>
>   http://assets.en.oreilly.com/1/event/45/Another%20Go%20at%20Language%20Design%20Presentation.pdf

The gain/loss ratio depends a lot on whether exponential growth of 
leaves of the USE-tree is actually taking place.  In a Fortran program, 
I'd venture to say it's the unlikely case, i.e. most modules are USEd in 
several different places, so there are lots of loops in the trees.  On 
the other hand, we're having exponential growth of module file sizes 
which is a real issue.

> Also, Joost pointed out that this might break the trick we use to
> avoid recompilation cascades. We store a checksum in the module file,
> and replace a module file only if the checksum is different when
> recompiling. If one then uses makefiles which specify dependencies in
> terms of mod files, this prevents unnecessary recompilation cascades,
> making development on large code bases much faster.
>
> Consider if we have a module A that uses module B which uses module C.
> If C changes such that C.mod also changes, then if A.mod and B.mod
> contain "USE" statements, then both A and B have to be recompiled as
> well. However, in the current scheme where we include transitive
> dependencies, if B.mod doesn't change (say, B used an ONLY clause to
> only import things which didn't change due to the C changes), then the
> recompilation cascade stops at B, and A doesn't need to be recompiled
> (nor further modules using A, and so forth).

This also depends very much on the structure of the codebase.  If there 
are lots of USE, ONLY and only few transitive USEs (i.e., module B uses 
module A, and module C uses module B but also module A), then nothing is 
gained with our current scheme.

> So yes, the current approach will inevitably use more disk space, but
> in this case I think it's worth it. There are, however, plenty of
> other opportunities for reducing disk space usage. My zlib patch being
> a relatively simple approach, that can of course be combined with
> other changes. For instance, we could store attributes and stuff only
> where they differ from the default and where they makes sense. E.g. no
> need to specify a module variable as UNKNOWN-INTENT since intent makes
> sense only for procedure dummy arguments. And so on.

None of these address the exponential growth of module file sizes, 
though.  Salvatore's observation that his kitchen sink module is a lot 
smaller than the sum of the modules it included shows that in his case 
there's a lot of redundancy between the intermediate modules which 
allows for some nice zip-like behavior when merging symbols from two 
modules, but this wouldn't be needed, nor would it happen, if the 
individual modules were not redundant.  In other words, in Salvatore's 
case our scheme is losing lots in the intermeidate steps as a lot of 
information that turns out to be redundant is stored in intermediate steps.

On the issue of caching the module files: if one uses a lisp-like data 
structure it should really be simple to keep the module files in-memory, 
as the encoding is meant to be lisp-like: lists composed of atoms, one 
could even encode the atoms as strings, yielding a very lightweight parser.

Cheers,
- Tobi