This is the mail archive of the
mailing list for the GCC project.
Re: Something is broken in repack
Linus Torvalds <email@example.com> writes:
> On Tue, 11 Dec 2007, Jon Smirl wrote:
>> > So if you want to use more threads, that _forces_ you to have a bigger
>> > memory footprint, simply because you have more "live" objects that you
>> > work on. Normally, that isn't much of a problem, since most source files
>> > are small, but if you have a few deep delta chains on big files, both the
>> > delta chain itself is going to use memory (you may have limited the size
>> > of the cache, but it's still needed for the actual delta generation, so
>> > it's not like the memory usage went away).
>> This makes sense. Those runs that blew up to 4.5GB were a combination
>> of this effect and fragmentation in the gcc allocator. Google
>> allocator appears to be much better at controlling fragmentation.
> Yes. I think we do have some case where we simply keep a lot of objects
> around, and if we are talking reasonably large deltas, we'll have the
> whole delta-chain in memory just to unpack one single object.
Eh, excuse me. unpack_delta_entry()
- first unpacks the base object (this goes recursive);
- uncompresses the delta;
- applies the delta to the base to obtain the target object;
- frees delta;
- frees (but allows it to be cached) the base object;
- returns the result
So no matter how deep a chain is, you keep only one delta at a time in
core, not whole delta-chain in core.
> So what you actually want to do is to just re-use already packed delta
> chains directly, which is what we normally do. But you are explicitly
> looking at the "--no-reuse-delta" (aka "git repack -f") case, which is why
> it then blows up.
While that does not explain, as Nico pointed out, the huge difference
between the two repack runs that have different starting pack, I would
say it is a fair thing to say. If you have a suboptimal pack (i.e. not
enough reusable deltas, as in the 2.1GB pack case), do run "repack -f",
but if you have a good pack (i.e. 300MB pack), don't.