Re: Counting amount of memory allocated?

[Chris Lattner <sabre at nondot dot org>]

Warning, X Rated code below.  Do not allow minors to see this abomination.
:)

On Tue, 5 Nov 2002, Phil Edwards wrote:
> On Mon, Nov 04, 2002 at 10:57:01PM -0600, Chris Lattner wrote:
> > Another idea... is there a (perhaps non-portable, perhaps ugly) way to ask
> > the libstdc++ pool allocator how much memory is allocated but not used
> > yet?
>
> Not really.  The free shared area is delimited by two pointers, and when
> we need to mark a chunk as taken, the function just subtracts them to find
> the memory remaining.

Ok, sounds good...

> You can add a hack to your local library if you want, as a public member
> function in the pool allocator.  :-)  Look for a line like
>     __bytes_remaining = _S_end_free - _S_start_free;
> and duplicate it.

Ok, I'm trying to work with an unmodified libstdc++ & the associated
headers, so this hack is really, truely gross, but it seems close to
working (attached below).

This method seems to work really well, except when all memory is free'd
(and other times it seems a bit off as well I guess).  The problem is that
after allocating the map and freeing stuff back, the numbers are
inaccurate.  Specifically, mallinfo claims a certain number of bytes have
been allocated, but the pool allocation stuff seems to think that it has
more bytes in its heap than have been allocated through malloc.  Because
of this, I'm seeing the following:

map destroyed:
 Absolute Malloc bytes allocated: 2206496
Slab memory free Size: 90856
Heap Size: 2492032
  For Size[8]: 27 Free entries
  For Size[16]: 30 Free entries
  For Size[24]: 100020 Free entries
Total Free Mem: 2492032
Total Alloc Mem: -285536

...as an example.  (the full source is below).  The rundown of this is
that mallinfo claims 2206496 bytes of memory are allocated.  The pool
allocator thinks (_S_end_free-_S_start_free) is 90856.  The _S_heap_size
holds 2492032 as the total heap size.  This total heap size is > the
number of bytes returned as allocated by mallinfo.  How can this be?

As it turns out, the "free memory calculation" routines correctly find
that all pool allocator memory is free (total free mem = heap size), but
the total allocated memory size (allocated bytes - total free mem) is off.
Anyone have any ideas why the numbers are off?

Thanks, for all of the help, I feel very close to my goal.  The test
program with commentary is below.

-Chris

-- 
http://llvm.cs.uiuc.edu/
http://www.nondot.org/~sabre/Projects/


// First, I need to get access to the private static variables in
// std::__default_alloc_template<true, 0>.  Since they're private, I can't
// get to the directly, so I do this nastyness:

#include <malloc.h>
#include <map>
#include <iostream>

extern "C" {
  // Enum values copied from stl_alloc.h
  enum {_ALIGN = 8};
  enum {_MAX_BYTES = 128};
  enum {_NFREELISTS = _MAX_BYTES / _ALIGN};

  // The basic structure of _Obj in __default_alloc_template:
  struct FreeList { FreeList *Next; };

  // The private static variables, in mangled form
  FreeList *_ZNSt24__default_alloc_templateILb1ELi0EE12_S_free_listE[_NFREELISTS];
  char *_ZNSt24__default_alloc_templateILb1ELi0EE13_S_start_freeE;
  char *_ZNSt24__default_alloc_templateILb1ELi0EE11_S_end_freeE;
  size_t _ZNSt24__default_alloc_templateILb1ELi0EE12_S_heap_sizeE;
}

// Now I don't want to have to deal with these mangled names directly, so
// I do this:

FreeList* (&TheFreeList)[_NFREELISTS] = _ZNSt24__default_alloc_templateILb1ELi0EE12_S_free_listE;
char * &StartFree = _ZNSt24__default_alloc_templateILb1ELi0EE13_S_start_freeE;
char * &EndFree   = _ZNSt24__default_alloc_templateILb1ELi0EE11_S_end_freeE;
size_t &HeapSize  = _ZNSt24__default_alloc_templateILb1ELi0EE12_S_heap_sizeE;

// Now I have my "printmem" function that prints statistics of memory
// allocated:

void printmem(const char *Msg) {
  // get the number of bytes that mallinfo thinks are allocated
  struct mallinfo MI = mallinfo();
  std::cerr << Msg << "\n Absolute Malloc bytes allocated: " << MI.uordblks << "\n";

  // Size of the free slab section...
  int FreePoolMem = (int)(EndFree-StartFree);
  std::cerr << "Slab memory free Size: " << FreePoolMem << "\n";
  std::cerr << "Heap Size: " << HeapSize << "\n";

  // Walk all of the free lists, adding memory to the free counter whenever we
  // have a free bucket.
  for (unsigned i = 0; i != _NFREELISTS; ++i) {
        unsigned NumEntries = 0;
        for (FreeList *FL = TheFreeList[i]; FL; ++NumEntries, FL = FL->Next) /*empty*/ ;
        if (NumEntries)
                std::cerr << "  For Size[" << (i+1)*_ALIGN << "]: " << NumEntries << " Free entries\n";
        FreePoolMem += NumEntries*(i+1)*_ALIGN;
  }

  std::cerr << "Total Free Mem: " << FreePoolMem << "\n";
  std::cerr << "Total Alloc Mem: " << (MI.uordblks-FreePoolMem) << "\n";
  std::cerr << "\n";
}

// Now the test driver:

int main () {
  printmem("Start...");
  {
  std::map<int, float> M;
  printmem("constructed map: ");
  for (unsigned i = 0; i != 100000; ++i)
    M[-i] = i;
  printmem("map full: ");
   M.clear();
  printmem("map cleared");
  }

  printmem("map destroyed: ");
}