Re: GC leaks debugging

[Erik Groeneveld <erik at cq2 dot nl>]

>> The tests work fine on OpenJDK. ÂWhat could cause GCJ to grow the heap
>> infinitely?
>
> I don't know because you won't show me the tests.

Yeah, of course, sorry.  I forgot to tell a few things.

This problem has been bothering me quite some time now, and I decided
to solve it once and forever.  I have written many different test, but
I am still not able to pinpoint a simple program to demonstrate the
results.  The problem occurred initially while using Lucene, but later
on also with Owlim.  Having written all kinds of test programs with
Lucene, from doing almost nothing to fully fletched indexing, I can
make no definite conclusions yet.

I am now circling around the problem, trying to enclose it from
different sides and I seek your help for giving me hints on what to
look for.  It is not lightly that I decided to bring it into this
mailing-list, knowing that it would claim many peoples time.

Now the test I am running is attached.  It indexes a very simple
document with a unique id each, first assuring is it deleted.  And
each loop, it reopens the index-reader and searcher.  This test starts
to get in trouble above 10,000,000 loops (documents).  The problem is
that when I remove code (I tested systematically), it only takes
longer for the heap to explode. The only test that ran properly was
when I only created Documents and not index them.  So perhaps it has
to do something with I/O.

I built the lucene dso with (full script attached):

gcj -shared -fPIC $JARFILE -o liblucene-core.so -Lgccinstall/lib -lgcj
-findirect-dispatch -fno-indirect-classes

and the test with (full scipt attached):

g++ -O0 -fPIC -g -o test test.cpp \
    -I../include/lucene \
    -L../gccinstall/lib64 \
    -lgcj \
    -L.. \
    -llucene-core \
    -findirect-dispatch \

GCC being the 4.6 branch from SVN, build with:

../gcc-4_6-branch/configure --prefix=$installdir --disable-multilib

I understand that this is a high-level approach, and you'll like a
smaller test that demonstrates the problem.  But I don't have that
yet.  Any suggestions you can come up with at this level are more than
welcome.

Meanwhile, I dive deeper into analysis of the heap.

Erik

#include <gcj/cni.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fstream>
#include <iostream>

#include "java/lang/Throwable.h"
#include "java/lang/Integer.h"
#include "java/lang/Boolean.h"
#include "java/util/Collection.h"
#include "java/util/ArrayList.h"
#include "java/io/File.h"

#include "org/apache/lucene/index/Term.h"
#include "org/apache/lucene/index/IndexReader.h"
#include "org/apache/lucene/index/IndexReader$FieldOption.h"
#include "org/apache/lucene/index/IndexWriter.h"
#include "org/apache/lucene/index/IndexWriter$MaxFieldLength.h"
#include "org/apache/lucene/search/IndexSearcher.h"
#include "org/apache/lucene/document/Document.h" 
#include "org/apache/lucene/document/Field.h" 
#include "org/apache/lucene/document/Field$Index.h" 
#include "org/apache/lucene/document/Field$Store.h" 
#include "org/apache/lucene/document/Fieldable.h" 
#include "org/apache/lucene/analysis/standard/StandardAnalyzer.h" 
#include "org/apache/lucene/analysis/Analyzer.h" 
#include "org/apache/lucene/util/Version.h"
#include "org/apache/lucene/store/Directory.h"
#include "org/apache/lucene/store/LockFactory.h"
#include "org/apache/lucene/store/SimpleFSDirectory.h"
#include "org/apache/lucene/store/SimpleFSLockFactory.h"

using namespace org::apache::lucene;
using namespace org::apache::lucene::util;
using namespace org::apache::lucene::index;
using namespace org::apache::lucene::document;
using namespace org::apache::lucene::search;
using namespace org::apache::lucene::store;
using namespace java::lang;
using namespace java::util;

Directory* makeDirectory(String* path) {
    LockFactory* lockFactory = new SimpleFSLockFactory();
    return (Directory*) new SimpleFSDirectory(new java::io::File(path), lockFactory);
}

void _Jv_RunGC(void);
long _Jv_GCTotalMemory (void);

int main(int argc, char *argv[]) {
    JvCreateJavaVM(NULL);
    JvAttachCurrentThread(NULL, NULL);
    
    JvInitClass(&Integer::class$);
    JvInitClass(&IndexReader::class$);
    JvInitClass(&IndexWriter::class$);
    JvInitClass(&IndexWriter$MaxFieldLength::class$);
    JvInitClass(&IndexReader$FieldOption::class$);
    JvInitClass(&Field::class$);
    JvInitClass(&Field$Store::class$);
    JvInitClass(&Field$Index::class$);
    JvInitClass(&Version::class$);

    store::Directory* indexDirectory = makeDirectory(JvNewStringUTF("index2"));
    String* fieldName = JvNewStringUTF("field");
    String* term = JvNewStringUTF("term");
    String* idFieldName = JvNewStringUTF("id");

    index::IndexWriter* writer = NULL;
    try { 
        analysis::Analyzer* analyzer = new analysis::standard::StandardAnalyzer(util::Version::LUCENE_30);
        writer = new IndexWriter(indexDirectory, analyzer, true, IndexWriter$MaxFieldLength::UNLIMITED);
        writer->close();
        writer = new IndexWriter(indexDirectory, analyzer, true, IndexWriter$MaxFieldLength::UNLIMITED);
        IndexReader* reader = IndexReader::open(indexDirectory);
        IndexSearcher* searcher = new IndexSearcher(reader);
        while ( true ) {
            static int i = 0;
            if (i++ % 1000 == 0) {
                printf("%d %d\n", i, _Jv_GCTotalMemory());
                fflush(stdout);
            }
            Document* doc = new Document();
            Fieldable* field = (Fieldable*) new Field(fieldName, term, Field$Store::YES, Field$Index::ANALYZED);
            doc->add(field);
            String* id = Integer::toString(i);
            Fieldable* idField = (Fieldable*) new Field(idFieldName, id, Field$Store::YES, Field$Index::ANALYZED);
            doc->add(idField);
            Term* term = new Term(idFieldName, id);
            writer->deleteDocuments(term);
            writer->addDocument(doc);
            searcher->close();
            reader->close();
            reader = IndexReader::open(indexDirectory);
            Collection* fields = reader->getFieldNames(IndexReader$FieldOption::ALL);
            searcher = new IndexSearcher(reader);
        }
    }
    catch (Throwable* e) {
        e->printStackTrace();
        if (writer != NULL) {
            writer->close();
        }
        return 1;
    }
    writer->close();
    
    return 0;
}

Attachment: buildlucenelib.sh
Description: Bourne shell script

Attachment: buildAndRun.sh
Description: Bourne shell script