It means that the garbage collector's attempt to obtain more memory
from the OS (on Linux it uses sbrk() or mmap()) failed.
By default the GC uses as much memory as it thinks it needs and the
OS will give it. Try running with the GC_PRINT_STATS environment
variable defined, and see what that says. Also keep something like
"top" running to see whether you are really running out of
memory/swap space. (I'm assuming a Linux environment.)
In general, for a large Java heap, you are better off running
a recent gcj. And ideally the collector should be built with
-DLARGE_CONFIG, which I don't think it is by default.