Ok i am only talking about the available a java solutions but the problem might be transferable to other programming languages/enviornments. While extending the memory which is available to a JVM the garbage collector will reduce the overall performance (GC takes time when you put more as 2 GB to a single VM). On the other hand this drawback is nevertheless backed by a “limitless” but -of course- slower disk store.

But – surprise surprise- Java has an out of-the-box solution for it. I am talking about the off-heap memory (term is taken from Terracotta way of wording).

For all these programs, a java.nio.ByteBuffer or Mapped Files

FileChannel channel = new RandomAccessFile(file, "r").getChannel();
ByteBuffer buf = channel.map(MapMode.READ_ONLY, 0L, file.length());

are an alternative to traditional Java objects – 6 x speedup with my testapp. I like this article http://www.kdgregory.com/index.php?page=java.byteBuffer.