Implement Virtual Memory with Memory Mapped Files

Question

Is it possible to wrap up memory mapped files something like this?

TVirtualMemoryManager = class
public
  function  AllocMem (Size : Integer) : Pointer;
  procedure FreeMem (Ptr : Pointer);
end;

Since the memory mapped file API functions all take offsets I don't know how to manage the free areas in the memory mapped files. My only idea is to implement some kind of basic memory management (mainting free lists for different block sizes) but I don' t know how efficient this will be.

EDIT: What I really want (as David made clear to me) is this:

IVirtualMemory = interface
  function  ReadMem (Addr : Int64) : TBytes;
  function  AllocateMem (Data : TBytes) : Int64;
  procedure FreeMem (Addr : Int64);
end;

I need to store continous blocks of bytes (each relatively small) in virtual memory and be able to read them back into memory using a 64-bit adress. Most of the time access is read-only. If a write is necessary I would just use FreeMem followed by AllocMem since the size will be different anyway.

I want a wrapper for a memory mapped file with this interface. Internally it has a handle to a memory mapped files and uses MapViewOfFile on each ReadMem request. The Addr 64-bit integers are just offsets into the memory mapped file. The open question is how to assign those adresses - I currently keep a list of free blocks that I maintain.

What do you mean by "virtual memory"? The system already does that for you. — David Heffernan
I want to circumvent the 2/3/4 GB memory limit and want the virtual memory backed by a file (the page file in this case). — jpfollenius
In that case you need a different interface than this. You need an interface that gets you past the address space limit and this doesn't. — David Heffernan
Can you propose an interface then? I know that you do some data-intensive stuff at work, so maybe you have experiences with that? — jpfollenius
@Smasher If there are very few FreeMems then just ignore them and view the memory as being infinite! That turns it from a hard job into a trivial one. — David Heffernan

Arnaud Bouchez Arnaud Bouchez · Accepted Answer · 2011-11-17T17:03:42

Your proposal that "Internally it has a handle to a memory mapped files and uses MapViewOfFile on each ReadMem request" will be just a waste of CPU resource, IMHO.
It is worth saying that your GetMem / FreeMem requirement won't be able to break the 3/4 GB barrier. Since all allocated memory will be mapped into memory until a call to FreeMem, you'll be short of memory space, just as with the regular Delphi memory manager. The best you can do is to rely of FastMM4, and change your program to reduce its memory use.
IMHO you'll have to change/update your specification. For instance, your "updated" question sounds just like a regular storage problem.

What you want is to be able to allocate more than 3/4 GB of data for your application. You have a working implementation of such a feature in our SynBigTable open source unit. This is a fast and light NoSQL solution in pure Delphi.

It is able to create a file of any size (only 64 bit limited), then will map the content of each record into memory, on request. It will use a memory mapping of the file, if possible. You can implement your interface very directly with TSynBigTable methods: ReadMem=Get, AllocMem=Add, FreeMem=Delete. The IDs will be your pointer-like values, and RawByteString will be used instead of TBytes.

You can access any block of data using an integer ID, or a string ID, or even use a sophisticated field layout (inside the record, or as in-memory metadata - including indexes and fast search).

Or rely on a regular embedded SQL database. For instance, SQLite3 is very good at handling BLOB fields, and is able to store huge amount of data. With a simple in-memory caching mechanism for most used records, it could be a powerful solution.

Implement Virtual Memory with Memory Mapped Files

1 Answers