Disposing of Notes Objects with COM, Memory Leaks

Question

I using C# to interact with the Domino COM. I am using Lotus Notes 8.5.2. Visual Studio 2008, Windows 7 SP1.

I am trying to prevent this error from Lotus:

LSXBE: ************************************
LSXBE: ****** Out of Backend Memory *******
LSXBE: ************************************

The following code copies each NotesDocument in an NSF to another NSF. After getting UNIDs from a NotesView having a selection query of SELECT @All, code copies one message at a time using

NotesDocument ndoc = nd.GetDocumentByUNID(nve.UniversalID);         
ndoc.CopyToDatabase(nd2);

Given, a source NSF of 10 GB, the amount of memory (private bytes, I believe) used by my application grows steadily to about 450 MB. ANTS Memory Profiler indicates that almost all of that memory is allocated to unmanaged memory, which would be the COM.

Could I reduce memory consumption by disposing of Notes objects? I have not seen methods to deallocate a NotesSession, NotesDatabase, NotesDocument, etc. Is there a way to deallocate the memory?

I have tried a version of the code calling GC.Collect() and GC.WaitForPendingFinalizers() after every 5000 documents, that eliminated the "Out of Backend Memory" errors on a machine with 16 GB RAM. Even when copying finishes, and I set the object containing the below code to null and call garbage collection, memory utilization remains around 450 MB.

I also tried putting the copy code in its own thread and then garbage collecting after the thread completes. That did not help.

If there are not disposal methods for Notes objects, how else might I reduce memory utilization?

Code C#

//Establish session
NotesSession ns = new Domino.NotesSessionClass();
ns.Initialize("");

//Open source NSF
NotesDatabase nd = ns.GetDatabase("", "test.nsf", false);
//Open destination NSF.
//Assume that all design elements of nd2 are identical to those of nd
NotesDatabase nd2 = ns.GetDatabase("", "test2.nsf", false);

//Create view that returns all documents.
NotesView nView2 = nd.GetView("$All");
nd.CreateView("All-DR", "SELECT @ALL", nView2, false);
NotesView nView = NotesConnectionDatabase.GetView("All-DR");

//Loop through entries in the new view
NotesViewEntry nvec = nView.AllEntries;
nve = nvec.GetFirstEntry();

for (int j = 1; j <= intEntryCount; j++)
{
     if (j == 1)
     {
          nve = nvec.GetFirstEntry();
     }
     else
     {
          nve = nvec.GetNextEntry(nve);
     }

     //Copy document to second database.
     NotesDocument ndoc = nd.GetDocumentByUNID(nve.UniversalID);         
     ndoc.CopyToDatabase(nd2);
 }
 //End loop.
 //All documents are copied.

I have these ideas, none of which is appealing:

Call garbage collection after every call to CopyToDatabase. I don't expect this will work given that I believe I am dealing with a memory leak in COM. I also expect it to reduce the speed of my application.
Try the C++ API. I have no idea whether the issue exists there.
This is a really clumsy way... Build a new manager app that a. Gets a list UNIDs and writes them to a text file. b. Starts the copy app. c. The copy app copies a subset of the records from the text file. d. The copy app terminates, freeing its memory. e. The manager app starts a new copy app process. f. Repeat c - e as needed.

Putting this in as a comment since I don't have anything definitive to say. I don't believe that the Domino COM classes leak, per se. If they did, Lotus would provide a recycle() method the way they do in their Java classes. But behind those classes is the entire Notes runtime, which does allocate a lot of memory for caching, so it's not surprising if you see memory going up. — Richard Schwartz
I see no evidence of a real problem here. If your program copies 10 jiggabytes and survives without OOM then the odds for a leak are very low. It is a very common mistake to expect the VM size to reduce on a GC.Collect() call, that's just not the way memory management works in Windows. It creates empty heap blocks, they stay around for the next allocation. It is just virtual memory, it doesn't cost anything. — Hans Passant
Prior to adding GC.Collect() after every 5000th document, on a machine with 4 GB of RAM, I did receive an error like LSXBE: ****** Out of Backend Memory *******. I have just tried GC.Collect after every single document. That kept memory utilization around 75 MB, rather than 450 MB. I am hopeful that this will give me the freedom to perform additional operations on each document. The speed of my application was reduced by about 50%, and this is acceptable. — Jacob Quisenberry
@RichardSchwartz , If you could please move your comment to an answer, I would like to mark it as an answer. I believe that what you stated about the lack of recycle methods in COM classes is true. — Jacob Quisenberry
BTW, I'm not sure if you've run into this information yet so I think I should mention it... While my experience is that the Domino classes work pretty well, there are some known bugs on win64 -- specifically with the methods that return lists of design elements (views, form, etc.). Since IBM has never officially certified that the classes are supported on win64, if you do run into any bugs there's a pretty low probability that IBM will ever fix them. — Richard Schwartz

Richard Schwartz Richard Schwartz · Accepted Answer · 2013-02-16T00:14:45

I don't believe that the Domino COM classes leak, per se. If they did, Lotus would provide a recycle() method the way they do in their Java classes. But behind those classes is the entire Notes runtime, which does allocate a lot of memory for caching, so it's not surprising if you see memory going up.

Disposing of Notes Objects with COM, Memory Leaks

2 Answers