Difference between NotesDocument objects from database and in-memory?

Question

I have noticed in one of my Java agents that when storing a lot of Document objects in a Hashmap I get OutofMemoryException. What I'm trying to achieve is this:

I want to populate my Domino database with information from an Oracle database. The logic is to get all from Oracle and then get all documents from Domino database. All new from Oracle is to be saved and all not in Oracle but in database shall be removed from Domino. A sort of replication of you will…

I start with looping all documents in the Domino database and store them in a HashMap with UNID as key and Document object as value:

dominoHash.put(doc.getUniversalId(), doc);

Then I loop the resultset and creates in-memory Document objects:

Document doc = db.createDocument();
doc.replaceItemValue("MyField", rset.getString("MYCOLUMN");
oracleHash.put(rset.getString("UNID"), doc);

In my TEST environment I have +20000 Domino documents and the same in Oracle. In this case they are identical. The OutOfMemoryException occurs always when trying to store in oracleHash even if I have excluded all Domino document logic from the code to "save memory"…

I thought the objects I'm storing in the different Hashmaps are the same but some how they are not. I can successfully store my Domino Document objects in a Hashmap but not my Oracle document objects.

To make it more strange the document stored from Domino has more fields than the Oracle view has columns.

Why could be the cause of this?

In this specific case I use org.openntf.domino API and haven't tried native API (yet).

Can I confirm, is it a Java OutOfMemoryException or causing a panic because of LookupHandles? Apparently the limit of Domino handles for R9 is 150,000, so it shouldn't hit that. But because you're putting the documents into a Map, they can't be recycled by the API, because they're still in use. You can use lotus.domino.Document and lotus.domino.Database to explicitly use non-ODA objects. — Paul Stephen Withers
Yes it is a memory exception. But the funny thing is I get it when creating in-memory objects from oracle but not from domino. — Mikael Andersson Wigander
If this is a pure java Agent then its not XPages and you should remove the XPages tag. If it's an "XAgent" then obviously you should keep the tag. I'm not exactly sure which it is so you should make that clear. Thanks — David Leedy
@PaulStephenWithers I begin to lean towards your answer where all Domino objects not being recycled. I managed to loop and add to Maps all resources this morning without any MemoryExceptions and I think if I run the agent more than once today, memory will be full… the only solution is to loop all SQL objects and all Domino objects but that will create enormous I/O… — Mikael Andersson Wigander

stwissel stwissel · Accepted Answer · 2014-11-08T08:41:20

It is hardly a good idea to keep all Domino documents in memory. At some point in time you run out of the C handles in the core API.

You might want/need to change your approach. Trying to keep everything in memory will always run out of memory once you reach "the size". What you need is, aptly named by @Andre Guirard, "The Pirate algorithm":

Have a view in Domino and Oracle each, that are sorted by your comparison key (The UNID as I understand it from your code). Lets presume N is Notes and O is Oracle. Your code (here pseudo code) would roughly look like this:

         OEntry = O.getFirst
         for all NEntry in N
             if NEntry.key > OEntry.key
                InsertFromOracle until NEntry.key = OEntry.key
             else if Nentry.key < OEntry.key
                DeleteFromNotes until NEntry.key = OEntry.key
             else
                SyncValuesFromOracleToNotes
             end if
          end for
         if O.hasMoreEntries
             InsertFromOracle until not O.hasMoreEntries
         end if

The InsertFromOracle of course need to check if there are elements left in Oracle to do that.

A LotusScript example had been written by Andre (Look for the Pirates Agent, named after the swagger walking of a pirate)

You don't need to keep entire tables or document collections in memory. Since loading and comparing run in a single loop it should also perform quite well.

Let us know how it goes!

Difference between NotesDocument objects from database and in-memory?

2 Answers