2
votes

In our application, we receive batches of files that need to be preprocessed and loaded into MarkLogic.

To do this, we need to:

  1. Load the files in a temporary MarkLogic working directory
  2. Preprocess (node operations on the XML file)
  3. Move documents to their intended destination MarkLogic directory using xdmp:document-insert

While doing (3), we get expanded tree cache error, for a batch of 1500 docs (upto 400 docs, it works ok. Any higher number, the error pops up).

The algorithm steps for our code:

    Get total number of docs in working directory = totalRec
    for Ctr = 1 to totalRec
        Get specific node values for current doc
        Frame the target URI where doc is to be loaded
        Insert document using xdmp:document-insert

We even tried using transaction begin/commits within the for loop but nothing seems to work. Any thoughts on how to fix this issue?

1
Have you tried xdmp:eval() while instering document xdmp:document-insert()? - Navin Rawat
Yes Navin. That didnt work too. - Gatt
Let me know your MarkLogic version. If I am right: totalRec=1500, 1 to totalRec=1 to 1500. How are you calculating number of ducments? and "Get specific node values for current doc"? - Navin Rawat
@Gatt It would be more useful if could post some actual code; pseudo-code is only limited use when trying to figure out what's going on. - Will Goring

1 Answers

3
votes

An expanded tree cache error simply means that you're trying to work with too many fragments at once. The preferred solution is to reduce the size of the working set. Usually less good, if you have enough spare memory you can increase the size of the expanded tree cache in the group settings. But it is usually better to reduce the size of the working set.

This particular use-case sounds like a content processing workflow. So you might be better off using a built-in product feature, the Content Processing Framework (CPF): http://docs.marklogic.com/guide/cpf has more about CPF.

Or an InfoStudio flow might be appropriate: http://docs.marklogic.com/guide/infostudio

Using an existing tool means that you don't have to reinvent the wheel.