MarkLogic : Switching between databases with xdmp.eval

Question

MarkLogic version : 9.0-6.2

We have a custom REST API that reads document from STAGING, then updates a few documents in FINAL, then runs an xdmp.documentRemoveCollections on the STAGING document.

Step 1: Start from STAGING DB. Read the document

Step 2: Switch to FINAL DB, apply the changes to multiple documents in FINAL DB

Step 3: Switch to STAGING DB, apply xdmp.documentRemoveCollections on the document read in step 1

We are using xdmp.eval to switch between databases, but noticed that the service is timing out, probably because of switching between the databases. (for example, if we remove xdmp.documentRemoveCollections step, then the service does not timeout, probably because it does not have to switch from FINAL to STAGING)

We tried using harmonization flow, but the behavior was not consistent, probably because of multiple document updates in FINAL.

Please suggest if there are any precautions to be followed in CUSTOM REST API to avoid time-out, while switching between databases back-and-forth.

Thanks in advance!

Answer 1

Timeout in this case is likely due to read and writes to the same document in the same transaction. Step 3 shouldnt have to 'Switch' DBs it should already be in the same DB, the only switch is step 2. This workflow could easily deadlock without special attention. Is step 2 required to be synchronous ? if not suggest queuing it to the task server. Are 1,3 required to be in the same transaction ? Step 1 is likely holding a lock on the documents it read -- then step 3 trying to wait for that lock to free in order to update. Try forcing step 1 to be a read transaction and validate that it stays that way until step 3. You can fully isolate them be doing ALL steps in sub-transactions (individually). Recommend using invoke-function (or module) over eval (with strings) -- its easy to get 'xquery injection' behaviour (from friend or foe) using eval("some string I built by concatenating user input")

Are you doing this concurrently with other activity (same REST call or different calls at the same time to the same DB).

Make use of 'query console' to look for what transactions are open at what time. If you see a 'hang' then almost certainly you will find a open transaction lurking.