How to wait for a MarkLogic spawn function to complete

Question

I have a requirement where I iterate through 10,000,000 documents and for each document I do some operation and store some values in '/count.xml'. When I iterate to second document I update '/count.xml' with updated value

Currently this is what I am doing, here $total-records is 10,000,000

let $total-records := xdmp:estimate(cts:search( //some code))
let $batch-size := 5000
let $pagination := 0
let $bs := 
   for $records in 1 to fn:ceiling($total-records  div $batch-size )
   let $start := fn:sum($pagination + 1)
   let $end := fn:sum($batch-size + $pagination)
   let $_ := xdmp:set($pagination, $end)
   return
    xdmp:spawn-function
    (
    function() {
     for $each in cts:search( //some code)[$start to $end]
     return //some operation and update '/count.xml' with some updated values
    },
    <options xmlns="xdmp:eval"><commit>auto</commit><update>true</update</options>
    )
let $doc := doc("/count.xml")
return ()

So here the issue is I need to read the '/count.xml' file after all documents are iterated, But with above code using spawn task

let $doc := doc("/count.xml")

will not be latest one as above spawn task will run on different threads.

I need a solution where

let $doc := doc("/count.xml")

waits till all spawn task are completed.

I have came across

<result>{fn:true()}</result>

option as well, but I do not know whether it will work or not because variable

$bs

not being used anywhere and documentation says 'When the calling request uses the value future in any operation, it will automatically wait for the spawned task to complete and it will use the result.'

Is there any other alternative where

let $doc := doc("/count.xml")

line will be executed only after all spawn task are completed

Answer 1

To process 10 mln documents, you probably need to spawn something like 10.000 batches of a 1000 docs. I don't think that will work well from within MarkLogic.

I'd advice looking into the built-in aggregation features of MarkLogic. See for instance cts:sum-aggregate. You might be able to pre-calculate per-document intermediate results, that you could aggregate at run-time using those aggregation features. That would definitely be most performant, and would scale best.

Alternative would be to orchestrate your calculations from outside of MarkLogic. Otherwise you end up either flooding the task queue, or running into timeout limits, or both. Tools like Corb2 and DMSDK could be of help with this.

Note: you can indeed make spawns wait for result by using the <result> option, but either use <result>true</result> or <result>{fn:true()}</result> (note the parentheses behind fn:true, it is a function).

HTH!

Answer 2

The requirement as given, one cannot tell the difference between writing once the final result of a query across 10,mil docs vs writing the result after query of 1 document at a time. Since your example does no writes to the queried documents it need not be spawned nor run in a seperate thread or transaction, rather as HTH says, you can aka use of aggregate functions to do a single query over the entire set, compute the final result and store it in 1 operation. Likely this will run very quickly (or can be made to).

If the requirements are actually that each single document MUST be queried, then sequentially another shared document written to -- this can only be observed by using seperate transactions, serially. Its going to be horrendously slow, almost certainly longer then the timeout for the calling transaction. This means you must orchestrate it from outside -- if the requirement is that the same caller start the process as finish it (a highly implementation specific requirement that if true is likely to have other implications beyond those given).

Something close thats achievable but still horrendously slow is to have an outside query poll on the updated shared document and return 'success' once the job is done. But again, with this many documents, if your forcing a write transaction for each one, its going to take longer (or atleast is not easily guaranteed to NOT take longer) then the a single transaction timeout so must be invoked from 'outside'.

This is where I would recommend revisiting the requirements to determine the core functionality/result that is desired and if it is truly required to implement exactly as stated vs a more performant implementation that achieves the desired result. If the core functionality needed is that every single query be 'checkpointed' with a document update, then there are other implications such as transaction rollback that need to be considered.