expanded tree cache full Error

Question

for the following code I am getting expanded tree cache full:

xquery version "1.0-ml";
let $result :=
  <root>{
    cts:search(
      fn:collection("metadata-search")/metadata,
      cts:and-query((   
          cts:element-value-query(xs:QName('AssetID'), "*")
      ))
    )/assetIdentifiers/assetIdentifier/AssetID


  }</root>
let $path := "D:\Output\input.xml"
return 
  xdmp:save($path, $result,
            <options xmlns="xdmp:save">
              <method>xml</method>
            </options>
  )

but if same code I ran in iteration mode it was working fine:

xquery version "1.0-ml";
let $result :=
  <root>{
    for $data in cts:search(
      fn:collection("metadata-search")/metadata,
      cts:and-query((   
         cts:element-value-query(xs:QName('AssetID'), "*")
      ))
    )
    return $data/assetIdentifiers/assetIdentifier/AssetID
  }</root>
let $path := "D:\Output\input.xml"
return 
  xdmp:save($path, $result,
            <options xmlns="xdmp:save">
              <method>xml</method>
            </options>
  )

there are 450000 assets in database and it took 10 min to run, I was wondering why other code construct works fine and not first one? Any Idea, please let me know in detail with the context of this code ?

grtjn grtjn · Accepted Answer · 2018-06-26T13:45:49

The first approach applies an XPath expression on the full result set. XPath enforces document ordered results, forcing MarkLogic to do sorting across the entire result set. The second does it for each document, which runs much faster.

Note though that it still won't scale well. You are likely better off batching search results, saving batches individually, and aggregating into a total xml outside of MarkLogic if that is really required. Tools like Corb2, and libraries like DMSDK can be of a lot of use here, doing the heavy lifting for you.

HTH!

expanded tree cache full Error

1 Answers