Xquery optimization

Question

I have this xquery as follows:

declare variable $i := doc()/some-element/modifier[empty(modifier-value)];
$i[1]/../..;

I need to run this query on Marklogic's Qconsole where we have 721170811 records. Since that is huge number of record, I am getting timeout error. Is there any way I can optimize this query to get the result?

P.S. I cannot request amdin to increase the timeout time.

Answer 1

Try creating an element range index (or a path range index if the target element is not unique) and using a cts:values() lexicon lookup.

That way, the request can read the values from the range index instead of having to read each document.

See:

http://docs.marklogic.com/guide/search-dev/lexicon

Answer 2

You could use xdmp:spawn, create a library when you will make the query, get the documents, iterate the result collecting 1000 documents per iteration and call another xdmp:spawn to process the information from that dataset, I would suggest summarize the result to return only the information you will need to don't crash the browser, at the end should look something like this:

xdmp:spawn("process.xqy")

into the library process.xqy

function local:start-process(){
   let $docs := (....)
   let $temp := for $x in $docs[$start to $end]
                return local:process-dataset($temp) (: Could use spawn here too if you want :) 
   return xdmp:spawn("collect.xqy",$temp)
}

local:start-process()

compact-data function should create a file or a set of files with your data, this way the server will run all the process and in some minutes you will be available to see your data without problems.

Answer 3

You don't want to run something like doc() or xdmp:directory - just returns a result set that will kill you every time. You need to lower your result set by a lot.

A few thoughts:

1. You want to have as much done in MarkLogic's d-node, and the least work done in the e-node as possible. This is a way over-generalization, but for the most part I look at it like d-node stuff is data, indexes, lexicon work, etc. e-node stuff handles xQuery and such. So, in your example, you're definitely working out the e-node more than you need to.

2. You're going to want to use cts:search, as it uses indexes, not xPath to resolve your query. So, something like this:

   declare variable $i := cts:search(fn:collection(), 
     cts:element-query(xs:QName("some-element"), 
       cts:element-value-query(xs:QName("modifier"), "", "exact")
     )
   )[1];
   

This will return document-node's, which it looks like what you were wanting with the $i[1]/../... This searches the xPath some-element for a modifier that is empty.

Answer 4

Please create element range index and attribute range index and use cts:search if you are familiar with marklogic it will be easy for you to write the query.