Marklogic performance with JAVA API on Rawcombined Query

1
votes

I have installed Marklogic 8 on my 8GB RAM machine and have 10 Million documents inserted to it. I am using Java API to query the database and get first 1000 matched documents.My Query(Rawcombined Query) involves 2 PathRange indexed JSON attributes and one Word query. When i logged the time taken in my java program, search function takes 20seconds to get the 1000 documents.

Can anyone please let me know what could be the factors that drive these type of performance hit?

2
marklogic
Thanks @grtjn for the suggestion. - user3584146
Could you please share your query, code, and some representative sample documents? It could make a big difference whether you are using the bulk API (DocumentManager.search) or search API (QueryManager.search). Also whether you are using summaries, extracting from documents, or getting back entire documents. Also, could you elaborate on why you want to get back 1000 documents instead of a normal page size (10-50)? - Sam Mefford
I am using search API . qm.search(querydef, resultsHandle); MatchDocumentSummary[] docSummaries = resultsHandle.getMatchResults(); System.out.println("Listing "+docSummaries.length+" documents:\n"); - user3584146
I tried pagination to get 50 documents, but even now the response time is 17seconds. I will restart marklogic server every time i take the stats for the run(to make sure the query is run for the first time in the system). Does the marklogic restart causing this big response time? - user3584146
Thanks for the code snippet. Now I know you're not using bulk. And thanks for trying a page size of 50, now we know it's not likely the doc download slowing things down, unless your docs are huge. Could you share your query, search options, and some representative sample documents? One of the first things to check is whether your query is running unfiltered. With that response time, it may be doing filtering which can really slow things down. - Sam Mefford

2 Answers

4
votes

I would recommend getting smaller pages, and allow for pagination through results. Searching is fast, pulling up the actual documents relatively slow. Pull up around 10 to 50 docs per page max for good performance.

HTH!

1
votes

Make sure your query runs unfiltered, meaning all query clauses can be resolved against indexes. That may require creating some range indexes. Given the pseudo-query you provided, I see nothing that would require a filtering step. If your query is running very slowly you either have a problem with the hardware sizing, OS config, or query. The first thing to check on a slow-running query is to make sure it runs unfiltered.

See Fast Pagination and Unfiltered Searches in the docs, and specific to the Java API, Query Options.