How can I reduce Qtime?

0
votes

I would like to know which parameters I need to change (if exist) to decrease the Qtime. 

 Qtime : The elapsed time (in milliseconds) between the arrival of the request (when the SolrQueryRequest object is created) and the completion of the request handler. It does not include time spent in the response writer formatting/streaming the response to the client.

I have logged curl responses when I update my documents and I have noticed that QTime increase during the time.

For example, my first response (corresponding to my first doc indexed) is 6293 ms. And after a large quantity of indexed documents, my QTime becomes longer : 1560781 ms so around 26 min !

Edit

First measure : 1 CSV file with 10000 solr docs -> QTime :6293ms

Second measure : 1 CSV file with 10000 solr docs -> Qtime : 1560781 ms

Delay between those measures = 32min 19s

Number documents indexed during this interval : 26720000 documents

I guess that is possible to improve that but I don't know witch settings modify to have better performances.

Informations related to my system

  • I got 1 solr instances with one core.
  • My system works on a Virtual Machine with 8 CPUs and 16GB RAM
  • I use around 30 % of RAM
  • My JVM is : 1.7.0_09-icedtea OpenJDK Runtime Environment (rhel-2.3.8.0.el6_4-x86_64) OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

Questions

  • Maybe I need to set more threads to update documents or something like that.
  • Could this behaviour be related with Jetty ? (I don't think there is a link with Jetty, may someone could confirm it ?)
  • How can I use more RAM to index data ? (I have allready set the JVM to allocate enough RAM for Solr with this command : java -Xms2048M -Xmx8192M -jar start.jar)
  • Should I use more solr instances (SolrCloud ?) to solve it ?
  • Why Qtime increase with the update load ? Is it Solr limitations (RAM, Disk)?

Any tips to help me to do the completion of my updates faster, would be appreciated.

Thanks.

2
solrlucene
How many documents do you have in index? What is index size? 6000ms to start with is already extremly slow. - Okke Klein
Hi Okke, yes I agree with you, it's very slow. I added details you asked in my questions. This behaviour is strange, it seems that there is a bottle neck at indexing process level because latency (or Qtime) increase with the laod. The worry is that I use only 30% of RAM. - Corentin
When do you commit? What DirectoryFactory are you using. When do you GC? A lot of factors play a role. The proposed link is a good start. Else a distributed or Solrcloud solution is always an option to consider. - Okke Klein

2 Answers

1
votes

Have a look into following link. This may be helpful for you..

http://wiki.apache.org/solr/SolrPerformanceFactors

0
votes

Firstly improve SQl Queries used in Delta Imports that will improve a lots especially data types and then use solr REST services for instant updates in solr.