Bulk insert performance in MongoDB for large collections

18
votes

I'm using the BulkWriteOperation (java driver) to store data in large chunks. At first it seems to be working fine, but when the collection grows in size, the inserts can take quite a lot of time.

Currently for a collection of 20M documents, bulk insert of 1000 documents could take about 10 seconds.

Is there a way to make inserts independent of collection size? I don't have any updates or upserts, it's always new data I'm inserting.

Judging from the log, there doesn't seem to be any issue with locks. Each document has a time field which is indexed, but it's linearly growing so I don't see any need for mongo to take the time to reorganize the indexes.

I'd love to hear some ideas for improving the performance

Thanks

3
performancemongodbinsertmongodb-java
Have you given a thought about sharding? Performance depends on lot of parameters like document size, initial data, hw, cluster setup etc. Also check if mongoimport can be used. while doing insert mongo validates the json object, if your document is large then that validation will also take time and can hamper performance in such cases disabling that validation can also help but boost will be minor if documents are small in size. - Nachiket Kate
What was the performance when the collection was 2M docs in size? And what indexes are set up on the collection, what is the average new doc size, what is the physical media, and what is the RAM of the primary? My gut says a smaller-scale infrastructure now has to deal with a bigger workload.... - Buzz Moschetti
Have you considered doing your bulk writes in parallel? - David Soroko
Could you try dropping the index, and seeing if it makes any difference to the performance. I suspect it won't, given what you've already said about it, but it would be a useful way of ruling that out as the culprit. - Vince Bowdren
Does your document size change a lot? Perhaps showing us a (few) sample document(s) would help to determine if that can be a problem. - p.streef

3 Answers

5
votes

You believe that the indexing does not require any document reorganisation and the way you described the index suggests that a right handed index is ok. So, indexing seems to be ruled out as an issue. You could of course - as suggested above - definitively rule this out by dropping the index and re running your bulk writes.

Aside from indexing, I would …

  • Consider whether your disk can keep up with the volume of data you are persisting. More details on this in the Mongo docs
  • Use profiling to understand what’s happening with your writes
4
votes
  1. Do have any index in your collection? If yes, it has to take time to build index tree.
  2. is data time-series? if yes, use updates more than inserts. Please read this blog. The blog suggests in-place updates more efficient than inserts (https://www.mongodb.com/blog/post/schema-design-for-time-series-data-in-mongodb)
  3. do you have a capability to setup sharded collections? if yes, it would reduce time (tested it in 3 sharded servers with 15million ip geo entry records)
1
votes

  • Disk utilization & CPU: Check the disk utilization and CPU and see if any of these are maxing out. Apparently, it should be the disk which is causing this issue for you.

  • Mongo log: Also, if a 1000 bulk query is taking 10sec, then check for mongo log if there are any few inserts in the 1000 bulk that are taking time. If there are any such queries, then you can narrow down your analysis

Another thing that's not clear is the order of queries that happen on your Mongo instance. Is inserts the only operation that happens or there are other find queries that run too? If yes, then you should look at scaling up whatever resource is maxing out.