MongoDB Limits : Slow to insert records after 250 million

0
votes

So I have installed the latest MongoDb on to a Windows 2016 server, it has been going like grease lighting up until I hit around 250 million records (2 fields, URL, body text, 1 index on the URL), now inserting takes a lot longer (from seconds to minutes).

Fast disks (SAN), plenty of ram (it's using 64gb out of 128gb), is there some sort of limit?

My aim is to insert 2.8 billion records.

Mongo 4.2 Windows Server 2016 DC

enter image description here enter image description here The data is simply ID/URL/Body index on URL

CFG (default)

mongod.conf

for documentation of all options, see:

http://docs.mongodb.org/manual/reference/configuration-options/

Where and how to store data. storage: dbPath: Z:\data journal:

enabled: true

engine:

mmapv1:

wiredTiger:

where to write logging data. systemLog: destination: file logAppend: true path: C:\Program

Files\MongoDB\Server\4.2\log\mongod.log

network interfaces net: port: 27017 bindIp: 192.168.0.150,127.0.0.1

processManagement:

security:

operationProfiling:

replication:

sharding:

Enterprise-Only Options:

auditLog:

snmp:

1
mongodbwindows-server-2016
More information needed. Look at the system and identify whether it is CPU, memory or disk bound to start. – D. SM
None of them seem to be even busy – Burf2000
You are saying your system is idle for more than 1 minute when you are doing an insert? – D. SM
Yes 128gb DDR, 64 Cores not even going above 5%, NAS writing 5/6meg. I think its hit an Index limit or somethign – Burf2000
The available filesystems su... re are ill suited for larger installations of MongoDB. From my experience, you get the best results with XFS tweaked according to MongoDB's production notes. Aside from that, URL's by definition are unique, so you have a spurious index on URL, because you could use the URL as _id: {_id:"https://example.com/foo/bar?baz=1", body:"Hello, 1"}. One index saved, should save RAM somewhere in the area of 3GB (250m * 12b, assuming you use an ObjectID as _id). – Markus W Mahlberg

1 Answers

0
votes

is there some sort of limit?

MongoDB limits are described here.

MongoDB does not limit its own performance.