mongodb and pymongo 16Mb limit on document size

Question

I'm analyzing the includes in a build using the '/showIncludes', on Windows, and '-H', on *nix, flags.

I'm parsing this info with a python script. Each file included is turned into an object that lists its children (files it includes) and ancestors (the include paths which included this file).

After these objects are created I want to push them into a mongodb database using pymongo.

This works fine for 99% of the includes. But ~5 are very large. When I try to add them to mongodb it complains.

Traceback (most recent call last):
  File "mongodb.py", line 94, in <module>
    includes_collection.update({'id': include.include_id}, { 'ancestor_tree': ancestor_tree_ids } )
  File "C:\Python27\lib\site-packages\pymongo-2.7.2-py2.7-win-amd64.egg\pymongo\collection.py", line 551, in update
    docs, check_keys, self.uuid_subtype, client)
DocumentTooLarge: command document too large

Reading up on mongo this seems to be a design choice. By default documents cannot exceed 16Mb. But that can be overridden with the --nssize command line option. See

http://docs.mongodb.org/manual/reference/program/mongod/#bin.mongod

So I re-ran mongod with --nssize 32/64/128. I don't think any of my include objects are above 128Mbs. But the issue persisted.

So I'm now wondering if pymongo is to blame. Does it respect this server setting?

My version of mongod is

db version v2.6.3 2014-08-28T16:56:51.534+0100 git version: 255f67a66f9603c59380b2a389e386910bbb52cb

I'm using pymongo-2.7.2-py2.7-win-amd64.

Is there anyway to work around this limitation?

I'm surprised I'm hitting it. But some of my Includes have a cost of ~2.5 Million. Where the cost is num of times included X num of includes it brings in. In addition an Include records each path to its own inclusion. e.g. A includes B which includes C which includes D. If D is included a million times I guess those paths can add up. — Shane Gannon

John Petrone John Petrone · Accepted Answer · 2014-08-28T16:40:56

Unfortunately you cannot exceed the BSON limit of 16MB per document. The mongod option you are using has nothing do to with document size. It's specifying the default size of a mongodb namespace file - not related at all to maximum documnent size. As suggested by the documentation, if you really need to store objects larger than 16MB I'd take a look at the GridFS API.

From the documentation:

BSON Documents

BSON Document Size

The maximum BSON document size is 16 megabytes.

The maximum document size helps ensure that a single document cannot use excessive amount of RAM or, during transmission, excessive amount of bandwidth. To store documents larger than the maximum size, MongoDB provides the GridFS API. See mongofiles and the documentation for your driver for more information about GridFS.

Size of Namespace File

Namespace files can be no larger than 2047 megabytes.

By default namespace files are 16 megabytes. You can configure the size using the nsSize option.

--nssize

Default: 16

Specifies the default size for namespace files, which are files that end in .ns. Each collection and index counts as a namespace.

Use this setting to control size for newly created namespace files. This option has no impact on existing files. The maximum size for a namespace file is 2047 megabytes. The default value of 16 megabytes provides for approximately 24,000 namespaces.

http://docs.mongodb.org/manual/reference/limits/

http://docs.mongodb.org/manual/reference/program/mongod/#bin.mongod

mongodb and pymongo 16Mb limit on document size

1 Answers