Problems with DictField in MongoEngine DynamicDocument

Question

My problem matches this post on Bountysource:

https://www.bountysource.com/issues/32756263-attribute-error-when-building-nested-index-for-dictfield-in-dynamicdocument

My work place has used PyMongo 2.6.3/MongoEngine 0.8.7 for a while, and we are considering upgrading to PyMongo 3.2.2/MongoEngine 0.10.6. I downloaded those versions and started testing them. I began running into issues with a class that was based on DynamicDocuments that used DictFields.

14:20:20  ERROR   nas.data.sync                            Traceback (most recent call last):
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/nas/data/sync.py", line 1484, in main
14:20:20  ERROR   nas.data.sync                                result = funct(external, syncRecord)
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/nas/data/sync.py", line 456, in syncAsset
14:20:20  ERROR   nas.data.sync                                success = scheduleUpdate(asset, v) and success
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/nas/data/sync.py", line 299, in scheduleUpdate
14:20:20  ERROR   nas.data.sync                                for wu in workunits:
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/auxiliary/__all/MongoEngine-0.10.6/mongoengine/queryset/queryset.py", line 80, in _iter_results
14:20:20  ERROR   nas.data.sync                                self._populate_cache()
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/auxiliary/__all/MongoEngine-0.10.6/mongoengine/queryset/queryset.py", line 92, in _populate_cache
14:20:20  ERROR   nas.data.sync                                self._result_cache.append(self.next())
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/auxiliary/__all/MongoEngine-0.10.6/mongoengine/queryset/base.py", line 1407, in next
14:20:20  ERROR   nas.data.sync                                raw_doc = self._cursor.next()
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/auxiliary/__all/MongoEngine-0.10.6/mongoengine/queryset/base.py", line 1481, in _cursor
14:20:20  ERROR   nas.data.sync                                self._cursor_obj = self._collection.find(self._query,
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/auxiliary/__all/MongoEngine-0.10.6/mongoengine/queryset/base.py", line 1515, in _query
14:20:20  ERROR   nas.data.sync                                self._mongo_query = self._query_obj.to_query(self._document)
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/auxiliary/__all/MongoEngine-0.10.6/mongoengine/queryset/visitor.py", line 90, in to_query
14:20:20  ERROR   nas.data.sync                                query = query.accept(QueryCompilerVisitor(document))
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/auxiliary/__all/MongoEngine-0.10.6/mongoengine/queryset/visitor.py", line 155, in accept
14:20:20  ERROR   nas.data.sync                                return visitor.visit_query(self)
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/auxiliary/__all/MongoEngine-0.10.6/mongoengine/queryset/visitor.py", line 78, in visit_query
14:20:20  ERROR   nas.data.sync                                return transform.query(self.document, **query.query)
14:20:20  ERROR   nas.data.sync                              File "E:/alpha_maya_pyside/auxiliary/__all/MongoEngine-0.10.6/mongoengine/queryset/transform.py", line 61, in query
14:20:20  ERROR   nas.data.sync                                raise InvalidQueryError(e)
14:20:20  ERROR   nas.data.sync                            InvalidQueryError: 'DictField' object has no attribute 'document_type'
14:20:20  ERROR   nas.data.sync                            sync failed

Having the class inherit from Document instead of DynamicDocument did fix the issue. I have been digging into the MongoEngine source code, but nothing really jumps out at me. My only guess is that MongoEngine might be having trouble converting a DynamicField into a DictField.

I added a print statement MongoEngine-0.10.6/mongoengine/queryset/transform.py:

try:
    fields = _doc_cls._lookup_field(parts)
    print fields
except Exception, e:
    raise InvalidQueryError(e)

I got this output just before the error message:

[<mongoengine.fields.DynamicField object at 0x00000000048B48D0>]

Interesting. If you look inside mongoengine/fields.py, you will see that the following fields do have document_type attribute: EmbeddedDocumentField, EmbeddedDocumentListField, CachedReferenceField. — Christopher Spears
The function _lookup_field is actually part of the BaseDocument (mongoengine/base/document.py). It takes your schema (which is a class) and whatever attributes you used to do the query and returns a list containing the field and its parents (according to the comments). I think something is breaking down in that function. — Christopher Spears

Christopher Spears Christopher Spears · Accepted Answer · 2016-04-28T01:25:18

I went into mongoengine/base/document.py and checked out the __lookup_field function. Inside that function is a rather long if/else statement. Here is an excerpt.

            if field is None:
                # Look up first field from the document
                if field_name == 'pk':
                    # Deal with "primary key" alias
                    field_name = cls._meta['id_field']
                if field_name in cls._fields:
                    field = cls._fields[field_name]
                elif cls._dynamic:
                    field = DynamicField(db_field=field_name)
                elif cls._meta.get("allow_inheritance", False) or cls._meta.get("abstract", False):
                    # 744: in case the field is defined in a subclass
                    for subcls in cls.__subclasses__():
                        try:
                            field = subcls._lookup_field([field_name])[0]
                        except LookUpError:
                            continue

                        if field is not None:
                            break
                    else:
                        raise LookUpError('Cannot resolve field "%s"' % field_name)
                else:
                    raise LookUpError('Cannot resolve field "%s"'
                                      % field_name)
            else:
                ReferenceField = _import_class('ReferenceField')
                GenericReferenceField = _import_class('GenericReferenceField')
                if isinstance(field, (ReferenceField, GenericReferenceField)):
                    raise LookUpError('Cannot perform join in mongoDB: %s' %
                                      '__'.join(parts))
                if hasattr(getattr(field, 'field', None), 'lookup_member'):
                    new_field = field.field.lookup_member(field_name)
                elif cls._dynamic and (isinstance(field, DynamicField) or
                                       getattr(getattr(field, 'document_type'), '_dynamic')):
                    new_field = DynamicField(db_field=field_name)

If you run a query and use, say, a StringField, then the variable field remains None. However, if you run the query with a DictField, then field gets set to something like <mongoengine.fields.DictField object at 0x0000000004506CF8>. I have no idea how field is getting set to that value. Anyway, if the class you are querying happens to be a DynamicDocument, then you are sent to this part of the loop.

elif cls._dynamic and (isinstance(field, DynamicField) or
                                       getattr(getattr(field, 'document_type'), '_dynamic')):
                    new_field = DynamicField(db_field=field_name)

I arrived to this conclusion through a process of elimination. Basically, I figured out where I was NOT going. As you can see, since a DictField is not a DynamicField, it will be queried for the document_type attribute. The DictField does not have that attribute resulting in an InvalidQueryError.

Looks like the solution for now is to not use DictFields with DynamicDocuments or see if you can use the EmbeddedDocumentField, EmbeddedDocumentListField, or ReferenceField with your DynamicDocument. Maybe this is the developers' intention?

Problems with DictField in MongoEngine DynamicDocument

1 Answers