Endpoints Proto Datastore - Path {requirement} variable used for keys randomly breaks

Question

I'm building an API using Endpoints and Endpoints Proto Datastore and testing with the App Engine Launcher. The code below is what I'm using to build nested entities two layers deep (Grandparent/Parent/Child).

The variable _child returns 'None' (set on line 98) seemingly at random, this breaks ndb.Key on lines 139 and 143 and causes BadArgumentError: Incomplete Key entry must be last.


import endpoints
from google.appengine.ext import ndb
from protorpc import remote

from endpoints_proto_datastore.ndb import EndpointsAliasProperty
from endpoints_proto_datastore.ndb import EndpointsModel

import logging


project_api = endpoints.api(name='project', version='v1', description='Project API')


""" Grandparent """


class GrandparentModel(EndpointsModel):
    _message_fields_schema = ('grandparent',)

    def set_grandparent(self, value):
        if not isinstance(value, basestring):
            raise TypeError('Grandparent name must be a string.')
        self.UpdateFromKey(ndb.Key(GrandparentModel, value))

    @EndpointsAliasProperty(setter=set_grandparent, required=True)
    def grandparent(self):
        if self.key is not None:
            return self.key.string_id()


""" Parent """


class ParentModel(EndpointsModel):

    _message_fields_schema = ('grandparent', 'parent',)

    _grandparent = None
    _parent = None

    def set_key(self):
        if self._grandparent is not None and self._parent is not None:
            key = ndb.Key(GrandparentModel, self._grandparent, ParentModel, self._parent)
            self.UpdateFromKey(key)

    def set_parts(self):
        if self.key is not None:
            grandparent_pair, parent_pair = self.key.pairs()

            self._grandparent = grandparent_pair[1]
            self._parent = parent_pair[1]

    # Grandparent

    def set_grandparent(self, value):
        if not isinstance(value, basestring):
            raise TypeError('Grandparent name must be a string.')

        self._grandparent = value

        if ndb.Key(GrandparentModel, value).get() is None:
            raise endpoints.NotFoundException('Grandparent %s does not exist.' % value)

        self.set_key()
        self._endpoints_query_info.ancestor = ndb.Key(GrandparentModel, value)

    @EndpointsAliasProperty(setter=set_grandparent, required=True)
    def grandparent(self):
        if self._grandparent is None:
            self.set_parts()
        return self._grandparent

    # Parent

    def set_parent(self, value):
        if not isinstance(value, basestring):
            raise TypeError('Parent must be a string.')

        self._parent = value
        self.set_key()

    @EndpointsAliasProperty(setter=set_parent, required=True)
    def parent(self):
        if self._parent is None:
            self.set_parts()
        return self._parent


""" Child """


class ChildModel(EndpointsModel):

    _message_fields_schema = ('grandparent', 'parent', 'child',)

    _grandparent = None
    _parent = None
    _child = None

    def set_key(self):
        if self._grandparent is not None and self._parent is not None and self._child is not None:
            key = ndb.Key(GrandparentModel, self._grandparent, ParentModel, self._parent, ChildModel, self._child)
            self.UpdateFromKey(key)

    def set_parts(self):
        if self.key is not None:
            grandparent_pair, parent_pair, child_pair = self.key.pairs()

            self._grandparent = grandparent_pair[1]
            self._parent = parent_pair[1]
            self._child = child_pair[1]

    # Grandparent

    def set_grandparent(self, value):
        if not isinstance(value, basestring):
            raise TypeError('Grandparent must be a string.')

        self._grandparent = value
        self.set_key()

    @EndpointsAliasProperty(setter=set_grandparent, required=True)
    def grandparent(self):
        if self._grandparent is None:
            self.set_parts()
        return self._grandparent

    # Parent

    def set_parent(self, value):
        if not isinstance(value, basestring):
            raise TypeError('Parent name must be a string.')

        self._parent = value

        logging.warning('Check _grandparent value: %s' % self._grandparent)
        logging.warning('Check _parent value: %s' % self._parent)

        if ndb.Key(GrandparentModel, self._grandparent, ParentModel, value).get() is None:
            raise endpoints.NotFoundException('Key %s does not exist.' % value)

        self.set_key()
        self._endpoints_query_info.ancestor = ndb.Key(GrandparentModel, self._grandparent, ParentModel, value)

    @EndpointsAliasProperty(setter=set_parent, required=True)
    def parent(self):
        if self._parent is None:
            self.set_parts()
        return self._parent

    # Child

    def set_child(self, value):
        if not isinstance(value, basestring):
            raise TypeError('Child must be a string.')

        self._child = value
        self.set_key()

    @EndpointsAliasProperty(setter=set_child, required=True)
    def child(self):
        if self._child is None:
            self.set_parts()
        return self._child


@project_api.api_class(resource_name='grandparent')
class Grandparent(remote.Service):

    @GrandparentModel.method(name='insert', path='grandparent')
    def grandparent_insert(self, grandparent):
        grandparent.put()
        return grandparent

    @GrandparentModel.query_method(name='list', path='grandparent')
    def grandparent_list(self, query):
        return query


@project_api.api_class(resource_name='parent', path='grandparent')
class Parent(remote.Service):

    @ParentModel.method(name='insert', path='{grandparent}/parent')
    def parent_insert(self, parent):
        parent.put()
        return parent

    @ParentModel.query_method(name='list', path='{grandparent}/parent', query_fields=('grandparent',))
    def parent_list(self, query):
        return query


@project_api.api_class(resource_name='child', path='grandparent')
class Child(remote.Service):

    @ChildModel.method(name='insert', path='{grandparent}/parent/{parent}/child')
    def child_insert(self, child):
        child.put()
        return child

    @ChildModel.query_method(name='list', path='{grandparent}/parent/{parent}/child',
                                  query_fields=('grandparent', 'parent',))
    def child_list(self, query):
        return query


application = endpoints.api_server([project_api], restricted=False)

Here's the log from when it breaks:

INFO     2015-08-16 10:26:50,503 module.py:809] default: "POST /_ah/spi/BackendService.getApiConfigs HTTP/1.1" 200 8577
INFO     2015-08-16 09:26:52,099 main.py:136] Check _grandparent value: g

INFO     2015-08-16 09:26:52,099 main.py:137] Check _parent value: p

INFO     2015-08-16 10:26:52,108 module.py:809] default: "POST /_ah/spi/Child.child_list HTTP/1.1" 200 2
INFO     2015-08-16 10:26:52,109 module.py:809] default: "GET /_ah/api/project/v1/grandparent/g/parent/p/child HTTP/1.1" 200 2
INFO     2015-08-16 10:26:59,165 module.py:809] default: "POST /_ah/spi/BackendService.getApiConfigs HTTP/1.1" 200 8577
INFO     2015-08-16 09:26:59,168 main.py:136] Check _grandparent value: None

INFO     2015-08-16 09:26:59,168 main.py:137] Check _parent value: p

ERROR    2015-08-16 09:26:59,168 service.py:191] Encountered unexpected error from ProtoRPC method implementation: BadArgumentError (Incomplete Key entry must be last)

Traceback (most recent call last):

  File "C:\Development\Google\google_appengine\lib\protorpc-1.0\protorpc\wsgi\service.py", line 181, in protorpc_service_app

    response = method(instance, request)

  File "C:\Development\Google\google_appengine\lib\endpoints-1.0\endpoints\api_config.py", line 1332, in invoke_remote

    return remote_method(service_instance, request)

  File "C:\Development\Google\google_appengine\lib\protorpc-1.0\protorpc\remote.py", line 414, in invoke_remote_method

    response = method(service_instance, request)

  File "C:\Development\_Projects\API\project\endpoints_proto_datastore\ndb\model.py", line 1574, in QueryFromRequestMethod

    request_entity = cls.FromMessage(request)

  File "C:\Development\_Projects\API\project\endpoints_proto_datastore\ndb\model.py", line 1245, in FromMessage

    setattr(entity, name, value)

  File "C:\Development\_Projects\API\project\main.py", line 139, in set_parent

    if ndb.Key(GrandparentModel, self._grandparent, ParentModel, value).get() is None:

  File "C:\Development\Google\google_appengine\google\appengine\ext\ndb\key.py", line 220, in __new__

    self.__namespace) = self._parse_from_args(**kwargs)

  File "C:\Development\Google\google_appengine\google\appengine\ext\ndb\key.py", line 245, in _parse_from_args

    'Incomplete Key entry must be last')

BadArgumentError: Incomplete Key entry must be last

INFO     2015-08-16 10:26:59,174 module.py:809] default: "POST /_ah/spi/Child.child_list HTTP/1.1" 500 512
INFO     2015-08-16 10:26:59,174 module.py:809] default: "GET /_ah/api/project/v1/grandparent/g/parent/p/child HTTP/1.1" 503 196

I've been unable to figure it out on my own so far, so any help would be appreciated.

Thanks!

bossylobster bossylobster · Accepted Answer · 2015-08-16T17:37:16

UPDATE: You are assuming in sort_keys that when parent is parsed from the path that the grandparent will already have been parsed:

    if self._grandparent is None:
        raise Exception('set_parent: self._grandparent is None')

To avoid this issue, don't depend on the order that grandparent and parent are set.

Explanation: It's possible that the fields may come in a different order every time you run the process. When the endpoints library detects you have fields in the path, a CombinedContainer message class is created on the fly: the fields of this class are set in an arbitrary order. (The fields used to create a CombinedContainer come via the output Message.all_fields(). But Message uses a dictionary to store fields by name and then all_fields() uses dict.values() for the value. In Python, dictionary order is non-deterministic, so these may come back in any order possible.)

Then in endpoints-proto-datastore, the fields in the CombinedContainer are sorted by field number. This sort order matters in user defined classes, but is "ignored" when the user doesn't control the creation of the CombinedContainer.

Notice that in the keys with ancestors example the IdSet method calls SetKey but SetKey makes sure both the parent and the ID have been set

if self._parent is not None and self._id is not None:
    ...

Short Answer: Use

'grandparent/{grandparent}/parent/{parent}/child'

as your path template.

Long Answer: The requests being sent don't match the PATH templates.

Your URI template is

'{grandparent}/parent/{parent}/child'

but your path is

/_ah/api/project/v1/grandparent/g/parent/p/child

which corresponds to a relative path of

grandparent/g/parent/p/child

Depending on how the matching works (I'm not sure), you'll either get no match at all of {grandparent} will match with grandparent/g.

From your stacktrace, it looks like you have no match at all.

Endpoints Proto Datastore - Path {requirement} variable used for keys randomly breaks

1 Answers