GAE converting dictionary to NDB datastore entity

Question

I would like to ask some guidelines on a small task that I am trying to solve. I am experimenting with a small app that uses JSON data to save entities.

I know that you can easily convert a dict to an entity by just creating the model but, I am trying to build a more generic approach that would convert any dict to an entity.

My steps are:

1. Get the dict.
2. Validate that the dict keys correspond to an entitys model definitions by reading the class.dict of the model.
3. Try to unpack the validated properties in the model class contructor (create the model instance)
4. return it.

So far I am ok but lack of my python knowledge, is either constraining me, or confusing me. Maybe I am as well forgetting or unaware of more simple way to do it.

So here is it:

@classmethod
def entity_from_dict(cls, parent_key, dict):
    valid_properties = {}
    logging.info(cls.__dict__)
    for property,value in dict.iteritems():
        if property in cls.__dict__: # should  not iterate over functions, classmethods, and @property
            logging.info(cls.__dict__[property]) # this outputs eg: StringProperty('title', required=True)
            logging.info(type(cls.__dict__[property])) #this is more interesting <class 'google.appengine.ext.ndb.model.StringProperty'>
            valid_properties.update({property: value})
    # Update the id from the dict
    if 'id' in dict: # if not creating a new entity
            valid_properties['id'] = dict['id']
    # Add the parent
    valid_properties['parent'] = parent_key
    #logging.info(valid_properties)
    try:
        entity = cls(**valid_properties)
    except Exception as e:
        logging.exception('Could not create entity \n' + repr(e))
        return False
    return entity

My problem is that I want only to validate ndb. Properties and not @classmethods, @property as well because this causes a conflict.

I am also using expando classes, so any property in the dict that is extra gets stored.

How can I check against these specific types?

Answer 1

Solved it as @Tim Hoffman proposed using the ._properties of the Ndb model. A thing I didn't know is that via the ._properties I could get the model definition properties and I thought that it would only return the instance properties :-).

Also I did not use populate because I find that it does the same as passing the valid dict unpacked in the model's contructor ;-)

So here it is:

@classmethod
def entity_from_dict(cls, parent_key, data_dict):
    valid_properties = {}
    for cls_property in cls._properties:
        if cls_property in data_dict:
            valid_properties.update({cls_property: data_dict[cls_property]})
    #logging.info(valid_properties)
    # Update the id from the data_dict
    if 'id' in data_dict: # if creating a new entity
            valid_properties['id'] = data_dict['id']
    # Add the parent
    valid_properties['parent'] = parent_key
    try:
        entity = cls(**valid_properties)
    except Exception as e:
        logging.exception('Could not create entity \n' + repr(e))
        return False
    return entity

Answer 2

The JSON dump method in python which we using during the converting models to JSON for export converts non-strings into strings. Therefore Jimmy Kane methods throw the error due to model incompatibility. To avoid this problem I updated his method and added a method named prop_literal just for converting non-string characters which capsuled in the string into their literal type.

I also added the entity.put() to add the entity to datastore because the aim was that :)

def prop_literal(prop_type,prop_val):
    """
    Convert non-string encapsulated in the string into literal type
    """
    if "Integer" in prop_type:
        return int(prop_val)
    elif "Float" in prop_type:
        return float(prop_val)
    elif "DateTime" in prop_type:
        # bos gecsin neticede locale
        return None
    elif ("String" in prop_type) or ("Text" in prop_type):
        return prop_val
    elif "Bool" in prop_type:
        return True if prop_val == True else False
    else:
        return prop_val


def entity_from_dict(cls, parent_key, data_dict):
    valid_properties = {}
    for cls_property in cls._properties:
        if cls_property in data_dict:
            prop_type = str(cls._properties[cls_property])
            # logging.info(prop_type)
            real_val = prop_literal(prop_type,data_dict[cls_property])
            try:

                valid_properties.update({cls_property: real_val})
            except Exception as ex:
                # logging.info("Veri aktariminda hata:"+str(ex))
        else:
            # logging.info("prop skipped")
    #logging.info(valid_properties)
    # Update the id from the data_dict
    if 'id' in data_dict: # if creating a new entity
            valid_properties['id'] = data_dict['id']
    # Add the parent
    valid_properties['parent'] = parent_key
    try:
        entity = cls(**valid_properties)
        logging.info(entity)
        entity.put()
    except Exception as e:
        logging.exception('Could not create entity \n' + repr(e))
        return False
    return entity