Bulk Upload with Google App Engine Datastore via CSV

2
votes

I am building a GAE webapp using Python. I am also using the Datastore and trying to bulk upload data to the DB using the terminal and a CSV file as per:

https://developers.google.com/appengine/docs/python/tools/uploadingdata

I have created a Loader class in a separate.py file in my app root directory. I am not really sure if this loader class should be in my main.py webapp file, or another file in the root directory.

Loader class:

import datetime
from google.appengine.ext import db
from google.appengine.tools import bulkloader
import models


class FTSELoader(bulkloader.Loader):
    def __init__(self):
        bulkloader.Loader.__init__(self, 'FTSE',
                               [('date', lambda x: datetime.datetime.strptime(x, '%Y/%m/%d')),                                 
                               ('close', float)])
loaders = [FTSELoader]

My kind class (i.e. my Datastore table) I am trying to create/upload is called "FTSE". I then run this command in Terminal:

appcfg.py upload_data --config_file=FTSEdataloader.py --filename=FTSEdata.csv --      kind=FTSE --url=http://<myapp.appspot.com>/_ah/remote_api

I get the following error:

File "FTSEdataloader.py", line 4, in import models ImportError: No module named models

I do not have a "models.py" like in the GAE demonstration. What should take its place?

Thanks

2
google-app-enginegoogle-cloud-datastore
Where are your models defined if not in models.py? – georgerw
Sorry I am new to programming so not sure if I fully understand. To define my tables in the DB, I have classes in my main.py app file. Should this then be "import main.py?" instead? – Reno
It's common to put your models in their own file, called models.py. In main.py you can import them with from .models import MyModel. By doing this you will avoid problems with circular imports which will occur when your application grows – georgerw

2 Answers

1
votes

I had the same problem. I'm not sure why the appcfg.py can't find the models module when running the upload script. I got around the problem by doing this:

import datetime
from google.appengine.ext import db
from google.appengine.tools import bulkloader

class FTSE(db.Model):
    date = DateTimeProperty()
    close = FloatProperty()

class FTSELoader(bulkloader.Loader):
    def __init__(self):
        bulkloader.Loader.__init__(self, 'FTSE',
                           [('date', lambda x: datetime.datetime.strptime(x, '%Y/%m/%d')),                                 
                           ('close', float)])
loaders = [FTSELoader]

Basically it is just putting your model definition in the bulkloader. It certainly isn't the best way to do this, but it will work around the PYTHONPATH problem that appcfg.py seems to have when it is running the bulk upload.

0
votes

You do this using a file of Python code. The file imports or defines the Model classes for the entities being created, defines a loader class for each kind you wish to import, and declares the available loader classes in a global variable.

For example, say you have a Model class named "FTSE" defined in a file named models.py (which is in your PYTHON PATH, such as the directory where you'll run the tool Ex: C:\Python27) that resembles the following:

models.py

from google.appengine.ext import db

class FTSE(db.Model):
   date = db.DateProperty()
   close = db.FloatProperty()