How to manipulate files in google app engine datastore

Question

My problem revolves around a user making a text file upload to my app. I need to get this file and process it with my app before saving it to the datastore. From the little I have read, I understand that user uploads go directly to the datastore as blobs, which is ok if I could then get that file, perform operations on it(meaning change data inside) and then re-write it back to the datastore. All these operations need to be done by the app. Unfortunately from the datastore documenation, http://code.google.com/appengine/docs/python/blobstore/overview.html an app cannot directly create a blob in the datastore. That's my main headache. I simply need a way of creating a new blob/file in the datastore from my app without any user upload interaction.

Answer 1

blobstore != datastore.

You can read and write data to the datastore as much as you like so long as your data is <1MB using a db.BlobProperty on your entity.

As Wooble comments, the new File API lets you write to the blobstore, but unless you are incrementally writting to the blobstore-file using tasks or something like the mapreduce library you are still limited by the 1MB API call limit for reading/writing.

Answer 2

Thanks for your help. After many sleepless nights, 3 App Engine Books and A LOT of Googling, I've found the answer. Here is the code (it should be pretty self explanatory):

from __future__ import with_statement
from google.appengine.api import files
from google.appengine.ext import blobstore
from google.appengine.ext import webapp
from google.appengine.ext.webapp import util

class MainHandler(webapp.RequestHandler):
    def get(self):
        self.response.out.write('Hello WOrld')
        form=''' <form action="/" method="POST" enctype="multipart/form-data">
Upload File:<input type="file" name="file"><br/>
<input type="submit"></form>'''
        self.response.out.write(form)
        blob_key="w0MC_7MnZ6DyZFvGjgdgrg=="
        blob_info=blobstore.BlobInfo.get(blob_key)
        start=0
        end=blobstore.MAX_BLOB_FETCH_SIZE-1
        read_content=blobstore.fetch_data(blob_key, start, end)
        self.response.out.write(read_content)
    def post(self):
        self.response.out.write('Posting...')
        content=self.request.get('file')
        #self.response.out.write(content)
        #print content
        file_name=files.blobstore.create(mime_type='application/octet-stream')
        with files.open(file_name, 'a') as f:
            f.write(content)
        files.finalize(file_name)
        blob_key=files.blobstore.get_blob_key(file_name)
        print "Blob Key="
        print blob_key

def main():
    application=webapp.WSGIApplication([('/', MainHandler)],debug=True)
    util.run_wsgi_app(application)

if __name__=='__main__':
    main()