I am fetching a GZipped LXML file and trying to write Product entries to a Databse Model. Previously I was having local memory issues, which were resolved by help on SO (question). Now I got everything working and deployed it, however on the server I get the following error:
Exceeded soft private memory limit with 158.164 MB after servicing 0 requests total
Now I tried all I know to reduce the memory usage and am currently using the code below. The GZipped file is about 7 MB whereas unzipped it is 80 MB. Locally the code is working fine. I tried running it as HTTP request as well as Cron Job but it didn't make a difference. Now I am wondering if there is any way to make it more efficient.
Some similar questions on SO referred to Frontend and Backend specification, which I am not familiar with. I am running the free version of GAE and this task would have to run once a week. Any suggestions on best way to move forward would be very much appreciated.
from google.appengine.api.urlfetch import fetch
import gzip, base64, StringIO, datetime, webapp2
from lxml import etree
from google.appengine.ext import db
class GetProductCatalog(webapp2.RequestHandler):
def get(self):
user = XXX
password = YYY
url = 'URL'
# fetch gziped file
catalogResponse = fetch(url, headers={
"Authorization": "Basic %s" % base64.b64encode(user + ':' + password)
}, deadline=10000000)
# the response content is in catalogResponse.content
# un gzip the file
f = StringIO.StringIO(catalogResponse.content)
c = gzip.GzipFile(fileobj=f)
content = c.read()
# create something readable by lxml
xml = StringIO.StringIO(content)
# delete unnecesary variables
del f
del c
del content
# parse the file
tree = etree.iterparse(xml, tag='product')
for event, element in tree:
if element.findtext('manufacturer') == 'New York':
if Product.get_by_key_name(element.findtext('sku')):
coupon = Product.get_by_key_name(element.findtext('sku'))
if coupon.last_update_prov != datetime.datetime.strptime(element.findtext('lastupdated'), "%d/%m/%Y"):
coupon.restaurant_name = element.findtext('name')
coupon.restaurant_id = ''
coupon.address_street = element.findtext('keywords').split(',')[0]
coupon.address_city = element.findtext('manufacturer')
coupon.address_state = element.findtext('publisher')
coupon.address_zip = element.findtext('manufacturerid')
coupon.value = '$' + element.findtext('price') + ' for $' + element.findtext('retailprice')
coupon.restrictions = element.findtext('warranty')
coupon.url = element.findtext('buyurl')
if element.findtext('instock') == 'YES':
coupon.active = True
else:
coupon.active = False
coupon.last_update_prov = datetime.datetime.strptime(element.findtext('lastupdated'), "%d/%m/%Y")
coupon.put()
else:
pass
else:
coupon = Product(key_name = element.findtext('sku'))
coupon.restaurant_name = element.findtext('name')
coupon.restaurant_id = ''
coupon.address_street = element.findtext('keywords').split(',')[0]
coupon.address_city = element.findtext('manufacturer')
coupon.address_state = element.findtext('publisher')
coupon.address_zip = element.findtext('manufacturerid')
coupon.value = '$' + element.findtext('price') + ' for $' + element.findtext('retailprice')
coupon.restrictions = element.findtext('warranty')
coupon.url = element.findtext('buyurl')
if element.findtext('instock') == 'YES':
coupon.active = True
else:
coupon.active = False
coupon.last_update_prov = datetime.datetime.strptime(element.findtext('lastupdated'), "%d/%m/%Y")
coupon.put()
else:
pass
element.clear()
UDPATE
According to Paul's suggestion I implemented the backend. After some troubles it worked like a charm - find the code I used below.
My backends.yaml looks as follows:
backends:
- name: mybackend
instances: 10
start: mybackend.app
options: dynamic
And my app.yaml as follows:
handlers:
- url: /update/mybackend
script: mybackend.app
login: admin
del cprobably won't do anything unless you explicit call gc.collect() as the stuff will not probably be collected for quite some time. Also look at your code , you have the read file/StringIO, xml (which is the StringIO wrapped version of c), then the full parsed tree. You said uncompressed it's 80MB, pluse you have at least one copy you haven't gc'd plus the tree. You might consider using a pull parse strategy which would mean you don't have a full parsed tree copy in memory as well as the string. - Tim Hoffman