New entity without ancestors caused "Too much contention" error

Question

Today my live Python application on App Engine had about 100 requests within a ten minute period for which an exception was thrown when writing a new entity to the datastore, due to "too much contention on these datastore entities". My app only writes one entity per request. Not all requests in this period failed; the majority were successful.

For weeks now the app has processed approx. 30k requests a day without problems, and I haven't seen the QPS exceed 20. In fact, there are examples of failed writes when there are gaps larger than one second between requests. I'm using the db library, here's the model definition:

class SiteImpression(db.Model):
    remoteIPAddr = db.StringProperty(required=True)
    aPersonID = db.StringProperty(required=True)
    requestURI = db.TextProperty(required=True)
    serverEnvironment = db.StringProperty(required=True)
    requestDateTime = db.DateTimeProperty(required=True)
    userAgent = db.TextProperty(required=True)

and the code to write to the datastore:

dtDateTime = datetime.now()
e = myd.SiteImpression(remoteIPAddr=strRemoteAddr,
                       aPersonID=straID,
                       requestURI=strRequestURI,
                       serverEnvironment=strEnvironment,
                       requestDateTime=dtDateTime,
                       userAgent=strUserAgent)
e.put()

From reading answers to this SO question, I gather this has to do with my indexed datetime property causing hot tablets (which I have read about here). I have a default index for this datetime property, and two custom indexes:

- kind: SiteImpression
  properties:
  - name: aPersonID
  - name: serverEnvironment
  - name: requestDateTime
    direction: asc

- kind: SiteImpression
  properties:
  - name: serverEnvironment
  - name: requestDateTime
    direction: asc

My question: am I doing something wrong to cause these contention issues, or was is likely just a temporary issue with the datastore?

Thanks.

You're doing exactly what those links say not to do. The article gives some solutions... but if you need that index, there's nothing you can really do about it except to retry the puts if they fail — Nick Franceschina
Thanks for your comment but I disagree with your phasing. Given that the second link says "Generally speaking, you don’t have to worry about this sort of thing until your application hits 100s of queries per second" in the opening paragraph, I'm doing exactly what it says to do. — IdRatherBeCoding

IdRatherBeCoding IdRatherBeCoding · Accepted Answer · 2015-04-16T12:26:44

After asking this question, I received an email from the google-appengine-downtime-notify group: https://groups.google.com/forum/#!topic/google-appengine-downtime-notify/cAr5V_EWs-g

There was a temporary issue with writing indexes on the same date as when I received my errors.

New entity without ancestors caused "Too much contention" error

1 Answers