python - "Too much contention" when creating new entity in dataStore

Question

This morning my GAE application generated several error log: "too much contention on these datastore entities. please try again.". In my mind, this type of error only happens when multiple requests try modify the same entity or entities in the same entity group.

When I got this error, my code is inserting new entities. I'm confused. Does this mean there is a limitation of how fast we can create new entity?

My code of model definition and calling sequence is show below:

# model defnition
class ExternalAPIStats(ndb.Model):
    uid = ndb.StringProperty()
    api = ndb.StringProperty()
    start_at = ndb.DateTimeProperty(auto_now_add=True)
    end_at = ndb.DateTimeProperty()

# calling sequence
stats = ExternalAPIStats(userid=current_uid, api="eapi:hr:get_by_id", start_at=start_at, end_at=end_at)
stats.put()  # **too much contention** happen here

That's pretty mysterious to me. I was wondering how I shall deal with this problem. Please let me know if any suggestion.

I have same issue. I'm using pipelines. I put() a new entity when running the pipeline. My code is not wrapped in a transaction (not of my own making anyway) but the line where I put is surfacing as TransactionFailedError(too much contention on these datastore entities. please try again.) I haven't been able to find an explanation for this behaviour in the docs — Anentropic
I only just noticed that this is a new bounty on a year-old question. :) Anentropic, are you creating entities using keys with a common ancestor? Creating new entities in the same entity group is equivalent to making concurrent changes to the same group, and these will contend for access even if you're not using an explicit transaction. (Each operation is in its own implicit transaction if there is no explicit transaction.) If you don't believe you're using a common ancestor in the keys, can you post your code, possibly on a new question? — Dan Sanderson
@DanSanderson actually I was, I thought I wasn't but it turned out I was. It seems it's still possible to get contention of the kind described by OP though, due to 'hot tablets' issue... I found that article that Bruyere posted as answer and it is also buried in the docs somewhere — Anentropic
Yeah, I got someone to confirm that contention errors are strictly about contention, and hot tablets will result in slowness or possibly timeouts. Glad it worked out! — Dan Sanderson

Ryan Ryan · Accepted Answer · 2014-09-12T14:45:52

Without seeing how the calls are made(you show the calling code but how often is it called, via loop or many pages calling the same put at the same time) but I believe the issue is better explained here. In particular

You will also see this problem if you create new entities at a high rate with a monotonically increasing indexed property like a timestamp, because these properties are the keys for rows in the index tables in Bigtable.

with the 'start_at' being the culprit. This article explains in more detail.

Possibly (though untested) try doing your puts in batches. Do you run queries on the 'start_at' field? If not removing its indexes will also fix the issue.

How is the puts called (ie what I was asking above in a loop, multiple pages calling)? With that it might be easier to narrow down the issue.

python - "Too much contention" when creating new entity in dataStore

2 Answers