Atomic operations in azure table storage

Question

I am looking to implement a page view counter in azure table storage. If say two users visit the page at the same time, and the current value on PageViews = 100, is it guaranteed that the PageViews = 102 after the update operation?

Answer 1

The answer depends on how you implement your counter. :-)

Table storage doesn't have an "increment" operator, so you'd need to read the current value (100) and update it to the new value (101). Table storage employs optimistic concurrency, so if you do what comes naturally when using the .NET storage client library, you'd likely see an exception when two processes tried to do this simultaneously. This would be the flow:

1. Process A reads the value of PageViews and receives 100.
2. Process B reads the value of PageViews and receives 100.
3. Process A makes a conditional update to PageViews that means "set PageViews to 101 as long as it's currently 100." This succeeds.
4. Process B performs the same operations and fails, because the precondition (PageViews == 100) is false.

The obvious thing to do when you receive the error is to repeat the process. (Read the current value, which is now 101, and update to 102.) This will always (eventually) result in your counter having the correct value.

There are other possibilities, and we did an entire Cloud Cover episode about how to implement a truly scalable counter: http://channel9.msdn.com/Shows/Cloud+Cover/Cloud-Cover-Episode-43-Scalable-Counters-with-Windows-Azure.

What's described in that video is probably overkill if collisions are unlikely. I.e., if your hit rate is one-per-second, the normal "read, increment, write" pattern will be safe and efficient. If, on the other hand, you receive 1000 hits per second, you'll want to do something smarter.

EDIT

Just wanted to clarify for people who read this to understand optimistic concurrency... the conditional operation isn't really "set PageViews to 101 as long as it's currently 100." It's more like "set PageViews to 101 as long as it hasn't changed since the last time I looked at it." (This is accomplished by using the ETag that came back in the HTTP request.)

Answer 2

You could also rethink the 'count' part. Why not turn this into a 2 step process?

Step 1 - Recording Page Views

Each time someone views a page add a record to a table (let's call it PageViews). The info you would add in one of these stores would be the following:

• PartitionKey = PageName
• RowKey = Random GUID

After a few views you would have something like this:

• MyPage.aspx - someGuid
• MyPage.aspx - someGuid
• SomePage.aspx - someGuid
• MyPage.aspx - someGuid

Step 2 - Counting Page Views

What we want to do now is get all those records, count them, increase a counter somewhere and delete all records. Let's assume you have multiple workers running. Both your workers would have a loop randomly running between 1 and 10 minutes. Each time the worker's time elapsed it will take the lease on a blob if no lease has been taken yet (this should always be the same blob, you can use AutoRenewLease).

The first worker getting the lock can go ahead and do the counting:

1. Get all records from the PageViewRecordings table or from cache
2. Count all page views per page
3. Update count somewhere
4. Delete the records that were taken into account when counting

The issue here is that it's very hard to turn this into an idempotent process. What happens if your instance crashes between the count and the delete? You'll have an increased page count, but since the items were not deleted they'll be added to the total count the next time you process them.

This is why I would suggest the following. In the same table (PageViews), you will also record the total page views, in that same partition. But the data will be a bit different (this will be a single record in that partition holding the total count):

• PartitionKey = PageName
• RowKey = Guid.Empty (just don't use a random guid, this way we know the difference between a recorded page view and the record holding the total count).
• Count = The current page view count

This is perfectly possible because Table Storage is schema less. And why are we doing this? Because we do have transactions if we limit ourselves to the same table + partition with a maxmium of 100 entities. What can we do with this?

1. Using Take, we get 100 records from that table + partition.
2. The first record we'll get is the 'counter' record. Why? Because its rowkey is Guid.Empty and sorting is lexicographical
3. Count these records (-1 because the first record isn't a page view, it's just our counter placeholder)
4. Update the Count property of the counter record
5. Delete the 99 (or less) other records
6. SaveChanges using Batch.
7. Repeat until there is only 1 record left (the counter record).

And each X minutes your workers will see if there isn't a lease on the blob, get a lease and restart the process.

Is this answer clear enough or should I add some code?

Answer 3

I came along with the same question. With Azure python library, I'm developing a simple counter increment using eTag and If-Match instead of lock. The basic idea is to retry to increase the counter until the update successfully runs under a certain criteria, which is no other updates interfere this running update. If the request of updates are heavy, sharding should be invoked.

https://github.com/flyakite/simple-scalable-datastore/blob/master/datastore/azuretable.py

Answer 4

If using Azure Websites, then Azure Queues and WebJobs is another option. In one scenario of mine though I am actually going to take the sharding approach and have WebJobs update the aggregates periodically. An Azure Table Storage Table of UserPageViews with PartitionKey = User and RowKey = Page . Two simultaneous users with the same user id will not be allowed.