High CPU usage during reads from table storage

2
votes

Prerequisites:

Azure Storage 3.0.0
VM size: Small(1 core)
Table: 1 partition - 2k rows, 10 columns

The story:

I have been observing the high CPU usage on one of my roles for a while and then I decided to digg a little. With a help of the profiler I spotted the hot path.

enter image description here Once I realized that it was not my code which generated such load, I removed everything but the reading from the table itself. I removed custom entity resolver as well. The reading code is like in the Microsoft's how-to.

Unfortunately, I found just one simillar situation and the answer there gave me nothing: similar situation here

Question:

Any ideas?

1
performanceazurecpu-usageazure-table-storage
Can you post relevant snippets of your code? How are you reading storage? Are you setting any ServicePointManager properties, like UseNagleAlgorithm and Expect100Continue to false and changing the default connection limit? – Igorek
@Igorek Sure, UseNagleAlgorithm and Expect100Continue are set to false. DefaultConnection limit wouldn't be a problem here. According to the hot path screenshot it consumes cpu right after EndGetResponse. Serialization? – Max
Serialization is most likely culprit... is it possible that the data was saved with custom entity resolver and now the engine is having a hard time dealing with de-serialization of that particular data only? – Igorek
@Igorek Originally those entities were saved with custom resolver. In the question I mentioned that I switched it off. So we can say that now there are 10 fields in this entity and only one is mapped back to entity during my test. Having said that, I think that the main problem here in the ODataReaderCore.InterceptException method. Could it be? – Max

1 Answers

2
votes

I am on the team that develops the client library for azure storage. We will be looking into this to see if we can reproduce it on our side. That being said If you are using 3.0 and POCO types you can switch to JSON nometadata by setting the PayloadFormat on the CloudTableClient, or passing it in via TableRequestOptions (for examples see here). This will no only reduce the payload, but it will use the Newtonsoft JSON deserializer rather than Odata. It would be good to see if you can reproduce the high cpu under this configuration.

Further, can you provide more information regarding you scenario, i.e. number of queries per second etc. It would also be good to run this under the debugger to see if there are exceptions being thrown and absorbed internally by odata.