Azure AppFabric caching. ErrorCode<ERRCA0017>:SubStatus<ES0006>

Question

there is a very strange thing going on
i have a web site made in MVC 3 deployed in an Azure WebRole
it was running fine and originally was made in VS2010
now i use VS2012
i decided to make some minor changes and it appeared to be working but keeps break down at random intervals
this is what is happening

after random intervals i get this error ErrorCode<ERRCA0017>:SubStatus<ES0006>
after 2-3 minutes of getting this error sites appears to be working fine

i havent changed anything from the previous version of the site related to caching
this error does nt happens when i test it in staging or in local environment
no where in my code im accessing this cache programmatically
i have copy-pasted entire configuration as it is from the azure portal and i know it it correct because these are the same configuration settings that were working before
i just using Azure AppFabric Caching because i need to save sessions some where. so the only place it is ever mentioned is in my websites web config and as i have told all the configuration can be assumed to be correct.

this is the stacktrace if this can be of some help

[DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)]
   Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody) +616
   Microsoft.ApplicationServer.Caching.DataCache.ExecuteAPI(RequestBody reqMsg, IMonitoringListener listener) +29
   Microsoft.ApplicationServer.Caching.DataCache.InternalGetAndLock(String key, TimeSpan timeout, DataCacheLockHandle& lockHandle, String region, Boolean lockKey, IMonitoringListener listener) +227
   Microsoft.ApplicationServer.Caching.<>c__DisplayClass78.<GetAndLock>b__77() +103
   Microsoft.ApplicationServer.Caching.DataCache.GetAndLock(String key, TimeSpan timeout, DataCacheLockHandle& lockHandle) +258
   Microsoft.Web.DistributedCache.<>c__DisplayClass31`1.<PerformCacheOperation>b__30() +19
   Microsoft.Web.DistributedCache.DataCacheRetryWrapper.PerformCacheOperation(Action action) +208
   Microsoft.Web.DistributedCache.DataCacheForwarderBase.GetAndLock(String key, TimeSpan timeout, DataCacheLockHandle& lockHandle) +190
   Microsoft.Web.DistributedCache.BlobBasedSessionStoreProvider.GetItem(HttpContextBase context, String id, Boolean acquireWriteLock, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) +593
   Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) +125
   System.Web.SessionState.SessionStateModule.GetSessionStateItem() +178
   System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData) +1076
   System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +115
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +375

UPDATE:-
suddenly i discovered a new error which seems to me as one of the inner errors many sites talk about
error description

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond [IP-Address]:[PORT]

and this is the new stack trace

[SocketException (0x274c): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond [IP-ADDRESS]:[PORT]]
   System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) +305
   System.Net.Sockets.Socket.Connect(EndPoint remoteEP) +162
   System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout) +774

[EndpointNotFoundException: Could not connect to [SOCKET-ENDPOINT]. The connection attempt lasted for a time span of 00:00:21.0308462. TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 111.221.89.111:22233. ]
   System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase) +518
   System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData) +401
   Microsoft.ApplicationServer.Caching.OpenDelegate.EndInvoke(IAsyncResult result) +0
   Microsoft.ApplicationServer.Caching.ChannelContainer.Opened(IAsyncResult ar) +122

Answer 1

There was an interesting blog post about this error over a year ago: Azure AppFabric Caching - ErrorCode:SubStatus : What to do?.

Before intializing the DataCacheFactory modify the DataCacheFactoryConfiguration.ChannelOpenTimeout of the dataCacheFactoryConfiguration instance that you are using to a larger value of 2 minutes. (can't use configuration to set the value, since 20 seconds in the allowed limit) . Also, this recommendation is only for debugging purposes and ideally should not be needed in production environments. This lets the underlying layer to respond back with the actual error that occurred. Now, if you check the inner exception, you should see what caused the request to fail.

Please follow these steps to find the inner exception which will explain why the request fails.

Note: If this is caused by a transient fault, you should consider using the Transient Fault Handling Application Block (which also supports AppFabric Cache) in order to implement a retry policy.