Azure: CacheInstaller and ConfigStoreException

Question

I'm trying to use co-located cache. And I'm getting the following exception during cloud emulator start proccess:

Application: CacheInstaller.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.ApplicationServer.Caching.ConfigStoreException
Stack:
   at Microsoft.ApplicationServer.Caching.AzureCommon.AzureUtility.RetryExponentialTask(System.Func`1<Boolean>, System.Predicate`1<System.Exception>, System.TimeSpan, System.TimeSpan, System.TimeSpan, Int32)
   at Microsoft.ApplicationServer.Caching.AzureCommon.Package.InitializeExternalStore(Microsoft.ApplicationServer.Caching.ICustomProvider)
   at Microsoft.ApplicationServer.Caching.AzureCommon.Package.InitializeExternalStoreIfRequired()
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

My config:

  <dataCacheClients>
    <tracing sinkType="DiagnosticSink" traceLevel="Error" />
    <dataCacheClient name="default">
      <autoDiscover isEnabled="true" identifier="Web" />
    </dataCacheClient>
  </dataCacheClients>

Could someone help me? :)

Edited:

I'm using VS 2010 SP1, Windows 7. I removed SDK 1.6 and reinstalled 1.7.

Cloud config (web role):

<ServiceConfiguration serviceName="Cloud" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" schemaVersion="2012-05.1.7">
  <Role name="Web">
    <Instances count="1" />
    <ConfigurationSettings>
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.Enabled" value="true" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountUsername" value="..." />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountEncryptedPassword" value="..." />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountExpiration" value="..." />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteForwarder.Enabled" value="..." />
      <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=..." />
      <Setting name="DiagnosticsConnectionString" value="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=..." />
      <Setting name="Microsoft.WindowsAzure.Plugins.Caching.NamedCaches" value="{&quot;caches&quot;:[{&quot;name&quot;:&quot;default&quot;,&quot;policy&quot;:{&quot;eviction&quot;:{&quot;type&quot;:0},&quot;expiration&quot;:{&quot;defaultTTL&quot;:10,&quot;isExpirable&quot;:true,&quot;type&quot;:1},&quot;serverNotification&quot;:{&quot;isEnabled&quot;:false}},&quot;secondaries&quot;:0}]}" />
      <Setting name="Microsoft.WindowsAzure.Plugins.Caching.Loglevel" value="" />
      <Setting name="Microsoft.WindowsAzure.Plugins.Caching.CacheSizePercentage" value="30" />
      <Setting name="Microsoft.WindowsAzure.Plugins.Caching.ConfigStoreConnectionString" value="UseDevelopmentStorage=true" />
    </ConfigurationSettings>
    <Certificates>
      <Certificate name="Microsoft.WindowsAzure.Plugins.RemoteAccess.PasswordEncryption" thumbprint="..." thumbprintAlgorithm="sha1" />
    </Certificates>
  </Role>
</ServiceConfiguration>

Edited 2:

I tried to create empty test project:

1. Created Windows Azure Cloud Service project
2. Selected ASP.NET Web role
3. Enabled cache at caching tab of web role properties (co-located)
4. Installed Windows Azure Caching Preview
5. Started cloud project by using NuGet
6. The same exception

Edited 3:

I installed VS 2012 RC and tried to create test project. The same problem occured.

Edited 4:

Exception details:

Microsoft.ApplicationServer.Caching.ConfigStoreException:
  Windows Azure Storage Service is not ready/available.  StorageAccountName = DevStorage, BlobNamePrefix = deployment17(44)__Web --->
  Microsoft.WindowsAzure.StorageClient.StorageClientException: ERROR --->
  System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at Microsoft.WindowsAzure.StorageClient.EventHelper.ProcessWebResponse(WebRequest req, IAsyncResult asyncResult, EventHandler`1 handler, Object sender)
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.get_Result()
   at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.ExecuteAndWait()
   at Microsoft.WindowsAzure.StorageClient.CommonUtils.<LazyEnumerateSegmented>d__0`1.MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at Microsoft.ApplicationServer.Caching.WindowsAzureBlobProvider.IsInitialized()
   --- End of inner exception stack trace ---
   at Microsoft.ApplicationServer.Caching.AzureCommon.AzureUtility.RetryExponentialTask(Func`1 doTask, Predicate`1 shouldRethrow, TimeSpan minBackoff, TimeSpan maxBackoff, TimeSpan deltaBackoff, Int32 retryCount)
   at Microsoft.ApplicationServer.Caching.AzureCommon.Package.InitializeExternalStore(ICustomProvider customProvider)
   at Microsoft.ApplicationServer.Caching.AzureCommon.Package.InitializeExternalStoreIfRequired()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Answer 1

It looks to me like there was an issue when it is attempting to initialize the backing store for the cache. You're getting a 400 Bad Request from your Blob storage provider. Without caching configured, from another test project are you able to access your BLOB storage in your dev storage? Is the devstorage emulator running?

Check the storage emulator is running by looking at the notification area / systray and right-clicking on the little Azure icon there for the emulators. If one of the content menus is "Start Storage Emulator" the storage service isn't running, click it. If it thinks it is running then click on the Show Storage Emulator and ensure that the Blob and other services are checked and have a status of running. If all that is the case then attempt to read/write to your development blob storage using another test app or a 3rd party tool like Cerebrata's Cloud Storage Studio.

The caching system is attempting to create a folder in BLOB storage called cacheclusterconfigs where each of the members of the cluster can come see the configuration. For whatever reason, yours can't access the BLOB storage to do this.

To answer your other question, yes, the CacheInstaller does log. When you start up your instance, even before it comes all the way up you should be able to look at the compute emulator and select an instance. Then right click the instance and choose "open local store". This will open an explorer window to where the emulator puts the files specific for each instance. It will be something like C:\Users{user}\AppData\Local\dftmp\Resources\8d403f60-3b09-481a-8b9c-86ab9e06954b. The GUID is the instance. From that window dig into the following directory: \directory\Microsoft.WindowsAzure.Plugins.Caching.FileStore. Here you will see log files from the caching service. If yours is stuck installing then you should see a file in this directory like DedicatedCacheLogs_CacheInstaller_xxxx.log (the xxxx are numbers). This might help you. Note that once the cache installer is done then it seems there is a service that is sweeping this directory and putting older files into the LogsBackup X directory which is also in this folder.

Note that the cacheinstaller.exe.config and related executable are actually copied into the package, and then into the instance for the emulator, from the C:\Program Files\Microsoft SDKs\Windows Azure.NET SDK\2012-06\bin\plugins\Caching directory. Perhaps you might be able to add another tracelistener to the exe.config from that SDK directory and get it to spit out additional information if you can't find what you need from the instance directory I indicated above.

Hopefully these hints will help you. I couldn't recreate the issue on my end unless I gave it a bad storage account address for an account in Azure and even then it wasn't a 100% match to your error.