Azure Compute Emulator error "The communication object ... cannot be used for communication because it is in the Faulted state."

Question

I have some setup & teardown scripts that use csrun.exe to prepare a local compute emulator for some automated tests.

C:\Program Files\Microsoft SDKs\Azure\Emulator\csrun.exe /removeall
C:\Program Files\Microsoft SDKs\Azure\Emulator\csrun.exe /devfabric:clean

C:\Program Files\Microsoft SDKs\Azure\Emulator\csrun.exe /run:"c:\myapp\csx\Release";"c:\myapp\bin\Release\app.publish\ServiceConfiguration.Local.cscfg"

The csrun call to removeall inconsistently fails with this error:

Encountered an unexpected error The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.IDisposable.Dispose()
at Microsoft.ServiceHosting.Tools.DevelopmentFabric.SingleInstanceFabricClient.Connect()
at Microsoft.ServiceHosting.Tools.DevelopmentFabric.DevFabric.EnsureClient(Boolean checkConnection)
at Microsoft.ServiceHosting.Tools.DevelopmentFabric.DevFabric.d__0.MoveNext()
at Microsoft.ServiceHosting.Tools.CloudServiceRun.DoActions.RemoveAll()
at Microsoft.ServiceHosting.Tools.CloudServiceRun.DoActions.ParseArguments(String[] args, Boolean doActions)
at Microsoft.ServiceHosting.Tools.CloudServiceRun.DoActions.ExecuteActions(String[] args).

And the csrun call for deployment also inconsistently fails with the same error (slightly different stack trace):

Encountered an unexpected error The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.IDisposable.Dispose()
at Microsoft.ServiceHosting.Tools.DevelopmentFabric.SingleInstanceFabricClient.RunServiceDeployment(String tenantName)
at Microsoft.ServiceHosting.Tools.DevelopmentFabric.Deployment.Start()
at Microsoft.ServiceHosting.Tools.CloudServiceRun.DoActions.Run(DirectoryInfo dir, FileInfo serviceConfiguration, Boolean launchBrowser, Boolean paused, String debugger, Boolean useIISExpress, List`1 portOverrides)
at Microsoft.ServiceHosting.Tools.CloudServiceRun.DoActions.ParseArguments(String[] args, Boolean doActions)
at Microsoft.ServiceHosting.Tools.CloudServiceRun.DoActions.ExecuteActions(String[] args).

This is the Compute Emulator v2.4.

I've confirmed the web.config is writable and valid. (As suggested by this thread.)

I've had a similar setup working on another machine with no issues. The only difference I'm aware of is that the working machine has a single-core CPU while this failing one is dual-core.

Answer 1

If the error reported in the files in %LocalAppData%\dftmp\DFServiceLogs looks like this:

DFService Information: 0 : [00002796:00000014, 2019/02/06 17:47:59.054] Exception:System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
   at GARStartRoleEx(UInt16* , UInt16* , UInt16* , _RUNTIME_CERTIFICATE_BLOB_TYPE , Byte* , UInt32 )
   at Microsoft.WindowsAzure.GuestAgent.EmulatorRuntime.EmulatorRuntimeImpl.StartRole(String roleInstanceId, CertificateBlobType certsBlobType, Byte[] certificatesBlob)
   at Microsoft.ServiceHosting.Tools.DevelopmentFabric.Fabricator.StartRoleInstance(RDConfig roleInstanceConfig)`

then your problem is probably that the C++ Runtime that the Compute Emulator depends on is not installed. It needs the VS 2012 C++ Runtime, which I'm guessing used to be installed by earlier versions of Visual Studio, but isn't by default with VS2017.

Installing the correct runtime from http://www.microsoft.com/en-us/download/details.aspx?id=30679 fixed the issue for me.

Answer 2

In my case, I deleted the certificate used for SSL and created a new one. Fixed my problem.

Answer 3

After enabling more logging in c:\Program Files\Microsoft SDKs\Azure\Emulator\csrun.exe.config, investigating csrun.svclog, and investigating the logs in %LOCALAPPDATA%\dftmp , determined the root error was likely this happening in the dfservice.exe: "System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))".

Using Procmon, found that around the time of the error, dfservice.exe encountered a "FILE LOCKED WITH ONLY READERS" issue when attempting to access myapp\csx\Release\roles\MyWorkerRoleName\base\x64\WaHostBootstrapper.exe

My best guess is that the emulator is holding onto files from prior instances despite my efforts to do cleans and removes.

I think I solved it by changing my build scripts to create a fresh unique directory in %TEMP% for each build execution (and emulator deployment) where I can copy the contents of csx\Release\ beforehand; then I deploy to the emulator from that. (With some additional script code that later attempts to recursively delete all those temp files, but ignores any errors.)

Answer 4

Same error in UI appears, if the Compute Emulator is not running at all. The inner exception from log is UnauthorizedAccessException, then. Starting it manually helps, because the Studio Debugger tries to start, but does not wait and the startup procedure takes a while and when it's up the debugger has already thrown the error and stopped.

Answer 5

Running Visual Studio as Administrator have solved the problem for me.