Microsoft Service Fabric Host Service (FabricHostSvc) Hangs on Start

Question

I've been working with Microsoft Service Fabric since November 2015 and have encountered many issues but now Service Fabric has become completely non-functional on my development machine. Uninstall/reinstall doesn't help.

I was using 1.5-preview and have since tried 2.0 to no avail.

The problem started when I attempted to run a Service Fabric Application from Visual Studio 2015 Update 1 (as I have done hundreds of times over the past few months).

My machine blue-screened (first time I've seen a Windows 10 blue screen). After rebooting, I was unable to get my Service Fabric Application to deploy via Visual Studio. The PowerShell script failed with the following message:

Starting service FabricHostSvc. This may take a few minutes... Start-Service : Failed to start service 'Microsoft Service Fabric Host Service (FabricHostSvc)'.

I went into the SCM and found "Microsoft Service Fabric Host Service" was in a state of Starting. It stayed that way for an hour. I tried stopping and starting the service several times and each time it hangs.

I uninstalled Service Fabric (Service Fabric, SDK and Tools for VS) and re-installed with the latest version 2.0 and it exhibits the same problem.

Reboot, same problem.

Removed c:\SfDevCluster folder, same problem.

Based on some other articles, I looked for any stray performance counters after uninstalling but there weren't any.

I tried looking through the registry but there are other Azure components with "Fabric" in the name. If I delete them, I will probably hose the rest of my Azure dev setup.

Now... when I attempted to start the service again, it did re-create the SfDevCluster folder and give me some logs. It seems to create two trace log files per minute and they have the EXACT same contents.

Every time it fails, the final line of the trace is:

Info ,11176,General.FabricSetup.Main,Operation failed with error 0xffffffff

An earlier trace (SF 1.5) seemed to use a constant rather than the hex value for the error. Seemed to indicate an invalid argument.

Whatever this failure is, it seems to be the cause of my woes. Unfortunately, the error is completely unhelpful.

I'm trying to avoid reinstalling Windows because that will kill an entire day of productivity.

Any help is greatly appreciated.

Answer 1

From an elevated powershell session please run: Unregister-ScheduledTask FabricCounters.

This will fix the issue.

Answer 2

I was seeing very similar behaviour and reinstalls of the fabric SDK and runtime, deleting scheduled tasks, deleting the SfDevCluster contents etc. all didn't work.

I was seeing repeated Docker errors in the event log and when I tried uninstalling Docker for Windows SF instantly woke up. I have no idea what the interaction is between the two but worth checking if you have both installed.

Answer 3

For the benefit of searchers, this is the Powershell script I use to fix my local cluster. It was adapted from this issue fix on github.

#
# WARNING: YOU MUST STOP 'SERVICE FABRIC HOST SERVICE' IN SERVICES FIRST
#          IF THE APPLICATION IS STUCK IN 'STARTING', RESTART YOUR MACHINE
# 
# This script will completely reset the local cluster
# 

Remove-Item 'C:\SfDevCluster' -Recurse -Force -ErrorAction Stop 

New-Item -ItemType directory -Path 'C:\SfDevCluster'

Set-Location 'C:\Program Files\Microsoft SDKs\Service Fabric\ClusterSetup'

./DevClusterSetup.ps1 -PathToClusterDataRoot 'C:\SfDevCluster\Data' -PathToClusterLogRoot 'C:\SfDevCluster\Log'

Answer 4

C:\ drive was full for me. Easy thing to overlook.

Make sure your hard disks aren't full... that was stopping mine from starting. Immediately after clearing up some logs, starts up right away.

Answer 5

It seems like some cluster related settings have gone into an inconsistent state on your machine. This will require looking at Service Fabric traces and figure out the actual cause. I am an engineer on the Service Fabric team. I can help you out if you can email me the Service Fabric traces (from logs folder) at harahma[at]microsoft[dot]com.

If you are familiar with logging support tickets on Azure, I would suggest you do that too so we can track this issue to resolution. In the meantime I will continue to work on this to see how we can unblock you.

Answer 6

I know this is an old and question, but maybe my pain can help someone else out.

You'll get a similar error if the Windows Firewall service is not running when Service Fabric attempts to start.

Check and make sure that the Windows Firewall service is set for Automatic and is running.