In order to test and evaluate SF for production use, I created one (single-machine) test cluster on a production machine with three nodes, which worked fine. However, I failed to create a multi-machine cluster with three nodes.
I followed these instructions: https://azure.microsoft.com/en-us/documentation/articles/service-fabric-cluster-creation-for-windows-server/
All machines:
- are on the same (secure) network with the following IPs: 10.0.10.12, 10.0.11.12, 10.0.12.12.
- are virtual and were created freshly from the same image.
- are not part of a domain. The setup is done with the administrator account with the same password on all machines.
- using Windows Server 2012 R2 with PowerShell 4.0.
- have disabled firewalls (public and private).
This is the clusterConfig.json:
{
"name":"SampleCluster",
"clusterManifestVersion":"1.0.0",
"apiVersion":"2015-01-01-alpha",
"nodes":[
{
"nodeName":"vm1",
"iPAddress":"10.0.10.12",
"nodeTypeRef":"NodeType0",
"faultDomain":"fd:/dc1/fd1",
"upgradeDomain":"UD0"
},
{
"nodeName":"vm2",
"iPAddress":"10.0.11.12",
"nodeTypeRef":"NodeType0",
"faultDomain":"fd:/dc1/fd2",
"upgradeDomain":"UD1"
},
{
"nodeName":"vm3",
"iPAddress":"10.0.12.12",
"nodeTypeRef":"NodeType0",
"faultDomain":"fd:/dc1/fd3",
"upgradeDomain":"UD2"
}
],
"diagnosticsFileShare": {
"etlReadIntervalInMinutes": "5",
"uploadIntervalInMinutes": "10",
"dataDeletionAgeInDays": "7",
"etwStoreConnectionString": "file:c:\\ProgramData\\SF\\FileshareETW",
"crashDumpConnectionString": "file:c:\\ProgramData\\SF\\FileshareCrashDump",
"perfCtrConnectionString": "file:c:\\ProgramData\\SF\\FilesharePerfCtr"
},
"properties":{
"reliabilityLevel": "Bronze",
"nodeTypes": [
{
"name": "NodeType0",
"clientConnectionEndpointPort": "19000",
"clusterConnectionEndpoint": "19001",
"httpGatewayEndpointPort": "19080",
"applicationPorts": {
"startPort": "20001",
"endPort": "20031"
},
"ephemeralPorts": {
"startPort": "20032",
"endPort": "20062"
},
"isPrimary": true
}
],
"fabricSettings": [
{
"name": "Setup",
"parameters": [
{
"name": "FabricDataRoot",
"value": "C:\\ProgramData\\SF"
},
{
"name": "FabricLogRoot",
"value": "C:\\ProgramData\\SF\\Log"
}
]
}
]
}
}
When I start the cluster setup from one of the machines (it was 10.0.10.12), this is written to the PowerShell console:
Cab extracted.
Creating Service Fabric Cluster...
If it's taking too long, please check in Task Manager details and see if Fabric.exe for each node is running. If not, p
lease look at: 1. traces in DeploymentTraces directory and 2. traces in FabricLogRoot configured in ClusterConfig.json.
Trace folder doesn't exist. Creating trace folder: C:\copy\DeploymentTraces
Verifying remote procedure call access against cluster machines.
Processing and validating cluster config.
Creating FabricSettingsMetadata from C:\copy\ServiceFabricPackage\bin\Fabric\Fabric.Code\Configurations.csv
Configuring nodes.
Copying installer & package to all machines.
Configuring machine 10.0.10.12
Configuring machine 10.0.11.12
Here the setup remains for a few minutes. Then a timeout occurs:
Timed out waiting for Installer Service to start for machine 10.0.11.12.
CreateCluster Error: System.InvalidOperationException: Cannot start service FabricInstallerSvc on computer '10.0.11.12'.
---> System.ComponentModel.Win32Exception: The system cannot find the file specified
--- End of inner exception stack trace ---
at System.ServiceProcess.ServiceController.Start(String[] args)
at System.Fabric.DeploymentManager.StartAndValidateInstallerServiceCompletion(String machineName, ServiceController i
nstallerSvc)
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Fabric.DeploymentManager.<CreateClusterAsyncInternal>d__a.MoveNext()
Errors occurred during cluster creation.
CreateCluster Exception 0: System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException
: Cannot start service FabricInstallerSvc on computer '10.0.11.12'. ---> System.ComponentModel.Win32Exception: The syste
m cannot find the file specified
--- End of inner exception stack trace ---
at System.ServiceProcess.ServiceController.Start(String[] args)
at System.Fabric.DeploymentManager.StartAndValidateInstallerServiceCompletion(String machineName, ServiceController i
nstallerSvc)
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Fabric.DeploymentManager.<CreateClusterAsyncInternal>d__a.MoveNext()
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.InvalidOperationException: Cannot start service FabricInstallerSvc on computer '10.0.11
.12'. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
--- End of inner exception stack trace ---
at System.ServiceProcess.ServiceController.Start(String[] args)
at System.Fabric.DeploymentManager.StartAndValidateInstallerServiceCompletion(String machineName, ServiceController i
nstallerSvc)
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Fabric.DeploymentManager.<CreateClusterAsyncInternal>d__a.MoveNext()<---
When I check the Services on the particular machine (10.0.11.12), I found the Service Fabric Installer Service in the list, but which is not running. Further I can find an error in the Windows Event Log showing this (which is in line with the error message above):
The Service Fabric Installer Service service failed to start due to the following error:
The system cannot find the file specified.
On the particular machine, I located the following log file: C:\ProgramData\SF\Log\traces\FabricInstallerService_5.1.150.9590_131111077992093094.trace. It contains this:
2016-06-22 22:23:19.224,Info ,708,General.FabricInstallerServiceImpl,FabricInstallerService starting ...
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b4480bcf0,Attempting to attach child AsyncOperation 3b4480bdf0.
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b4480bdf0,Calling OnStart
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b4480bdf0,Attempting to attach child AsyncOperation 3b4480c9b0.
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b4480c9b0,Calling OnStart
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b4480c9b0,FinishComplete called with S_OK
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b44811270,Attempting to attach child AsyncOperation 3b44811630.
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b44811630,Calling OnStart
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b4480bdf0,FinishComplete called with S_OK
2016-06-22 22:23:19.224,Noise ,1652,General.FabricInstallerServiceImpl,FabricUpgradeManager open returned S_OK
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b4480bcf0,Detaching child AsyncOperation 3b4480bdf0.
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b4480bdf0,Detaching child AsyncOperation 3b4480c9b0.
2016-06-22 22:23:19.224,Info ,1652,FabricInstallerService.FabricUpgradeManager,Upgrade started with FabricDataRoot:C:\ProgramData\SF, FabricLogRoot:C:\ProgramData\SF\Log, FabricCodePath:C:\Program Files\Microsoft Service Fabric\bin\fabric\fabric.code, FabricRoot:C:\Program Files\Microsoft Service Fabric, TargetInformationFilePath:C:\ProgramData\SF\TargetInformation.xml, TargetInformationDescription:TargetInformationFileDescription { CurrentInstallation = WindowsFabricDeploymentDescription { IsValid = true, InstanceId = 0, MSILocation = , ClusterManifestLocation = , InfrastructureManifestLocation = , NodeName = , UpgradeEntryPointExe = , UpgradeEntryPointExeParameters = , UndoUpgradeEntryPointExe = FabricSetup.exe, UndoUpgradeEntryPointExeParameters = /operation:Uninstall , }TargetInstallation = WindowsFabricDeploymentDescription { IsValid = false, InstanceId = , MSILocation = , ClusterManifestLocation = , InfrastructureManifestLocation = , NodeName = , UpgradeEntryPointExe = , UpgradeEntryPointExeParameters = , UndoUpgradeEntryPointExe = , UndoUpgradeEntryPointExeParameters = , }}
2016-06-22 22:23:19.224,Info ,1652,FabricInstallerService.FabricUpgradeManager,Stopping fabric host
2016-06-22 22:23:19.224,Info ,1652,FabricInstallerService.FabricUpgradeManager,Error 0x80070424 while waiting for fabric host service to stop.
2016-06-22 22:23:19.224,Error ,1652,FabricInstallerService.FabricUpgradeManager,Unable to stop fabric host service; error
2016-06-22 22:23:19.224,Error ,1652,FabricInstallerService.FabricUpgradeManager,Error E_FAIL while trying to stop fabric host service
2016-06-22 22:23:19.224,Noise ,1652,General.AsyncOperation@3b44811630,FinishComplete called with E_FAIL
2016-06-22 22:23:19.224,Warning ,1652,FabricInstallerService.FabricUpgradeManager,Upgrade finished with error E_FAIL
2016-06-22 22:23:19.224,Info ,1636,General.FabricInstallerServiceImpl,service stopping (shutdown = false) ...
2016-06-22 22:23:19.224,Info ,1636,General.FabricInstallerServiceImpl,Stop FabricUpgradeManager called
2016-06-22 22:23:19.240,Info ,2472,General.FabricInstallerServiceImpl,Close FabricUpgradeManager, with timeout 5:00.000
2016-06-22 22:23:19.240,Noise ,2472,General.AsyncOperation@3b4480be00,Attempting to attach child AsyncOperation 3b4480c4d0.
2016-06-22 22:23:19.240,Noise ,2472,General.AsyncOperation@3b4480c4d0,Calling OnStart
2016-06-22 22:23:19.240,Noise ,2472,General.AsyncOperation@3b4480c4d0,Attempting to attach child AsyncOperation 3b4480c5d0.
2016-06-22 22:23:19.240,Noise ,2472,General.AsyncOperation@3b4480c5d0,Calling OnStart
2016-06-22 22:23:19.240,Noise ,2472,General.AsyncOperation@3b4480c5d0,FinishComplete called with S_OK
2016-06-22 22:23:19.240,Noise ,2472,General.AsyncOperation@3b4480c4d0,FinishComplete called with S_OK
2016-06-22 22:23:19.240,Noise ,2472,General.FabricInstallerServiceImpl,Close FabricUpgradeManager returned S_OK
2016-06-22 22:23:19.240,Noise ,2472,General.AsyncOperation@3b4480be00,Detaching child AsyncOperation 3b4480c4d0.
2016-06-22 22:23:19.240,Noise ,2472,General.AsyncOperation@3b4480c4d0,Detaching child AsyncOperation 3b4480c5d0.
This is the point where I am stuck. My thoughts are:
- Communication and accessibility between the machines seem to be OK, since the setup files were copied, and the setup process started.
- The Service Fabric Installer Service seem to play an important role here.
- It seems that the Service Fabric Installer Service works properly on the machine where I started the setup process, but on the remote machines they fail.
Any ideas? Thanks.
