0
votes

service fabric cluster communication fail when using my domainname.com certificate. I'm getting this error using the portal and/or using ARM template:

Failed to communicate with the cluster (get cluster health: Client certificate required).

Template (variables were removed):

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
    "contentVersion": "1.0.0.0",
    "parameters": {
        
    },
    "variables": {
        
    "resources": [
        {
            "apiVersion": "[variables('storageApiVersion')]",
            "type": "Microsoft.Storage/storageAccounts",
            "name": "[parameters('supportLogStorageAccountName')]",
            "location": "[parameters('computeLocation')]",
            "dependsOn": [],
            "properties": {},
            "kind": "Storage",
            "sku": {
                "name": "[parameters('supportLogStorageAccountType')]"
            },
            "tags": {
                "resourceType": "Service Fabric",
                "clusterName": "[parameters('clusterName')]"
            }
        },
        {
            "apiVersion": "[variables('storageApiVersion')]",
            "type": "Microsoft.Storage/storageAccounts",
            "name": "[parameters('applicationDiagnosticsStorageAccountName')]",
            "location": "[parameters('computeLocation')]",
            "dependsOn": [],
            "properties": {},
            "kind": "Storage",
            "sku": {
                "name": "[parameters('applicationDiagnosticsStorageAccountType')]"
            },
            "tags": {
                "resourceType": "Service Fabric",
                "clusterName": "[parameters('clusterName')]"
            }
        },
        {
            "apiVersion": "[variables('vNetApiVersion')]",
            "type": "Microsoft.Network/virtualNetworks",
            "name": "[parameters('virtualNetworkName')]",
            "location": "[parameters('computeLocation')]",
            "properties": {
                "addressSpace": {
                    "addressPrefixes": [
                        "[parameters('addressPrefix')]"
                    ]
                },
                "subnets": [
                    {
                        "name": "[parameters('subnet0Name')]",
                        "properties": {
                            "addressPrefix": "[parameters('subnet0Prefix')]"
                        }
                    }
                ]
            },
            "tags": {
                "resourceType": "Service Fabric",
                "clusterName": "[parameters('clusterName')]"
            }
        },
        {
            "apiVersion": "[variables('publicIPApiVersion')]",
            "type": "Microsoft.Network/publicIPAddresses",
            "name": "[concat(parameters('lbIPName'),'-','0')]",
            "location": "[parameters('computeLocation')]",
            "properties": {
                "dnsSettings": {
                    "domainNameLabel": "[parameters('dnsName')]"
                },
                "publicIPAllocationMethod": "Dynamic"
            },
            "tags": {
                "resourceType": "Service Fabric",
                "clusterName": "[parameters('clusterName')]"
            }
        },
        {
            "apiVersion": "[variables('lbApiVersion')]",
            "type": "Microsoft.Network/loadBalancers",
            "name": "[concat('LB','-', parameters('clusterName'),'-',parameters('vmNodeType0Name'))]",
            "location": "[parameters('computeLocation')]",
            "dependsOn": [
                "[concat('Microsoft.Network/publicIPAddresses/',concat(parameters('lbIPName'),'-','0'))]"
            ],
            "properties": {
                "frontendIPConfigurations": [
                    {
                        "name": "LoadBalancerIPConfig",
                        "properties": {
                            "publicIPAddress": {
                                "id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(parameters('lbIPName'),'-','0'))]"
                            }
                        }
                    }
                ],
                "backendAddressPools": [
                    {
                        "name": "LoadBalancerBEAddressPool",
                        "properties": {}
                    }
                ],
                "loadBalancingRules": [
                    {
                        "name": "LBRule",
                        "properties": {
                            "backendAddressPool": {
                                "id": "[variables('lbPoolID0')]"
                            },
                            "backendPort": "[parameters('nt0fabricTcpGatewayPort')]",
                            "enableFloatingIP": "false",
                            "frontendIPConfiguration": {
                                "id": "[variables('lbIPConfig0')]"
                            },
                            "frontendPort": "[parameters('nt0fabricTcpGatewayPort')]",
                            "idleTimeoutInMinutes": "5",
                            "probe": {
                                "id": "[variables('lbProbeID0')]"
                            },
                            "protocol": "tcp"
                        }
                    },
                    {
                        "name": "LBHttpRule",
                        "properties": {
                            "backendAddressPool": {
                                "id": "[variables('lbPoolID0')]"
                            },
                            "backendPort": "[parameters('nt0fabricHttpGatewayPort')]",
                            "enableFloatingIP": "false",
                            "frontendIPConfiguration": {
                                "id": "[variables('lbIPConfig0')]"
                            },
                            "frontendPort": "[parameters('nt0fabricHttpGatewayPort')]",
                            "idleTimeoutInMinutes": "5",
                            "probe": {
                                "id": "[variables('lbHttpProbeID0')]"
                            },
                            "protocol": "tcp"
                        }
                    }
                ],
                "probes": [
                    {
                        "name": "FabricGatewayProbe",
                        "properties": {
                            "intervalInSeconds": 5,
                            "numberOfProbes": 2,
                            "port": "[parameters('nt0fabricTcpGatewayPort')]",
                            "protocol": "tcp"
                        }
                    },
                    {
                        "name": "FabricHttpGatewayProbe",
                        "properties": {
                            "intervalInSeconds": 5,
                            "numberOfProbes": 2,
                            "port": "[parameters('nt0fabricHttpGatewayPort')]",
                            "protocol": "tcp"
                        }
                    }
                ],
                "inboundNatPools": [
                    {
                        "name": "LoadBalancerBEAddressNatPool",
                        "properties": {
                            "backendPort": "3389",
                            "frontendIPConfiguration": {
                                "id": "[variables('lbIPConfig0')]"
                            },
                            "frontendPortRangeEnd": "4500",
                            "frontendPortRangeStart": "3389",
                            "protocol": "tcp"
                        }
                    }
                ]
            },
            "tags": {
                "resourceType": "Service Fabric",
                "clusterName": "[parameters('clusterName')]"
            }
        },
        {
            "apiVersion": "[variables('storageApiVersion')]",
            "type": "Microsoft.Storage/storageAccounts",
            "name": "[variables('uniqueStringArray0')[copyIndex()]]",
            "location": "[parameters('computeLocation')]",
            "dependsOn": [],
            "properties": {},
            "copy": {
                "name": "storageLoop",
                "count": 5
            },
            "kind": "Storage",
            "sku": {
                "name": "[parameters('storageAccountType')]"
            },
            "tags": {
                "resourceType": "Service Fabric",
                "clusterName": "[parameters('clusterName')]"
            }
        },
        {
            "apiVersion": "[variables('vmssApiVersion')]",
            "type": "Microsoft.Compute/virtualMachineScaleSets",
            "name": "[parameters('vmNodeType0Name')]",
            "location": "[parameters('computeLocation')]",
            "dependsOn": [
                "[concat('Microsoft.Network/virtualNetworks/', parameters('virtualNetworkName'))]",
                "[concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[0])]",
                "[concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[1])]",
                "[concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[2])]",
                "[concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[3])]",
                "[concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[4])]",
                "[concat('Microsoft.Network/loadBalancers/', concat('LB','-', parameters('clusterName'),'-',parameters('vmNodeType0Name')))]",
                "[concat('Microsoft.Storage/storageAccounts/', parameters('supportLogStorageAccountName'))]",
                "[concat('Microsoft.Storage/storageAccounts/', parameters('applicationDiagnosticsStorageAccountName'))]"
            ],
            "properties": {
                "overprovision": "[parameters('overProvision')]",
                "upgradePolicy": {
                    "mode": "Automatic"
                },
                "virtualMachineProfile": {
                    "extensionProfile": {
                        "extensions": [
                            {
                                "name": "[concat(parameters('vmNodeType0Name'),'_ServiceFabricNode')]",
                                "properties": {
                                    "type": "ServiceFabricNode",
                                    "autoUpgradeMinorVersion": false,
                                    "protectedSettings": {
                                        "StorageAccountKey1": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('supportLogStorageAccountName')),'2015-05-01-preview').key1]",
                                        "StorageAccountKey2": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('supportLogStorageAccountName')),'2015-05-01-preview').key2]"
                                    },
                                    "publisher": "Microsoft.Azure.ServiceFabric",
                                    "settings": {
                                        "clusterEndpoint": "[reference(parameters('clusterName')).clusterEndpoint]",
                                        "nodeTypeRef": "[parameters('vmNodeType0Name')]",
                                        "dataPath": "D:\\\\SvcFab",
                                        "durabilityLevel": "Bronze",
                                        "certificate": {
                                            "thumbprint": "[parameters('certificateThumbprint')]",
                                            "x509StoreName": "[parameters('certificateStoreValue')]"
                                        }
                                    },
                                    "typeHandlerVersion": "1.0"
                                }
                            },
                            {
                                "name": "[concat('VMDiagnosticsVmExt','_vmNodeType0Name')]",
                                "properties": {
                                    "type": "IaaSDiagnostics",
                                    "autoUpgradeMinorVersion": true,
                                    "protectedSettings": {
                                        "storageAccountName": "[parameters('applicationDiagnosticsStorageAccountName')]",
                                        "storageAccountKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('applicationDiagnosticsStorageAccountName')),'2015-05-01-preview').key1]",
                                        "storageAccountEndPoint": "https://core.windows.net/"
                                    },
                                    "publisher": "Microsoft.Azure.Diagnostics",
                                    "settings": {
                                        "WadCfg": {
                                            "DiagnosticMonitorConfiguration": {
                                                "overallQuotaInMB": "50000",
                                                "EtwProviders": {
                                                    "EtwEventSourceProviderConfiguration": [
                                                        {
                                                            "provider": "Microsoft-ServiceFabric-Actors",
                                                            "scheduledTransferKeywordFilter": "1",
                                                            "scheduledTransferPeriod": "PT5M",
                                                            "DefaultEvents": {
                                                                "eventDestination": "ServiceFabricReliableActorEventTable"
                                                            }
                                                        },
                                                        {
                                                            "provider": "Microsoft-ServiceFabric-Services",
                                                            "scheduledTransferPeriod": "PT5M",
                                                            "DefaultEvents": {
                                                                "eventDestination": "ServiceFabricReliableServiceEventTable"
                                                            }
                                                        }
                                                    ],
                                                    "EtwManifestProviderConfiguration": [
                                                        {
                                                            "provider": "cbd93bc2-71e5-4566-b3a7-595d8eeca6e8",
                                                            "scheduledTransferLogLevelFilter": "Information",
                                                            "scheduledTransferKeywordFilter": "4611686018427387904",
                                                            "scheduledTransferPeriod": "PT5M",
                                                            "DefaultEvents": {
                                                                "eventDestination": "ServiceFabricSystemEventTable"
                                                            }
                                                        }
                                                    ]
                                                }
                                            }
                                        },
                                        "StorageAccount": "[parameters('applicationDiagnosticsStorageAccountName')]"
                                    },
                                    "typeHandlerVersion": "1.5"
                                }
                            }
                        ]
                    },
                    "networkProfile": {
                        "networkInterfaceConfigurations": [
                            {
                                "name": "[concat(parameters('nicName'), '-0')]",
                                "properties": {
                                    "ipConfigurations": [
                                        {
                                            "name": "[concat(parameters('nicName'),'-',0)]",
                                            "properties": {
                                                "loadBalancerBackendAddressPools": [
                                                    {
                                                        "id": "[variables('lbPoolID0')]"
                                                    }
                                                ],
                                                "loadBalancerInboundNatPools": [
                                                    {
                                                        "id": "[variables('lbNatPoolID0')]"
                                                    }
                                                ],
                                                "subnet": {
                                                    "id": "[variables('subnet0Ref')]"
                                                }
                                            }
                                        }
                                    ],
                                    "primary": true
                                }
                            }
                        ]
                    },
                    "osProfile": {
                        "adminPassword": "[parameters('adminPassword')]",
                        "adminUsername": "[parameters('adminUsername')]",
                        "computernamePrefix": "[parameters('vmNodeType0Name')]",
                        "secrets": [
                            {
                                "sourceVault": {
                                    "id": "[parameters('sourceVaultValue')]"
                                },
                                "vaultCertificates": [
                                    {
                                        "certificateStore": "[parameters('certificateStoreValue')]",
                                        "certificateUrl": "[parameters('certificateUrlValue')]"
                                    }
                                ]
                            }
                        ]
                    },
                    "storageProfile": {
                        "imageReference": {
                            "publisher": "[parameters('vmImagePublisher')]",
                            "offer": "[parameters('vmImageOffer')]",
                            "sku": "[parameters('vmImageSku')]",
                            "version": "[parameters('vmImageVersion')]"
                        },
                        "osDisk": {
                            "vhdContainers": [
                                "[concat(reference(concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[0]), variables('storageApiVersion')).primaryEndpoints.blob, parameters('vmStorageAccountContainerName'))]",
                                "[concat(reference(concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[1]), variables('storageApiVersion')).primaryEndpoints.blob, parameters('vmStorageAccountContainerName'))]",
                                "[concat(reference(concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[2]), variables('storageApiVersion')).primaryEndpoints.blob, parameters('vmStorageAccountContainerName'))]",
                                "[concat(reference(concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[3]), variables('storageApiVersion')).primaryEndpoints.blob, parameters('vmStorageAccountContainerName'))]",
                                "[concat(reference(concat('Microsoft.Storage/storageAccounts/', variables('uniqueStringArray0')[4]), variables('storageApiVersion')).primaryEndpoints.blob, parameters('vmStorageAccountContainerName'))]"
                            ],
                            "name": "vmssosdisk",
                            "caching": "ReadOnly",
                            "createOption": "FromImage"
                        }
                    }
                }
            },
            "sku": {
                "name": "[parameters('vmNodeType0Size')]",
                "capacity": "[parameters('nt0InstanceCount')]",
                "tier": "Standard"
            },
            "tags": {
                "resourceType": "Service Fabric",
                "clusterName": "[parameters('clusterName')]"
            }
        },
        {
            "apiVersion": "2016-09-01",
            "type": "Microsoft.ServiceFabric/clusters",
            "name": "[parameters('clusterName')]",
            "location": "[parameters('clusterLocation')]",
            "dependsOn": [
                "[concat('Microsoft.Storage/storageAccounts/', parameters('supportLogStorageAccountName'))]"
            ],
            "properties": {
                "azureActiveDirectory": {
                    "clientApplication": "55dae335-8436-4900-8f33-abbff16e8036",
                    "clusterApplication": "d0392358-9fa2-4f9a-aa2a-101859e31c34",
                    "tenantId": "c8656f45-daf5-42c1-9b29-ac27d3e63bf3"
                },
                "certificate": {
                    "thumbprint": "[parameters('certificateThumbprint')]",
                    "x509StoreName": "[parameters('certificateStoreValue')]"
                },
                "clientCertificateCommonNames": [],
                "clientCertificateThumbprints": [
                    {
                        "certificateThumbprint": "C5EFB021F5D8BA8966B43B523B2A6BF8EE8202C5",
                        "isAdmin": true
                    },
                    {
                        "certificateThumbprint": "C5EFB021F5D8BA8966B43B523B2A6BF8EE8202C5",
                        "isAdmin": false
                    }
                ],
                "clusterState": "Default",
                "diagnosticsStorageAccountConfig": {
                    "blobEndpoint": "[reference(concat('Microsoft.Storage/storageAccounts/', parameters('supportLogStorageAccountName')), variables('storageApiVersion')).primaryEndpoints.blob]",
                    "protectedAccountKeyName": "StorageAccountKey1",
                    "queueEndpoint": "[reference(concat('Microsoft.Storage/storageAccounts/', parameters('supportLogStorageAccountName')), variables('storageApiVersion')).primaryEndpoints.queue]",
                    "storageAccountName": "[parameters('supportLogStorageAccountName')]",
                    "tableEndpoint": "[reference(concat('Microsoft.Storage/storageAccounts/', parameters('supportLogStorageAccountName')), variables('storageApiVersion')).primaryEndpoints.table]"
                },
                "fabricSettings": [
                    {
                        "parameters": [
                            {
                                "name": "ClusterProtectionLevel",
                                "value": "[parameters('clusterProtectionLevel')]"
                            }
                        ],
                        "name": "Security"
                    }
                ],
                "managementEndpoint": "[concat('https://',reference(concat(parameters('lbIPName'),'-','0')).dnsSettings.fqdn,':',parameters('nt0fabricHttpGatewayPort'))]",
                "nodeTypes": [
                    {
                        "name": "[parameters('vmNodeType0Name')]",
                        "applicationPorts": {
                            "endPort": "[parameters('nt0applicationEndPort')]",
                            "startPort": "[parameters('nt0applicationStartPort')]"
                        },
                        "clientConnectionEndpointPort": "[parameters('nt0fabricTcpGatewayPort')]",
                        "durabilityLevel": "Bronze",
                        "ephemeralPorts": {
                            "endPort": "[parameters('nt0ephemeralEndPort')]",
                            "startPort": "[parameters('nt0ephemeralStartPort')]"
                        },
                        "httpGatewayEndpointPort": "[parameters('nt0fabricHttpGatewayPort')]",
                        "isPrimary": true,
                        "vmInstanceCount": "[parameters('nt0InstanceCount')]"
                    }
                ],
                "provisioningState": "Default",
                "security": {
                    "metadata": "The Credential type X509 indicates this is cluster is secured using X509 Certificates.",
                    "ClusterCredentialType": "X509",
                    "ServerCredentialType": "X509",
                    "CertificateInformation": {
                        "ClusterCertificate": {
                            "Thumbprint": "[parameters('certificateThumbprint')]",                            
                            "X509StoreName": "My"
                        },
                        "ServerCertificate": {
                            "Thumbprint": "[parameters('certificateThumbprint')]",                            
                            "X509StoreName": "My"
                        },
                        "ClientCertificateThumbprints": [
                            {
                                "CertificateThumbprint": "[parameters('certificateThumbprint')]",
                                "IsAdmin": false
                            },
                            {
                                "CertificateThumbprint": "[parameters('certificateThumbprint')]",
                                "IsAdmin": true
                            }
                        ]
                    }
                },
                "reliabilityLevel": "Bronze",
                "upgradeMode": "Automatic",
                "vmImage": "Windows"
            },
            "tags": {
                "resourceType": "Service Fabric",
                "clusterName": "[parameters('clusterName')]"
            }
        }
    ],
    "outputs": {
        "clusterProperties": {
            "value": "[reference(parameters('clusterName'))]",
            "type": "object"
        }
    }
}

hint?

enter image description here

2

2 Answers

2
votes

It might be as simple as the client certificate is not imported (installed) in the machine from where you are trying to access the explorer. When you open the Service Fabric explorer, it might pop up to select one of the certificates installed in your machine. You have to choose the client certificate at this point.

I compared the ARM template with my (working) template and it looks OK. No obvious issues.

0
votes

Your ARM template shows you are missing the client certificate within the VMSS OS Profile, which means it will not be installed to each of the nodes during deployment then it is missing when Service Fabric attempts to use it.

Here is an example from one of my ARM templates



    "osProfile": {
                "adminUsername": "[parameters('adminUsername')]",
                "adminPassword": "[parameters('adminPassword')]",
                "computernamePrefix": "[parameters('vmNodeType0Name')]",
                "secrets": [
                                {
                                    "sourceVault": {
                                        "id": "[parameters('sourceVault')]"
                                    },
                                    "vaultCertificates": [
                                        {
                                            "certificateStore": "My",
                                            "certificateUrl": "[parameters('clusterCertificateUrl')]"
                                        },
                                        {
                                            "certificateStore": "My",
                                            "certificateUrl": "[parameters('adminCertificateUrl')]"
                                        }
                                    ]
                                }
                            ]
              },

EDIT:

Where you have two client certificates, I would expect your OS Profile to have three vault certificates for installation on deployment