Stateless Web API service with HTTPS endpoint throws health state error

Question

I want a https endpoint for my in a local service fabric (GA version) cluster hosted stateless Web API service. After achieving that, I want to deploy my cluster in Azure.

I followed the steps in the "Secure a Service Fabric cluster" article of the service fabric documentation and created a self-signed certificate and uploaded it to my key vault. I also imported my certificate to my machine's "trusted people" store with the Import-PfxCertificate commands on step 2.5.

AddCertToKeyVault:

Invoke-AddCertToKeyVault -SubscriptionId <Id> -ResourceGroupName 'ResourceGroupName' -Location 'West Europe' -VaultName 'VaultName' -CertificateName 'TestCert' -Password '****' -CreateSelfSignedCertificate -DnsName 'www.<clustername>.westeurope.cloudapp.azure.com' -OutputPath 'C:\MyCertificates'

Now I adjusted the ServiceManifest.xml, ApplicationManifest.xml (like in RunAs: Run a Service Fabric application with different security permissions) and my OwinCommunicationListener.cs:

ServiceManifest.xml (MasterDataServiceWebApi):

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="MasterDataServiceWebApiPkg"
                 Version="1.0.0"
                 xmlns="http://schemas.microsoft.com/2011/01/fabric"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
    <StatelessServiceType ServiceTypeName="MasterDataServiceWebApiType" />
  </ServiceTypes>

  <CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <ExeHost>
        <Program>MasterDataServiceWebApi.exe</Program>
      </ExeHost>
    </EntryPoint>
  </CodePackage>

  <ConfigPackage Name="Config" Version="1.0.0" />

  <Resources>
    <Endpoints>
      <Endpoint Name="ServiceEndpoint" Type="Input" Protocol="https" Port="5030" CertificateRef="TestCert"/>
    </Endpoints>
  </Resources>
</ServiceManifest>

ApplicationManifest:

<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="exCHANGETestCluster2Type" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
   <Parameters>
      <Parameter Name="MasterDataServiceWebApi_InstanceCount" DefaultValue="-1" />
   </Parameters>
   <ServiceManifestImport>
      <ServiceManifestRef ServiceManifestName="MasterDataServiceWebApiPkg" ServiceManifestVersion="1.0.0" />
      <ConfigOverrides />
      <Policies>
         <EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="TestCert" />
      </Policies>
   </ServiceManifestImport>
   <DefaultServices>
      <Service Name="MasterDataServiceWebApi">
         <StatelessService ServiceTypeName="MasterDataServiceWebApiType" InstanceCount="[MasterDataServiceWebApi_InstanceCount]">
            <SingletonPartition />
         </StatelessService>
      </Service>
   </DefaultServices>
   <Certificates>
      <EndpointCertificate X509FindValue="<Thumbprint>" Name="TestCert" />
   </Certificates>
</ApplicationManifest>

OwinCommunicationListener.cs:

[...]
public Task<string> OpenAsync(CancellationToken cancellationToken)
    {
      var serviceEndpoint = this.serviceContext.CodePackageActivationContext.GetEndpoint(this.endpointName);
      int port = serviceEndpoint.Port; //NEW!

      if (this.serviceContext is StatefulServiceContext)
      {
        [...]
      }
      else if (this.serviceContext is StatelessServiceContext)
      {
        var protocol = serviceEndpoint.Protocol;

        this.listeningAddress = string.Format(
            CultureInfo.InvariantCulture,
            //"http://+:{0}/{1}",
            "{0}://+:{1}/{2}", //NEW!
            protocol,
            port,
            string.IsNullOrWhiteSpace(this.appRoot)
                ? string.Empty
                : this.appRoot.TrimEnd('/') + '/');
      }
      else
      {
        throw new InvalidOperationException();
      }
[...]

When I deploy the stateless service to my local cluster now, my service fabric explorer reports some very "expressive" errors and I am not able to access my service:

Kind        Health State  Description
=============================================================================
Services    Error         Unhealthy services: 100% (1/1), ServiceType='MasterDataServiceWebApiType', MaxPercentUnhealthyServices=0%.
Service     Error         Unhealthy service: ServiceName='fabric:/sfCluster/MasterDataServiceWebApi', AggregatedHealthState='Error'.
Partitions  Error         Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%.
Partition   Error         Unhealthy partition: PartitionId='e5635b85-3c23-426b-bd12-13ae56796f23', AggregatedHealthState='Error'.
Event       Error         Error event: SourceId='System.FM', Property='State'. Partition is below target replica or instance count.

Visual Studio isn't providing me with any further error details. It's quite the opposite. The stacktrace prints: fabric:/sfCluster/MasterDataServiceWebApi is ready.

What did I miss? Did I configured something wrong?

BTW: After that, I created a new cluster in Azure with my self-signed certificate, but when I try to acess the Service Fabric Explorer of this cluster I have no UI and a blank site..

MegaMax MegaMax · Accepted Answer · 2016-05-09T11:11:52

I have learned, that Service Fabric uses the local machine store for the certificate validation. (https://github.com/Azure-Samples/service-fabric-dotnet-web-reference-app/issues/3)

So I had to import the certificate into my local machine store by this slightly modified powershell-script:

Import-PfxCertificate -Exportable -CertStoreLocation cert:\localMachine\my -FilePath C:\MyCertificates\TestCert.pfx -Password (Read-Host -AsSecureString -Prompt "Enter Certificate Password")

Before that, I imported my certificate into Cert:\CurrentUser\TrustedPeople and Cert:\CurrentUser\My. But the local Service Fabric cluster doesn't look up there.

BTW: I still get a blank site, when I try to access the Service Fabric Explorer of my azure-hosted Service Fabric cluster, that I have secured with the same certification key. I will create another question for this problem. EDIT: Using the Internet Explorer instead of Firefox solved my blank site issue.

Stateless Web API service with HTTPS endpoint throws health state error

1 Answers