Service Fabric Multiple service instances with config override

Question

Our service fabric application includes a stateless service that exposes an HTTP endpoint through OwinCommunicationListener.

The ServiceManifest.Xml for this service specifies the service endpoint <Endpoint Name="ServiceEndpoint" Type="Input" Protocol="http" Port="8090" />

The stateless service can then be accessed via a browser on http://localhost:8090/

What we are trying to do is instantiate multiple instances of this service on different endpoints in the same Service Fabric application through the ApplicationManifest.

The ServiceManifestImport imports our service package and allows configuration overrides at the application level. We're not able to override the ServiceEndpoint this way, only the values in Settings.xml

<ServiceManifestImport>
  <ServiceManifestRef ServiceManifestName="FooServicePkg" ServiceManifestVersion="1.0.0" >
    <ConfigOverrides Name="Config">
      <Settings>
        <SectionName Name="MySettings">
        <Parameter Name="MySetting" Value="SomeValue">
      </Settings>
    </ConfigOverrides>
</ServiceManifestImport>

We can create named instances of the service by specifying multiple Service nodes under DefaultServices

<DefaultServices>
  <Service Name="FooInstanceA">
    <StatelessService ServiceTypeName="FooType" InstanceCount="1" />
      <SingletonPartition />
    </StatelessService>
  </Service>
  <Service Name="FooInstanceB">
    <StatelessService ServiceTypeName="FooType" InstanceCount="1" />
      <SingletonPartition />
    </StatelessService>
  </Service>
</DefaultServices>

Is it possible to specify configuration overrides per instance of a service through configuration?

I tried to make the service instances listen on a specific port by using their service name to work out which port so FooInstanceA listens on port 8090 and FooInstanceB listens on 8091.

Clearly Service Fabric is doing some magic during deployment because when FooInstanceB listens on a port other than the one specified on the ServiceEndpoint configuration the service is not accessible.

The first reason is the DACL is not set on the endpoint, this is resolved by running;

netsh http add urlacl http://+:8091/ user=everyone listen=yes

This allows the services to come up and show healthy in the Service Fabric Explorer, however the FooInstanceB is responding with an HTTP 503 error when we access with http://localhost:8091/

How can we get the service instances listening on different ports?

I hope that's clear, thank you.

Answer 1

Not a lot of great options to accomplish this. Here are some ideas:

1. Create multiple application instances instead of multiple services of the same type within an app. That would allow you to use the app parameters to configure the behavior of the particular service.
2. Create multiple config packages within your service type. Each config package would be intended for one of the service instances. Determining which config package a service instance is assigned to would need to be dynamic, maybe based on the service instance's name? Not a great option, of course, since it couples the service definition with the number of instances that will be created.
3. A custom configuration implementation. Maybe have your service expose an endpoint that allows you to configure it post-deployment. Or have the service call some other management service that provides its configuration at activation time.

Answer 2

You can also let Service Fabric assign the ports automatically and then use the reverse proxy that comes with Service Fabric.

Answer 3

I think its a very good idea to have a url rewriting service in front of a SF cluster .

This can give you multiple end points and do url rewriting , firewalls/ black listing , https etc .

Other way is to package the service as a lib or in Nuget and create the N services you need with different service manifests.

Answer 4

I've managed to do this using ApplicationParameter xml files and choose the right one during deployment in VSTS. This, for example, is my Cloud.xml which I use for deployment to my test environment. The trick is the Name parameter on the second line. I've specified the port using this documentation: https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-how-to-specify-port-number-using-parameters

<?xml version="1.0" encoding="utf-8"?>
<Application xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="fabric:/DotNetCoreTest" xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Parameters>
    <Parameter Name="Web1_InstanceCount" Value="2" />
    <Parameter Name="ServiceEndpoint_PortNumber" Value="8909" />
  </Parameters>
</Application>

This is the xml file for Staging: Staging.xml

<?xml version="1.0" encoding="utf-8"?>
<Application xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="fabric:/DotNetCoreStaging" xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Parameters>
    <Parameter Name="Web1_InstanceCount" Value="2" />
    <Parameter Name="ServiceEndpoint_PortNumber" Value="8910" />
  </Parameters>
</Application>

Using the Traefik Reverse Proxy as described here I can use a hostname to reach my service. In VSTS I'm using a tokenizer to replace both the host name in the ServiceManifest and replacing the ApplicationTypeName in the ApplicationManifest during deployment.

This is my ServiceManifest:

<ServiceTypes>
    <!-- This is the name of your ServiceType. 
         This name must match the string used in RegisterServiceType call in Program.cs. -->
    <StatelessServiceType ServiceTypeName="Web1Type">
      <Extensions>
        <Extension Name="Traefik">
          <Labels xmlns="http://schemas.microsoft.com/2015/03/fabact-no-schema">
            <Label Key="traefik.frontend.rule.hostname">Host: #{HostName}#</Label>
            <Label Key="traefik.expose">true</Label>
            <Label Key="traefik.frontend.passHostHeader">true</Label>
          </Labels>
        </Extension>
      </Extensions>
    </StatelessServiceType>
  </ServiceTypes>