Partition is below target replica or instance count

Question

When attempting to publish a Service Fabric application to a local cluster, the cluster fails to load the application stating the error in the title. The stack trace points me to an exception line in OwinCommunicationListener.cs:

        try
        {
            this.eventSource.LogInfo("Starting web server on " + this.listeningAddress);

            this.webApp = WebApp.Start(this.listeningAddress, appBuilder => this.startup.Invoke(appBuilder));

            this.eventSource.LogInfo("Listening on " + this.publishAddress);

            return Task.FromResult(this.publishAddress);
        }
        catch (Exception ex)
        {
            var logString = $"Web server failed to open endpoint {endpointName}. {ex.ToString()}";
            this.eventSource.LogFatal(logString);

            this.StopWebServer();

            throw ex; // points to this line from cluster manager
        }

I am unable to inspect the exception thrown, but there is no useful exception information other than a TargetInvocationException with a stack trace to the line noted above. Why won't this application load on my local cluster?

Is this service an instance? I mean, how many replicas does it have? Looks like it needs several replicas. Probably the exception is about owin unable to start service on a specific port because each replica would do the same and port should be already taken by the first replica, so every next replica will fail. — cassandrad
Since you call 'throw ex;' the exception you are unable to inspect is 'ex' itself, with a new stacktrace. You could put a breakpoint higher in the catch block and examine 'ex'. Likely WebApp.Start is failing. — LoekD
Examining the exception is not possible in any circumstance, I've tried every way possible — Dagrooms
My application parameters are as follows: <Parameter Name="DataService_PartitionCount" Value="1" /> <Parameter Name="DataService_MinReplicaSetSize" Value="2" /> <Parameter Name="DataService_TargetReplicaSetSize" Value="3" /> <Parameter Name="Webservice_InstanceCount" Value="1" /> — Dagrooms
Well it works now. Literally no variable in the equation has changed, it just worked the next morning. Service Fabric is getting a little Kafkaesque. — Dagrooms

Vaclav Turecek Vaclav Turecek · Accepted Answer · 2016-08-10T20:48:09

It's hard to say without an actual exception message or stack trace but judging by the location from which the exception was thrown and the fact that the problem resolved itself the next morning, the most likely and most common cause of this is that the port you were trying to use to open the web listener was taken by some other process at the time, and the next morning the port was free again. This, by the way, isn't really specific to Service Fabric. You're just trying to open a socket on a port that was taken by someone else.

I'm honestly more curious about why you couldn't inspect the exception. I can think of three things off the top of my head to help with that:

Use "throw" instead of "throw ex" so you don't reset the stack trace.
Look at your logs. It looks like you're writing out an ETW event in your catch block. What did it say?
Use the Visual Studio debugger: Simply set a breakpoint in the catch block and start the application with debugging by pressing F5.

Partition is below target replica or instance count

1 Answers