Service Fabric Reverse Proxy - 404 - FABRIC_E_SERVICE_DOES_NOT_EXIST

1
votes

Is anyone experiencing intermittent issues (404 - FABRIC_E_SERVICE_DOES_NOT_EXIST) with the Service Fabric Reverse Proxy when using ServiceFabricIntegrationOptions.UseUniqueServiceUrl. The service actually exists. We double checked the "casing". This is not a casing issue. The same request will work on some nodes but fail on other.

We have a 5 nodes service fabric cluster running in Azure (version 5.7.198.9494). We have many stateless services (Web API) built using ASP.NET Core 1.1, each in their own Application.

Because they are only accessed internally from the cluster, each Web API services only have 1 instance and are configured to "UseServiceFabricIntegration" with "ServiceFabricIntegrationOptions.UseUniqueServiceUrl" using Kestrel as the host server. To simplify discovery, we decided to leverage the Reverse Proxy when communicating with those services.

Everything appeared to be working fine until we noticed some intermittent 404 (FABRIC_E_SERVICE_DOES_NOT_EXIST) in the Reverse Proxy log (ETW Microsoft-ServiceFabric:4:0x4000000000000020). After some investigation we found that the same HTTP request to the Reverse Proxy would work on some nodes but not on other. i.e :

  1. Open an RDP session on Node 1;
  2. Open Powershell and run "wget http://localhost:19081/CoreApi.Currency.Application/Currency.Web/?s=USD";
  3. Result: 200 (OK)
  4. Repeat steps 1-2 on Node 2;
  5. Result: 404 (Not Found);
  6. The 404 could occur on 1, 2 or 3 nodes while succeeding on the other nodes;

Redeploying the failing stateless services sometime temporarily fix the issue. Rebooting a "client" node (a node not hosting the service) make the Reverse Proxy of this node fail with a 404. Running the powershell command "Get-ServiceFabricApplicationName -ServiceName fabric:/CoreApi.Currency.Application/Currency.Web" on the failing node will fix the issue until the node is rebooted.

Is anyone experiencing that kind of issue?

A comment from "Vislor" dated Dec 15, 2016 on "https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-reverseproxy" describe a very similar issue.

Thanks in advance.

Update 1: This is an extract of the logs when NOT working (after a reboot):

<Event MSec= "18564.7439" PID="3716" PName=        "" TID="3788" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:386 False  586B @ qsize 0/0B " id="25752f187a0" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="386" isReply="False" localContext="" messageSize="586" messagesToSend="0" bytesToSend="0"/>
<Event MSec= "18564.9152" PID="3752" PName=        "" TID="3604" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:386 False 1 586B NamingGateway:PrefixResolveServiceRequest caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0 " id="2a8e02c01b0" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="386" isReply="False" dispatchCountForThisReceive="1" size="586" actor="NamingGateway" action="PrefixResolveServiceRequest" activityIdid="caf08066-5e0d-46a0-80de-bb83ea1a0fdd" activityIdindex="0"/>
<Event MSec= "18565.0640" PID="3752" PName=        "" TID="3592" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:386 False  717B @ qsize 0/0B " id="2a8e03a8e70" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="386" isReply="False" localContext="" messageSize="717" messagesToSend="0" bytesToSend="0"/>
<Event MSec= "18565.2792" PID="3208" PName=        "" TID="6008" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:386 False 1 650B " id="2d5f393d360" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="386" isReply="False" dispatchCountForThisReceive="1" size="650"/>
<Event MSec= "18565.3366" PID="3208" PName=        "" TID="9260" EventName="NamingGateway" ProviderName="Microsoft-ServiceFabric" FormattedMessage="caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0 Processing request, Action: PrefixResolveServiceRequest, RemainingTime 9,999ms " activityIdid="caf08066-5e0d-46a0-80de-bb83ea1a0fdd" activityIdindex="0" action="PrefixResolveServiceRequest" remainingTimetimespan="9,999"/>
<Event MSec= "18565.4133" PID="3208" PName=        "" TID="9260" EventName="NamingGateway" ProviderName="Microsoft-ServiceFabric" FormattedMessage="[d5bd7fb376fe0f815e01efab2c69b10e:131484874195545265+caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0]: prefix resolve from cache: fabric:/CoreApi.Currency.Application/Currency.Web " id="" type="PrefixResolveServiceAsyncOperation" text="[d5bd7fb376fe0f815e01efab2c69b10e:131484874195545265+caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0]: prefix resolve from cache: fabric:/CoreApi.Currency.Application/Currency.Web"/>
<Event MSec= "18565.4395" PID="3208" PName=        "" TID="9260" EventName="Common" ProviderName="Microsoft-ServiceFabric" FormattedMessage="fabric:/CoreApi.Currency.Application/Currency.Web " contextSequenceId="23,041" scheme="fabric" path="/CoreApi.Currency.Application/Currency.Web" querySeparator="" query="" fragmentSeparator="" fragment=""/>
<Event MSec= "18565.4408" PID="3208" PName=        "" TID="9260" EventName="NamingGateway" ProviderName="Microsoft-ServiceFabric" FormattedMessage="[d5bd7fb376fe0f815e01efab2c69b10e:131484874195545265+caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0]: fetching PSD from Naming: name=23,041 prefix=True " traceId="[d5bd7fb376fe0f815e01efab2c69b10e:131484874195545265+caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0]" contextSequenceId_name="23,041" prefix="True"/>
<Event MSec= "18565.5921" PID="3208" PName=        "" TID="9260" EventName="LeaseLayer" ProviderName="Microsoft-ServiceFabric" FormattedMessage="Get TTL for Leasing application 131,484,874,196,365,091 with request TTL 1,000 " a1="131,484,874,196,365,091" a2="1,000"/>
<Event MSec= "18565.6096" PID="3208" PName=        "" TID="9260" EventName="LeaseAgent" ProviderName="Microsoft-ServiceFabric" FormattedMessage="d5bd7fb376fe0f815e01efab2c69b10e:[email protected]:1026 lease valid for 1,000ms " app="d5bd7fb376fe0f815e01efab2c69b10e:131484874195545265" addr="10.51.2.8:1026" ms="1,000"/>
<Event MSec= "18565.6342" PID="3208" PName=        "" TID="9260" EventName="P2P" ProviderName="Microsoft-ServiceFabric" FormattedMessage="d5bd7fb376fe0f815e01efab2c69b10e send message PrefixResolveRequest to -5,034,481,294,210,871,1187,182,030,848,336,970,117:131,484,109,128,799,478 id e223b95c-5da8-46fa-bf72-d3efd5d072a6:9,858 False (0) " id="PrefixResolveRequest" site="d5bd7fb376fe0f815e01efab2c69b10e" toid_omitZero_high="-5,034,481,294,210,871,118" toidlow="7,182,030,848,336,970,117" toinstance="131,484,109,128,799,478" msgIdid="e223b95c-5da8-46fa-bf72-d3efd5d072a6" msgIdindex="9,858" isReply="False" retry="0"/>
<Event MSec= "18565.6431" PID="3208" PName=        "" TID="9260" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="e223b95c-5da8-46fa-bf72-d3efd5d072a6:9,858 False  990B @ qsize 0/0B " id="2d5d1a19a30" messageid="e223b95c-5da8-46fa-bf72-d3efd5d072a6" messageindex="9,858" isReply="False" localContext="" messageSize="990" messagesToSend="0" bytesToSend="0"/>
<Event MSec= "18566.9032" PID="3208" PName=        "" TID="6008" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="e223b95c-5da8-46fa-bf72-d3efd5d072a6:9,858 True 1 1,431B " id="2d5d1a19a30" messageid="e223b95c-5da8-46fa-bf72-d3efd5d072a6" messageindex="9,858" isReply="True" dispatchCountForThisReceive="1" size="1,431"/>
<Event MSec= "18566.9579" PID="3208" PName=        "" TID="6008" EventName="P2P" ProviderName="Microsoft-ServiceFabric" FormattedMessage="d5bd7fb376fe0f815e01efab2c69b10e receive message RejectFault from -5,034,481,294,210,871,1187,182,030,848,336,970,117:131,484,109,128,799,478 id e223b95c-5da8-46fa-bf72-d3efd5d072a6:9,858 True (0) " id="RejectFault" site="d5bd7fb376fe0f815e01efab2c69b10e" fromid_omitZero_high="-5,034,481,294,210,871,118" fromidlow="7,182,030,848,336,970,117" frominstance="131,484,109,128,799,478" msgIdid="e223b95c-5da8-46fa-bf72-d3efd5d072a6" msgIdindex="9,858" isReply="True" retry="0"/>
<Event MSec= "18567.0017" PID="3208" PName=        "" TID="6008" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:386 True  179B @ qsize 0/0B " id="2d5f393d360" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="386" isReply="True" localContext="" messageSize="179" messagesToSend="0" bytesToSend="0"/>
<Event MSec= "18567.0703" PID="3208" PName=        "" TID="6008" EventName="Routing" ProviderName="Microsoft-ServiceFabric" FormattedMessage="Routing of message e223b95c-5da8-46fa-bf72-d3efd5d072a6:9858(1) failed with FABRIC_E_SERVICE_DOES_NOT_EXIST " id="" type="Complete" text="Routing of message e223b95c-5da8-46fa-bf72-d3efd5d072a6:9858(1) failed with FABRIC_E_SERVICE_DOES_NOT_EXIST"/>
<Event MSec= "18567.0961" PID="3752" PName=        "" TID="3604" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:386 True 1 179B " id="2a8e03a8e70" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="386" isReply="True" dispatchCountForThisReceive="1" size="179"/>
<Event MSec= "18567.1471" PID="3752" PName=        "" TID="3592" EventName="EntreeServiceProxy" ProviderName="Microsoft-ServiceFabric" FormattedMessage="ProcessClientOperation for action &apos;PrefixResolveServiceRequest&apos; encountered error FABRIC_E_SERVICE_DOES_NOT_EXIST. activity id = caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0 " id="[d5bd7fb376fe0f815e01efab2c69b10e:131484874195545265]" type="EntreeServiceProxy.RequestProcessingFailed" text="ProcessClientOperation for action &apos;PrefixResolveServiceRequest&apos; encountered error FABRIC_E_SERVICE_DOES_NOT_EXIST. activity id = caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0"/>
<Event MSec= "18567.1778" PID="3752" PName=        "" TID="3592" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:386 True  181B @ qsize 0/0B " id="2a8e02c01b0" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="386" isReply="True" localContext="" messageSize="181" messagesToSend="0" bytesToSend="0"/>
<Event MSec= "18567.2904" PID="3716" PName=        "" TID= "632" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:386 True 1 181B " id="25752f187a0" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="386" isReply="True" dispatchCountForThisReceive="1" size="181"/>
<Event MSec= "18567.3439" PID="3716" PName=        "" TID= "632" EventName="Client" ProviderName="Microsoft-ServiceFabric" FormattedMessage="caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0: gateway replied with error=FABRIC_E_SERVICE_DOES_NOT_EXIST " id="69361768-d4c7-440c-9d26-f6a9da5784f5" type="ClientConnectionManager" text="caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0: gateway replied with error=FABRIC_E_SERVICE_DOES_NOT_EXIST"/>
<Event MSec= "18567.3516" PID="3716" PName=        "" TID= "632" EventName="Client" ProviderName="Microsoft-ServiceFabric" FormattedMessage="caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0: Prefix resolution failed: fabric:/CoreApi.Currency.Application/Currency.Web error = FABRIC_E_SERVICE_DOES_NOT_EXIST " id="" type="CacheManager.PrefixResolveServiceAsyncOperation" text="caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0: Prefix resolution failed: fabric:/CoreApi.Currency.Application/Currency.Web error = FABRIC_E_SERVICE_DOES_NOT_EXIST"/>
<Event MSec= "18567.3647" PID="3716" PName=        "" TID= "632" EventName="Client" ProviderName="Microsoft-ServiceFabric" FormattedMessage="caf08066-5e0d-46a0-80de-bb83ea1a0fdd:0: error = FABRIC_E_SERVICE_DOES_NOT_EXIST " id="69361768-d4c7-440c-9d26-f6a9da5784f5" activityIdid="caf08066-5e0d-46a0-80de-bb83ea1a0fdd" activityIdindex="0" errorval="FABRIC_E_SERVICE_DOES_NOT_EXIST"/>
<Event MSec= "18567.3737" PID="3716" PName=        "" TID= "632" EventName="ReverseProxy" ProviderName="Microsoft-ServiceFabric" FormattedMessage="caf08066-5e0d-46a0-80de-bb83ea1a0fdd Error while processing request, cannot forward to service: request url = http://localhost:19081/CoreApi.Currency.Application/Currency.Web/?symbols=CADUSD, verb = GET, remote (client) address = ::1, request processing start time = 13:47:12.959333 (18,561.035 MSec), error = FABRIC_E_SERVICE_DOES_NOT_EXIST, message = , phase = ResolveServicePartition,  " traceId="caf08066-5e0d-46a0-80de-bb83ea1a0fdd" requestUrl="http://localhost:19081/CoreApi.Currency.Application/Currency.Web/?symbols=CADUSD" verb="GET" remoteAddress="::1" requestStartTime="13:47:12.959333 (18,561.035 MSec)" errorval="FABRIC_E_SERVICE_DOES_NOT_EXIST" errorMessage="" processRequestPhase="ResolveServicePartition" errorDetails=""/>
<Event MSec= "18567.3772" PID="3716" PName=        "" TID= "632" EventName="ReverseProxy" ProviderName="Microsoft-ServiceFabric" FormattedMessage="caf08066-5e0d-46a0-80de-bb83ea1a0fdd Error while processing request, cannot forward to service: request url = http://localhost:19081/CoreApi.Currency.Application/Currency.Web/?symbols=CADUSD, verb = GET, remote (client) address = ::1, request processing start time = 13:47:12.959333 (18,561.035 MSec), error = FABRIC_E_SERVICE_DOES_NOT_EXIST, message = , phase = ResolveServicePartition,  " traceId="caf08066-5e0d-46a0-80de-bb83ea1a0fdd" requestUrl="http://localhost:19081/CoreApi.Currency.Application/Currency.Web/?symbols=CADUSD" verb="GET" remoteAddress="::1" requestStartTime="13:47:12.959333 (18,561.035 MSec)" errorval="FABRIC_E_SERVICE_DOES_NOT_EXIST" errorMessage="" processRequestPhase="ResolveServicePartition" errorDetails=""/>

And this is and extract of the log from the same node after first resolving the partition manually using ServicePartitionResolver(C#):

<Event MSec= "31146.3672" PID="3716" PName=        "" TID="3724" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:417 False  586B @ qsize 0/0B " id="25752f187a0" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="417" isReply="False" localContext="" messageSize="586" messagesToSend="0" bytesToSend="0"/>
<Event MSec= "31146.5498" PID="3752" PName=        "" TID="3604" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:417 False 1 586B NamingGateway:PrefixResolveServiceRequest ca94c2c8-b69e-4272-851b-ea6183f340dc:0 " id="2a8e02c01b0" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="417" isReply="False" dispatchCountForThisReceive="1" size="586" actor="NamingGateway" action="PrefixResolveServiceRequest" activityIdid="ca94c2c8-b69e-4272-851b-ea6183f340dc" activityIdindex="0"/>
<Event MSec= "31146.7695" PID="3752" PName=        "" TID= "512" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:417 False  717B @ qsize 0/0B " id="2a8e03a8e70" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="417" isReply="False" localContext="" messageSize="717" messagesToSend="0" bytesToSend="0"/>
<Event MSec= "31146.9528" PID="3208" PName=        "" TID="9260" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:417 False 1 650B " id="2d5f393d360" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="417" isReply="False" dispatchCountForThisReceive="1" size="650"/>
<Event MSec= "31147.0243" PID="3208" PName=        "" TID= "924" EventName="NamingGateway" ProviderName="Microsoft-ServiceFabric" FormattedMessage="ca94c2c8-b69e-4272-851b-ea6183f340dc:0 Processing request, Action: PrefixResolveServiceRequest, RemainingTime 9,999ms " activityIdid="ca94c2c8-b69e-4272-851b-ea6183f340dc" activityIdindex="0" action="PrefixResolveServiceRequest" remainingTimetimespan="9,999"/>
<Event MSec= "31147.0999" PID="3208" PName=        "" TID= "924" EventName="NamingGateway" ProviderName="Microsoft-ServiceFabric" FormattedMessage="[d5bd7fb376fe0f815e01efab2c69b10e:131484874195545265+ca94c2c8-b69e-4272-851b-ea6183f340dc:0]: prefix resolve from cache: fabric:/CoreApi.Currency.Application/Currency.Web " id="" type="PrefixResolveServiceAsyncOperation" text="[d5bd7fb376fe0f815e01efab2c69b10e:131484874195545265+ca94c2c8-b69e-4272-851b-ea6183f340dc:0]: prefix resolve from cache: fabric:/CoreApi.Currency.Application/Currency.Web"/>
<Event MSec= "31147.1977" PID="3208" PName=        "" TID= "924" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:417 True  1,271B @ qsize 0/0B " id="2d5f393d360" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="417" isReply="True" localContext="" messageSize="1,271" messagesToSend="0" bytesToSend="0"/>
<Event MSec= "31147.5547" PID="3752" PName=        "" TID="3604" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:417 True 1 1,271B " id="2a8e03a8e70" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="417" isReply="True" dispatchCountForThisReceive="1" size="1,271"/>
<Event MSec= "31147.6057" PID="3752" PName=        "" TID= "512" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:417 True  1,307B @ qsize 0/0B " id="2a8e02c01b0" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="417" isReply="True" localContext="" messageSize="1,307" messagesToSend="0" bytesToSend="0"/>
<Event MSec= "31147.7000" PID="3716" PName=        "" TID="3596" EventName="Transport" ProviderName="Microsoft-ServiceFabric" FormattedMessage="b7e15d5b-5674-43c8-9388-67e011d53e60:417 True 1 1,307B " id="25752f187a0" messageid="b7e15d5b-5674-43c8-9388-67e011d53e60" messageindex="417" isReply="True" dispatchCountForThisReceive="1" size="1,307"/>
<Event MSec= "31147.8414" PID="3716" PName=        "" TID= "632" EventName="Client" ProviderName="Microsoft-ServiceFabric" FormattedMessage="ca94c2c8-b69e-4272-851b-ea6183f340dc:0: updated prefix cache: fabric:/CoreApi.Currency.Application/Currency.Web : fabric:/CoreApi.Currency.Application/Currency.Web ():Fintech.CoreApi.Currency.ApplicationType_App57:Fintech.CoreApi.Currency.WebPkg:Fintech.CoreApi.Currency.WebType@fabric:/CoreApi.Currency.Application [SharedProcess] (1.0:1.0:131480539341800141) [1]/1/1  131480539360365465:0    ()  0 () 0 () false : (1, 541) (cc0b3c60-d84d-4553-83c4-b88e578f5f47) - [0,0] () " id="" type="CacheManager.PrefixResolveServiceAsyncOperation" text="ca94c2c8-b69e-4272-851b-ea6183f340dc:0: updated prefix cache: fabric:/CoreApi.Currency.Application/Currency.Web : fabric:/CoreApi.Currency.Application/Currency.Web ():Fintech.CoreApi.Currency.ApplicationType_App57:Fintech.CoreApi.Currency.WebPkg:Fintech.CoreApi.Currency.WebType@fabric:/CoreApi.Currency.Application [SharedProcess] (1.0:1.0:131480539341800141) [1]/1/1  131480539360365465:0    ()  0 () 0 () false : (1, 541) (cc0b3c60-d84d-4553-83c4-b88e578f5f47) - [0,0] ()"/>
<Event MSec= "31147.8951" PID="3716" PName=        "" TID= "632" EventName="ReverseProxy" ProviderName="Microsoft-ServiceFabric" FormattedMessage="ca94c2c8-b69e-4272-851b-ea6183f340dc Request url = http://localhost:19081/CoreApi.Currency.Application/Currency.Web/?symbols=CADUSD, verb = GET, remote (client) address = ::1, resolved service url = http://10.51.2.7:50080/cc0b3c60-d84d-4553-83c4-b88e578f5f47/131484872201597609/?symbols=CADUSD, request processing start time = 13:56:07.667608 (31,144.850 MSec) " traceId="ca94c2c8-b69e-4272-851b-ea6183f340dc" requestUrl="http://localhost:19081/CoreApi.Currency.Application/Currency.Web/?symbols=CADUSD" verb="GET" remoteAddress="::1" resolvedServiceUrl="http://10.51.2.7:50080/cc0b3c60-d84d-4553-83c4-b88e578f5f47/131484872201597609/?symbols=CADUSD" requestStartTime="13:56:07.667608 (31,144.850 MSec)"/>

Update 2: Opened an issue on github.com/Azure/service-fabric-issues/issues/429 with steps to reproduce

1
azure-service-fabric
I faced similar issue long back (If I remember correctly). I faced this issue when my partitioned stateless service was deployed as Singleton. – Pratik Mehta

1 Answers

0
votes

I had the exact same issue 5.7.198.9494 and issue now resolved for me after upgrading Service Fabric to 6.0.219.9494.

Thanks Mark for raising the bug in github - issue has been resolved.

So, in essense:

  1. The Service Fabric Explorer reported no errors/warnings
  2. The 404 FABRIC_E_SERVICE_DOES_NOT_EXIST error only occurred for some nodes (in my scenario, 4/5 nodes worked fine).
  3. This HTTP error was due to a bug in Service Fabric Reverse Proxy service resolution.
  4. Microsoft has fixed this in 6.0 CU1. As of today, the current latest available version is 6.0.219.9494 - and thus far the issue appears fixed for me.