Why does my Service Fabric Inter application service RPC take 2 minutes?

Question

I have two applications(say app1 and app2) in my cluster. Both these applications have Stateless Service as a gateway to connect to internal Stateful services. I am trying to make calls between these two applications. I have made Reverse Proxy Service Communication from app1 to app2.

Url to app2 from app1 : http://localhost:19081/app2/stateless_app2_service/api/values

The above scenario works fine within local cluster. But when deployed on Azure Cluster it's taking 2 minutes to reach app2.

Can someone help me in pointing what I am doing wrong that's causing this delay? Is this related to any of the cluster configurations? I have enabled port 19081 for reverseproxy in the ARM while creating the cluster.

Below is my event logger for the calls made through services on Azure cluster. Call is received after 2 minutes. It's happening with every call that involves Inter-application communication in the Azure cluster.

Code:

Stateless_app1_service

protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
        {
            return new ServiceInstanceListener[]
            {
                new ServiceInstanceListener(serviceContext =>
                    new KestrelCommunicationListener(serviceContext, "ServiceEndpoint", (url, listener) =>
                    {
                        ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting Kestrel on {url}");

                        return new WebHostBuilder()
                                    .UseKestrel()
                                    .ConfigureAppConfiguration((builderContext, config) =>
                                    {
                                       config.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
                                    })
                                    .ConfigureServices(
                                        services => services
                                            .AddSingleton<HttpClient>(new HttpClient())
                                            .AddSingleton<StatelessServiceContext>(serviceContext)
                                            .AddSingleton<ILogger>(GenerateLogger()))                                        
                                    .UseContentRoot(Directory.GetCurrentDirectory())                                    
                                    .UseStartup<Startup>()
                                    .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                    .UseUrls(url)
                                    .Build();
                    }))
            };
        }

Making the call:

_logger.TrackEvent("SF - Call made to other application");                
var response2 = await _httpClient.GetAsync($"{_reverseProxy}/{_appNameConnectors}/{_apiNameConnectors}/api/Values");
_logger.TrackEvent("SF- RPC - Call successfully finished!");

Diego Mendes Diego Mendes · Accepted Answer · 2019-02-08T15:04:41

Without tracing on the network level will be a bit difficult to guess the problem.

I would recommend you test a direct connection to the service using the service dns name(if enabled), as http://service1.application1

I.e:

Uri uri = new Uri("http://service1.application1:8080/api/values");
HttpClient client = new HttpClient();
var response = await client.GetAsync(uri);

if this solve the problem, we can identify the issue to be related to the proxy, if doesn't solve, you will need to investigate the network communication or:

put both on same node in azure and test the communication
check for firewall rules
Anti-virus
Check if the issue is not on the endpoint receiving the call

Why does my Service Fabric Inter application service RPC take 2 minutes?

1 Answers