We have a vendor-provided .NET web application running on IIS 8.5 (.NET 4.6.1). It communicates with Microsoft Dynamics CRM 2016 (on-premise) via the WCF Organization.svc.
As I've been load testing, the bottleneck seems to be in between the web server (henceforth "WFE") and the CRM server (that hosts the WCF Organization.svc). CPU/memory/disk on both servers show no issues. As I increase load, response time worsens and CPU/memory/disk usage remains unchanged.
If I look at IIS - Worker Processes on the WFE, I see my requests to the web application queueing up under load with wait times up to 40-50 seconds, meanwhile on the CRM server - IIS - Worker Processes, the requests there that queue up only wait up until 1-2 seconds.
So there seems to be a large wait happening before it gets to CRM to process. Config settings I've tried:
- System.net – connectionManagement – increased maxConnections on WFE and/or CRM machine.config.
- System.web – processModel – increased maxWorkerThreads, maxIoThreads, and minWorkerThreads. Also httpRuntime - minFreeThreads and minLocalRequestFreeThreads. Did this on both WFE and CRM server
- In web.config of CRM Server – CRMWeb: System.serviceModel – bindings – netTcpBindings – increased maxConnections System.serviceModel – behaviors – serviceBehaviors – behavior – serviceThrottling – maxConcurrentCalls, maxConcurrentSessions, and maxConcurrentInstances all set to 65536.
None of these individual changes make any observable difference. Response times and resource usage on the servers remain constant, so they don’t seem to do anything.
But if I disable "Claims-based authentication" in CRM "Deployment Manager", all response times improve dramatically (10x as fast).
I used Fiddler and could see that the web application only authenticates to ADFS once, so it's not re-authenticating over and over. CRM also has "authPersistNonNTLM" already set to True. And Fiddler on the CRM server shows it's never communicating with ADFS.
The web application is using version 8.0 of the CRM SDK and is specifically using the OrganizationServiceProxy class.
I appreciate any help/insight into further troubleshooting this issue.
