StackExchange.Redis timeout on High number of requests

Question

I have a distributed webservices (WCF) that uses StackExchange.Redis, with about 5 milion requests per day.

I getting this error some times (the values can be diferent some times):

System.TimeoutException: Timeout performing SISMEMBER KeyAllUsersFlag, inst: 7, queue: 6, qu: 0, qs: 6, qc: 0, wr: 0, wq: 0, in: 20, ar: 0, clientName: SERVER, IOCP: (Busy=20,Free=3180,Min=2400,Max=3200), WORKER: (Busy=7,Free=793,Min=400,Max=800), Local-CPU: unavailable

on webconfig i have:

httpRuntime targetFramework="4.6.1" minFreeThreads="704" minLocalRequestFreeThreads="608"

on machine config i have:

processModel maxWorkerThreads="100" maxIoThreads="400" minWorkerThreads="50" minIoThreads="300"

On config for MultiPlexer i have:

var configurationOptions = new ConfigurationOptions
{
    AbortOnConnectFail = false,
    SyncTimeout = 2000,
};

What i can do to otimize the call to redis, to have a fast response and not get this error?

Answer 1

You might have problem with amount of established connections to redis server. Connection are establishing through TCP port. The maximum number of TCP sessions a single source IP can make to a single destination IP and port is 65,535. This assumes that every available source port in the 16-bit source port range (excluding 0) from the source IP is utilized.In practice, most operating systems only create emphemeral source ports - the type of port that clients use - starting at port 32768 (the lowest number in the second half of the 0-65535 range), which means that you can make up to 32,768 connections to a single destination IP and listening port. Obviously, other resource constraints may prevent this, but that's the theoretical maximum. To make long story short - please ensure that you are closing connection after worker finishes his work (SISMEMBER command). Error message says that you have available workers but they are unable to establish connection to redis server and fails due to client timeout. Please check number of connection and if you are running out of available amount of connections use keepAlive option in your connection to redis. You can also use connectTimeout option to set the timeout for connect operations (in ms) and connectRetry to set the number of times to repeat connect attempts during initial Connect command.