StackExchange.Redis System.TimeoutException

Question

I got this timeout exception suddenly when I try to persist a range of data, it was working before and I didn't do any changes:

Timeout performing HMSET {key}, inst: 0, mgr: ExecuteSelect, err: never, queue: 2, qu: 1, qs: 1, qc: 0, wr: 1, wq: 1, in: 0, ar: 0, clientName: {machine-name}, serverEndpoint: Unspecified/localhost:6379, keyHashSlot: 2689, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=0,Free=2047,Min=4,Max=2047), Local-CPU: 100% (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)

I'm using Redis on windows.

Answer 1

In your timeout error message, I see Local-CPU: 100%. This is the CPU on your client that is calling into Redis server. You might want to look into what is causing the high CPU load on your client.

This article describes why high CPU usage can lead to client-side timeouts. https://gist.github.com/JonCole/db0e90bedeb3fc4823c2#high-cpu-usage

Answer 2

I found out what causing the issue, as I was trying to bulk inserting into hash. What I did is that I chunked the inserted list into smaller ones.

Answer 3

So, I battled with this issue for a few days and almost gave up. Like @Amr Reda said, breaking a large sets into smaller ones might work but that's not optimal.

In my case, I was trying to move 27,000 records into redis and i kept encountering the issue.

To resolve the issue, increase the SyncTimeout value in your redis connection string. It's set by default to 1000ms ie 1second. Large datasets typically take longer to add.

Answer 4

Quick suggestions that worked in my case, using a console .net project with very high concurrency using multithread (around 30.000).

In the program.cs, I added some ThreadPool settings:

int newWorkerThreadsPerCore = 50, newIOCPPerCore = 100;
ThreadPool.SetMinThreads(newWorkerThreadsPerCore, newIOCPPerCore);

Also, I had to change everything from:

var redisValue = dbCache.StringGet("SOMETHING");

To:

var redisValue = dbCache.StringGetAsync("SOMETHING").Result;

Even if you might think they look almost the same (considering you always end up waiting for a result), if you use the non-async version and one single thread receives a redis timeout, it will make all the other 29.999 threads waiting for redis to timeout too, while the async one will only cause a timeout in that only single thread.