Gettting error Timeout performing ZADD (10000ms), next: HSCAN in .net core 3.0 application with redis version 4

Question

I am running my .net core 3.0 application in ubuntu 18.04 machine in same network where redis present I am getting error

Timeout performing ZADD (10000ms), next: HSCAN InProccessingMsisdnTransaction_34234234, inst: 1, qu: 0, qs: 81, aw: False, rs: ReadAsync, ws: Idle, in: 34117, in-pipe: 0, out-pipe: 0, serverEndpoint: 10.10.10.10:6379, mgr: 10 of 10 available, clientName: SafeRedisConnection, IOCP: (Busy=0,Free=1000,Min=10,Max=1000), WORKER: (Busy=32,Free=32735,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

My implementation for redis interaction is as following

public static class RedisManager
    {

        private static readonly bool LogRedisRelatedActivities;
           private static readonly Lazy<ConfigurationOptions> ConfigOptions = new Lazy<ConfigurationOptions>(() =>
        {
            ConfigurationOptions configOptions = null;

            try
            {
                var redisInfo = ConfigurationHandler.GetSection<RedisElement>("RedisSection");

                if (redisInfo != null )
                {
                    if (redisInfo.IsActive)
                    {
                        redisInfo.RedisServerNameOrIp = ConfigurationHandler.GetSection<string>(StringConstants.EnvConfig.RedisRedisServerNameOrIp, Configurationtype.RedisSection);
                        redisInfo.RedisServerPort = ConfigurationHandler.GetSection<string>(StringConstants.EnvConfig.RedisRedisServerPort, Configurationtype.RedisSection);
                        redisInfo.RedisDefaultDatabase = ConfigurationHandler.GetSection<int>(StringConstants.EnvConfig.RedisDefaultDatabase, Configurationtype.RedisSection);


                        configOptions = new ConfigurationOptions();
                        configOptions.EndPoints.Add(redisInfo.RedisServerNameOrIp + ":" + redisInfo.RedisServerPort);
                        configOptions.ClientName = "SafeRedisConnection";

                        configOptions.ConnectTimeout = redisInfo.ConnectTimeout * 1000;
                        configOptions.SyncTimeout = redisInfo.SyncTimeout * 1000;
                        configOptions.AbortOnConnectFail = false;
                        configOptions.KeepAlive = redisInfo.KeepAliveDuration;
                        configOptions.DefaultDatabase = redisInfo.RedisDefaultDatabase;

                    }
                    else
                    {
                        Logger.Error("RedisSection is in-active");
                    }
                }
                else
                {
                    Logger.Error("RedisSection not found");
                }
            }
            catch (Exception ex)
            {
                Logger.Fatal(ex, ex);
            }
            return configOptions;
        });
        private static readonly Lazy<ConnectionMultiplexer> Conn = new Lazy<ConnectionMultiplexer>(
            () =>
            {
                try
                {
                    if (ConfigOptions != null && ConfigOptions.Value != null)
                    {
                        return ConnectionMultiplexer.Connect(ConfigOptions.Value);
                    }
                    return null;
                }
                catch (Exception ex)
                {
                    Logger.Fatal(ex.Message, ex);
                    return null;
                }
            });

        private static ConnectionMultiplexer Muxer => Conn.Value;

        static RedisManager()
        {
            try
            {
                LogRedisRelatedActivities = ConfigurationHandler.GetSection<bool>(StringConstants.AppSettingsKeys.LogRedisRelatedActivities);

                if (Muxer != null && Muxer.IsConnected)
                {
                    Logger.Info("Redis Connected ");
                }
                else
                {
                    Logger.Info("Redis Not Connected ");
                }

            }
            catch (Exception ex)
            {
                Logger.Fatal(ex.Message, ex);
            }
        }

        public static string GetStringItem(string key)
        {
            string val = null;
            try
            {
                IDatabase getDatabase;
                if (Muxer != null && Muxer.IsConnected && (getDatabase = Muxer.GetDatabase()) != null)
                {
                    val = getDatabase.StringGet(key);

                }
            }
            catch (Exception ex)
            {
                Logger.Fatal(ex.Message, ex);
            }
            return val;
        }       

        public static bool AddZList(string key, object value)
        {
            var isAdded = false;
            try
            {
                IDatabase getDatabase;
                if (Muxer != null && Muxer.IsConnected && (getDatabase = Muxer.GetDatabase()) != null)
                {
                    isAdded = getDatabase.SortedSetAdd(key, JsonSerializer.Serialize(value), Utility.GetCurrentSystemTime().Ticks);

                }
            }
            catch (Exception ex)
            {
                Logger.Fatal(ex.Message, ex);
            }
            return isAdded;
        }      
    }

When running same application from windows server i am not getting any such error

Did you take a look at the URL in the error message? Is it possible your ZADD command is actually taking a long time on the Redis server? Is the Redis server certifiably up and running and in good shape? — AKX
... So where are you running this, then? How do the environments differ? — AKX
Then it sounds like a misconfiguration of the Linux server. Have you checked its system logs? Firewalls/iptables/...? — AKX
I don't have any experience with Redis + C# but from my experience with Redis and the link you've posted it looks like the server can't keep up with replying. It has 81 commands enqueued and has already written 34kb of data to it's output buffer and is waiting for a client to read it. Are there any other clients connected to this Redis instance? Have you checked slowlog and TPS? Are you sure Redis received your command? — Grabusz
@grabusz other way around: 81 commands have been written and are awaiting responses, and 34kb of data is available on the receive socket; the receiver is on "readasync", which suggests that it is actively processing the data. My initial thought here is: were any of the prior commands really big BLOBs? Although I also need to check: you (OP) mention AWS - can I assume there is some TLS in play here? I wonder if the brittle bridge here is TLS+Async playing up. — Marc Gravell♦

Kamran Shahid Kamran Shahid · Accepted Answer · 2020-04-29T16:10:25

Problem was related to some threadpool implemenation in stackexchange.redis. Workaround is to initialize high number of minimum thread in our application. done it via some configuration in .csproj file

Gettting error Timeout performing ZADD (10000ms), next: HSCAN in .net core 3.0 application with redis version 4

1 Answers