StackExchange.Redis API: ListLeftPopAsync is taking infinite time or timing out

Question

We are using Redis List to maintain the queue of items to be processed by our C# application. The application is using StackExchange.Redis 1.0.488 (aka SER) to connect to Redis and work with Lists by invoking ListRightPushAsync and ListLeftPopAsync SER APIs.

We are primarily verifying a scenario that how our application will behave when redis server becomes unavailable.

We have observed an issue after Redis server is restarted while application is busy processing the queue. Application resumes processing certain amount of items after that we are seeing an issue wherein ListLeftPopAsync takes infinite time and application hangs without any exceptions. Note that amount of load application was processing was high say 100000 items.Below is code snippet.

Consumer layer:

    public async Task<byte[]> DiscardProcessedItem()
    {
        var result = await ListLeftPopAsync(2, "2:l:queue1");
        return result;
    }

SER API Wrapper:

    public Task<byte[]> ListLeftPopAsync(int redisDb, string key)
    {
        var task = conn.GetDatabase(redisDb).ListLeftPopAsync(key);
        return Task.FromResult<byte[]>(task.Result);
    }

Note that the syncTimeout is 5000.

Then we tried using synchronous SER API by modifying the wrapper function as below

    public byte[] ListLeftPopAsync(int redisDb, string key)
    {
        return conn.GetDatabase(redisDb).ListLeftPop(key);
    }

After this change, application was not getting hung and entire queue was processed. But StackExchange.Redis thrown timeout exceptions while performing ListLeftPop for few items. The exception is pasted below.

Timeout performing LPOP 2:l:queue1, inst: 1, mgr: ProcessReadQueue, err: never, queue: 6, qu: 0, qs: 6, qc: 0, wr: 0, wq: 0, in: 260, ar: 1, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=0,Free=2047,Min=4,Max=2047), clientName: MYPC

Any clues would be much appreciated.

ankur ankur · Accepted Answer · 2016-11-06T20:19:38

increase the sync timeout or use some async mechanism provided by the stack exchange library.

 ConfigurationOptions co = new ConfigurationOptions()
    {
        SyncTimeout = 500000,
        EndPoints =
        {
            {url,portNumber }
        },
        AbortOnConnectFail = false // this prevents that error
    };

    seClient = ConnectionMultiplexer.Connect(co);

this could also be happening as you're using an older version of the client

StackExchange.Redis API: ListLeftPopAsync is taking infinite time or timing out

1 Answers