Correct way to connect to redis pod in Kubernetes with Node.JS

Question

I'm setting up a new k8s environment with multiple pods running microservices written in node.js. Several of these services connect to a redis cache.

This is all working most of the time, but I am receiving intermittent errors when accessing redis that make me believe that I'm not connecting correctly, most commonly:

RedisServerException: READONLY You can't write against a read only slave.

If I try again, I am often successful after two or three attempts.

This is my redis demployment:

RESOURCES:

==> v1/Service
NAME                       TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)             AGE
cache-redis-ha-announce-0  ClusterIP  100.xxx.xxx.xxx <none>       6379/TCP,26379/TCP  163m
cache-redis-ha-announce-1  ClusterIP  100.xxx.xxx.xxx <none>       6379/TCP,26379/TCP  163m
cache-redis-ha-announce-2  ClusterIP  100.xxx.xxx.xxx <none>       6379/TCP,26379/TCP  163m
cache-redis-ha             ClusterIP  None            <none>       6379/TCP,26379/TCP  163m

==> v1/StatefulSet
NAME                   DESIRED  CURRENT  AGE
cache-redis-ha-server  3        3        94s

==> v1/Pod(related)
NAME                     READY  STATUS   RESTARTS  AGE
cache-redis-ha-server-0  2/2    Running  0         94s
cache-redis-ha-server-1  2/2    Running  0         64s
cache-redis-ha-server-2  2/2    Running  0         36s

==> v1/ConfigMap
NAME                      DATA  AGE
cache-redis-ha-configmap  3     163m
cache-redis-ha-probes     2     163m


NOTES:
Redis can be accessed via port 6379 and Sentinel can be accessed via port 26379 on the following DNS name from within your cluster:
cache-redis-ha.devtest.svc.cluster.local

In my service, I connect to redis thus:

this.client = redis.createClient(6379, "cache-redis-ha.devtest.svc.cluster.local");


this.delAsync = promisify(this.client.del).bind(this.client);


async flush(matchPattern: string): Promise<CacheResult> {
    let result: CacheResult = { matchPattern: matchPattern, reply: true };
    return await this.keysAsync(matchPattern).then(async (keys) => {
        result.matchedKeys = keys;
        if (keys.length) {
            return await this.delAsync(keys).then((reply) => {
                result.reply = reply;
                return result;
            });
        }
        return result;
    });
}

I tried to connecting to the Sentinel port in createClient, but this didn't work.

Is there something obviously wrong in my implementation?

Anton Kostenko Anton Kostenko · Accepted Answer · 2019-04-04T14:36:56

In a Redis cluster, you have one master which is R/W and several slaves which are RO.

When you use a single service for all Redis pods, your connections will round-robin to all available pods and K8s doesn't know which of them is a master one, that's why you sometimes meet that error. It happens when your connection to a service terminates on RO slave instead of RW master.

You need additional Service and something like a controller or other automation which will point that Service to a right pod which is a master now.

Also, you can get that information from Sentel using discovery:

Sentinels stay connected with other Sentinels in order to reciprocally check the availability of each other, and to exchange messages. However you don't need to configure a list of other Sentinel addresses in every Sentinel instance you run, as Sentinel uses the Redis instances Pub/Sub capabilities in order to discover the other Sentinels that are monitoring the same masters and slaves.

This feature is implemented by sending hello messages into the channel named sentinel:hello.

Similarly you don't need to configure what is the list of the slaves attached to a master, as Sentinel will auto discover this list querying Redis.

Every Sentinel publishes a message to every monitored master and slave Pub/Sub channel sentinel:hello, every two seconds, announcing its presence with ip, port, runid.

Every Sentinel is subscribed to the Pub/Sub channel sentinel:hello of every master and slave, looking for unknown sentinels. When new sentinels are detected, they are added as sentinels of this master.

Hello messages also include the full current configuration of the master. If the receiving Sentinel has a configuration for a given master which is older than the one received, it updates to the new configuration immediately.

Before adding a new sentinel to a master a Sentinel always checks if there is already a sentinel with the same runid or the same address (ip and port pair). In that case all the matching sentinels are removed, and the new added.

Also, you can call sentinel master mymaster on any node to get a current master.

So, finally you need to get a Redis Master address (or ID) and use it's Service (in your installation it is cache-redis-ha-announce-*) to connect to a current master.

Correct way to connect to redis pod in Kubernetes with Node.JS

1 Answers