External connection to Mongodb cluster in kubernetes

Question

I've used mongo k8s sidecar to provision a 3 member replica set mongo cluster on kubernetes. I need to expose mongodb service externally and hence created a LoadBalancer.

This is how the service looks like

LoadBalancer Ingress:     xxx.yyy.elb.amazonaws.com
Port:                     <unset>  27017/TCP
TargetPort:               27017/TCP
NodePort:                 <unset>  30994/TCP
Endpoints:            100.14.1.3:27017,100.14.1.4:27017,100.14.2.5:27017

Trying to connect using mongodb shell 3.6 works fine mongo --host xxx.yyy.elb.amazonaws.com

But in the java client code I see the following exception. java.net.UnknownHostException: mongo-1.mongo.dev.svc.cluster.local

I can confirm that the mongo pods are up and running. I am able to connect to mongo from other pods within the cluster - just not able to reach it externally.

Few things I don't understand is whats exactly happening in the java client.

The java client (which uses spring-data-mongo for configuration) is being created as follows.

MongoClient mongoClient = new MongoClient( "xxx.yyy.elb.amazonaws.com" , 27017 );

The fullstack trace is as follows

Caused by: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@161f6623. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo-2.mongo.dev.svc.cluster.local:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongo-2.mongo.dev.svc.cluster.local}, caused by {java.net.UnknownHostException: mongo-2.mongo.dev.svc.cluster.local}}, {address=mongo-0.mongo.dev.svc.cluster.local:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongo-0.mongo.dev.svc.cluster.local}, caused by {java.net.UnknownHostException: mongo-0.mongo.dev.svc.cluster.local}}, {address=mongo-1.mongo.dev.svc.cluster.local:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongo-1.mongo.dev.svc.cluster.local}, caused by {java.net.UnknownHostException: mongo-1.mongo.dev.svc.cluster.local}}] at com.mongodb.internal.connection.BaseCluster.createTimeoutException(BaseCluster.java:401) at com.mongodb.internal.connection.BaseCluster.selectServer(BaseCluster.java:120)

Why is the mongoClient using the pod name , even though I've passed the loadbalancer address? How do I fix this?

Thanks in advance

Rico Rico · Accepted Answer · 2019-01-07T20:50:15

You are getting an error for mongo-1.mongo.dev.svc.cluster.local and that's the internal endpoint within the cluster. In other words, that's how you would get to your Mongo instance from other pods in the cluster.

On the Java client, you need to use xxx.yyy.elb.amazonaws.com:27017 as the Mongo endpoint configuration.

Something like this:

MongoClient mongoClient = new MongoClient( "xxx.yyy.elb.amazonaws.com" , 27017 );

To give you an overview of the path, your Mongo instance is exposed through a LoadBalancer Kubernetes Service on port 27017.

Then the traffic comes into the load balancer and from there gets forwarded to your endpoints 100.14.1.3:27017, etc.

Then from there, they enter the Node on the NodePort 30994 on each node.

Then the nodes that have a pod running reply with an answer.

The Mongo process in the container itself runs on port 27017 so the moment the traffic gets to the node on port 30994 the container runtime forwards the traffic to your application in the container to 27017.

External connection to Mongodb cluster in kubernetes

1 Answers