Springboot application in one pod not able to connect to mongodb on another pod through Openshift

Question

I am trying to deploy mongodb pods (1 Primary & 2 Secondary) in OpenShift, they got deployed but went to "Error" state with the following error,

exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/db, terminating

The mongo pods are up and running successfully after the issue of following adm policy,

oc adm policy add-scc-to-group anyuid system:authenticated

Now I am trying to deploy a springboot application that connects to mongodb but the pod is restarting multiple times because of the following error,

 INFO Exception in monitor thread while connecting to server mongo-node-2.database:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
Caused by: java.net.SocketTimeoutException: connect timed out
                        INFO Exception in monitor thread while connecting to server mongo-node-3.database:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
Caused by: java.net.SocketTimeoutException: connect timed out
                        INFO Exception in monitor thread while connecting to server mongo-node-1.database:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
Caused by: java.net.SocketTimeoutException: connect timed out
                        WARN Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exceptio  is org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

The springboot app pod is uanble to connect to mongodb pod when deployed through openshift, so I have created a dummy pod using k8s and tried to connect mongodb pod and the connection is successful,

mongo mongo-node-1.database.svc (successful)

I am able to connect to mongodb from outside the OpenShift cluster through NodePort, from within the cluster using dns (mongo-node-1.database.svc) and from a pod created though k8s but not from the OpenShift deployment.

I suspect that the issue might be because of some policy that OpenShift is blocking the springboot pod to connect with the mongodb pod.

Any suggestions please.

Updated the exception in the above question. Please let me know if you need any more info — Bhavani Prasad
Have you tried to test if you can resolve the dns name of you mongodb service, or try to reach the mongodb service from other pods using telnet or curl? See here — Mr.KoopaKiller

Bhavani Prasad Bhavani Prasad · Accepted Answer · 2020-01-22T06:23:52

The issue is due to the pods and services in one project (namespace) are unable to access the pods and services in another project (namespace) in OpenShift.

All the pods and services in and can now access any pods and services in and vice versa using the following command.

$ oc adm pod-network join-projects --to=<project1> <project2> <project3>

Reference link - https://docs.openshift.com/container-platform/3.11/admin_guide/managing_networking.html

Springboot application in one pod not able to connect to mongodb on another pod through Openshift

1 Answers