For context, I work at a large company with teams in different clouds (Azure and IBM cloud formerly Bluemix). The Ops team will have control of Jenkins master eventually, but since my team is the first needing this, we are setting it up.
What we have:
- Jenkins master is in Azure (cloud A), latest version, and we have the Kubernetes plugin (among others) to have dynamic provisioning of agents.
- My team works in IBM Kubernetes clusters (cloud B). In this cloud, we have a namespace for devops in a cluster of k8s, where we want the slave agents to be created and run. In this cloud we also created a service accounts in that namespace, a role, and rolebinding.
Problem: Configuring Jenkins master to connect with such namespace in IBM cloud to provision agents and run the jobs as needed. Especifically, in Jenkins > Manage > COnfigure System > Cloud. Here I have:
- Kubernetes URL set to the IP:port of the master node of the cluster
- Kubernetes server certificate key, I put here the certificate I got when I created the service account.
- Namespace, Credentials I have the service account and finally the Jenkins URL.
When I try that configuration, and test the connection I get:
Error testing connection https://ip:port: Failure executing: GET at: https://ip:port/api/v1/namespaces/devops/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
Questions:
- Anybody knows how to do this or has links to something similar?
- In the image, what should I put in kubernetes URL? According to the docs, I should enter the container engine cluster endpoint, what is that in IBM Cloud? I have the IP and port of the Master node.
I´ve read a lot of documentation from IBM, the kubernetes plugin for Jenkins (https://github.com/jenkinsci/kubernetes-plugin) and tons of other posts explaining how to configure jenkins with kubernetes and dynamic provisioning and many of them said it is possible to have jenkins and the slaves in different clouds, but none of them explained how to do it..
Thank you in advance.
++++++++++++++++++ UPDATE +++++++++++++++++++++++
In the following screenshot I show the configurations that I am using. In particular, the fields Kubernetes URL and namespace.
Thanks to @samhain1138 for his help so far, but I cannot get a connection test successful in the Jenkins Kubernetes plugin configuration section. I think I may not be entering the correct info in some of the fields in that section.
Note: Please keep in mind that my setup is as follows: Jenkins master is in Cloud A (Azure) and I want the agent nodes to run is in a different cluster in another cloud, call it Cloud B (which in my case is IBM Cloud).
In the screenshot above I am certain that I have the correct values for the Kubernetes URL and namespace, but I am unsure about the other fields (Kubernetes server certificate key and Credentials.)
In the Kubernetes server certificate key field I tried putting:
- The ca.cert of the Service Account in the kubernetes cluster in IBM. (Obtained by 1. creating service account, getting the secret of that service account, and extracting the ca.cert from such secret)
- The "Token" from the server, which I get doing a kubectl config view in the kubernetes cluster in IBM. The token is in the field users>user>auth-provider>config>id-token when you execute the command "kubectl config view"
In Credentials I created and tried different Kinds:
- Username and Password: I tried this with my credentials for the IBM cluster and with the service account credentials (When I created the service account, role, and role-binding, I noticed that user was created which was serviceaccountname-namespace-cluster)
- Kubernetes Service Account
I tried all the combinations of those, and could never get Connection Successful when Testing the Connection.
Erros I get:
Error testing connection https://APISERVER: Failure executing: GET at: https://APISERVER/api/v1/namespaces/a-devops-namespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
Error testing connection https://APISERVER: Failure executing: GET at: https://APISERVER/api/v1/namespaces/a-devops-namespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:anonymous" cannot list pods in the namespace "a-devops-namespace".