failed calling webhook inferenceservice.kfserving-webhook-server.defaulter , x509 certificate error in kubeflow

0
votes

I am new to kubeflow. I have installed it on a azure machine via minikube. Whenever I am trying to spawn a notebook I encounter the error as shown in the the following diagram.

enter image description here

i.e rror from server (InternalError): error when creating "something.yaml": Internal error occurred: failed calling webhook "inferenceservice.kfserving-webhook-server.defaulter": Post https://kfserving-webhook-server-service.kubeflow.svc:443/mutate-inferenceservices?timeout=30s . x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

Can someone help?

2
kubernetesistiokubeflow
are you using service mesh ? istio it's looks like it is mutual TLS error while service is trying to connect to another service. - Harsh Manvar
@HarshManvar Yes istio is being used. I just got the output for kubectl get pods -n istio-system. - Shivangi Bhardwaj
Then it might be mTLS issue - Harsh Manvar
some of the pod status is node-affinity and are not up. I think that may be the cause? @HarshManvar - Shivangi Bhardwaj
Please check service you are trying access is up and also check the policy in istio for mTLS - Harsh Manvar

2 Answers

2
votes

Seems to be an issue with anything compiled with Golang 1.15. What version of Minikube are you running? That might be the issue. See info on the kubernetes 1.19 release. I know you're running Minikube but maybe related.

Kubernetes is now built with golang 1.15.0-rc.1. The deprecated, legacy behavior of treating the CommonName field on X.509 serving certificates as a host name when no Subject Alternative Names are present is now disabled by default. It can be temporarily re-enabled by adding the value x509ignoreCN=0 to the GODEBUG environment variable. (#93264, @justaugustus) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Storage and Testing]

1
votes

we had a similar problem, and found the solution in this Issue:

kubectl delete mutatingwebhookconfigurations inferenceservice.serving.kubeflow.org && kubectl delete validatingwebhookconfigurations inferenceservice.serving.kubeflow.org && kubectl delete po kfserving-controller-manager-0 -n kfserving-system