ReplicaFailure set to FailedCreate while creating bookinfo sample with istio

Question

I have installed istio 1.5.4 version. I am trying to deploy the bookinfo app using the command - kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v1 created
deployment.apps/reviews-v2 created
deployment.apps/reviews-v3 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created

-bash-4.2$ kubectl get services
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
details       ClusterIP   10.108.221.161   <none>        9080/TCP   9s
kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP    35d
productpage   ClusterIP   10.106.237.68    <none>        9080/TCP   9s
ratings       ClusterIP   10.96.155.198    <none>        9080/TCP   9s
reviews       ClusterIP   10.107.119.19    <none>        9080/TCP   9s
-bash-4.2$ kubectl get pods
No resources found in default namespace.

Pod is not getting created.

-bash-4.2$ kubectl get all
NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/details       ClusterIP   10.108.221.161   <none>        9080/TCP   107s
service/kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP    35d
service/productpage   ClusterIP   10.106.237.68    <none>        9080/TCP   107s
service/ratings       ClusterIP   10.96.155.198    <none>        9080/TCP   107s
service/reviews       ClusterIP   10.107.119.19    <none>        9080/TCP   107s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/details-v1       0/1     0            0           107s
deployment.apps/productpage-v1   0/1     0            0           107s
deployment.apps/ratings-v1       0/1     0            0           107s
deployment.apps/reviews-v1       0/1     0            0           107s
deployment.apps/reviews-v2       0/1     0            0           107s
deployment.apps/reviews-v3       0/1     0            0           107s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/details-v1-78d78fbddf       1         0         0       107s
replicaset.apps/productpage-v1-85b9bf9cd7   1         0         0       107s
replicaset.apps/ratings-v1-6c9dbf6b45       1         0         0       107s
replicaset.apps/reviews-v1-564b97f875       1         0         0       107s
replicaset.apps/reviews-v2-568c7c9d8f       1         0         0       107s
replicaset.apps/reviews-v3-67b4988599       1         0         0       107s

-bash-4.2$ kubectl describe deployment.apps/reviews-v1
Name:                   reviews-v1
Namespace:              default
CreationTimestamp:      Thu, 06 Aug 2020 07:04:25 -0400
Labels:                 app=reviews
                        version=v1
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"reviews","version":"v1"},"name":"reviews-v1","na...
Selector:               app=reviews,version=v1
Replicas:               1 desired | 0 updated | 0 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=reviews
                    version=v1
  Service Account:  bookinfo-reviews
  Containers:
   reviews:
    Image:      docker.io/istio/examples-bookinfo-reviews-v1:1.15.0
    Port:       9080/TCP
    Host Port:  0/TCP
    Environment:
      LOG_DIR:  /tmp/logs
    Mounts:
      /opt/ibm/wlp/output from wlp-output (rw)
      /tmp from tmp (rw)
  Volumes:
   wlp-output:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
   tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
Conditions:
  Type             Status  Reason
  ----             ------  ------
  Progressing      True    NewReplicaSetCreated
  Available        False   MinimumReplicasUnavailable
  ReplicaFailure   True    FailedCreate
OldReplicaSets:    reviews-v1-564b97f875 (0/1 replicas created)
NewReplicaSet:     <none>
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  5m54s  deployment-controller  Scaled up replica set reviews-v1-564b97f875 to 1

I am getting ReplicaFailure set to FailedCreate

I tried increasing the progressDeadLineSeconds as mentioned in another thread but still I am getting the same issue.

-bash-4.2$ kubectl patch deployment.apps/reviews-v1 -p '{"spec":{"progressDeadlineSeconds":6000}}'
deployment.apps/reviews-v1 patched
-bash-4.2$ kubectl get pods
No resources found in default namespace.

Pods are not getting created. Any idea what could have gone wrong?

-bash-4.2$ kubectl get events
LAST SEEN   TYPE      REASON              OBJECT                                 MESSAGE
4m32s       Warning   FailedCreate        replicaset/details-v1-78d78fbddf       Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: x509: certificate signed by unknown authority
59m         Normal    ScalingReplicaSet   deployment/details-v1                  Scaled up replica set details-v1-78d78fbddf to 1
4m32s       Warning   FailedCreate        replicaset/productpage-v1-85b9bf9cd7   Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: x509: certificate signed by unknown authority
59m         Normal    ScalingReplicaSet   deployment/productpage-v1              Scaled up replica set productpage-v1-85b9bf9cd7 to 1
4m32s       Warning   FailedCreate        replicaset/ratings-v1-6c9dbf6b45       Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: x509: certificate signed by unknown authority
59m         Normal    ScalingReplicaSet   deployment/ratings-v1                  Scaled up replica set ratings-v1-6c9dbf6b45 to 1
4m31s       Warning   FailedCreate        replicaset/reviews-v1-564b97f875       Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: x509: certificate signed by unknown authority
59m         Normal    ScalingReplicaSet   deployment/reviews-v1                  Scaled up replica set reviews-v1-564b97f875 to 1
4m32s       Warning   FailedCreate        replicaset/reviews-v2-568c7c9d8f       Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: x509: certificate signed by unknown authority
59m         Normal    ScalingReplicaSet   deployment/reviews-v2                  Scaled up replica set reviews-v2-568c7c9d8f to 1
4m34s       Warning   FailedCreate        replicaset/reviews-v3-67b4988599       Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: x509: certificate signed by unknown authority
59m         Normal    ScalingReplicaSet   deployment/reviews-v3                  Scaled up replica set reviews-v3-67b4988599 to 1
-bash-4.2$ kubectl describe rs replicasetname
Error from server (NotFound): replicasets.extensions "replicasetname" not found

add output of kubectl get events and output of kubectl describe rs replicasetname — Arghya Sadhu
-bash-4.2$ kubectl describe rs replicasetname Error from server (NotFound): replicasets.extensions "replicasetname" not found — preethi gnanesh
LAST SEEN TYPE REASON OBJECT MESSAGE 4m32s Warning FailedCreate replicaset/details-v1-78d78fbddf Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: x509: certificate signed by unknown authority 59m Normal ScalingReplicaSet deployment/details-v1 Scaled up replica set details-v1-78d78fbddf to 1 — preethi gnanesh
Edit the question to add these...not in the comments..it's not possible to read these in comments — Arghya Sadhu
All your istio and kubernetes components are healthy and running? Could you show `kubectl get pods --all-namespaces/? What is your infrastructure? I tried your commands on gke and everything worked just fine. — Jakub

Arghya Sadhu Arghya Sadhu · Accepted Answer · 2020-08-06T12:12:06

From the docs here

x509: certificate signed by unknown authority errors are typically caused by an empty caBundle in the webhook configuration

Verify the caBundle in the mutatingwebhookconfiguration matches the root certificate mounted in the istiod pod.

$ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | md5sum
4b95d2ba22ce8971c7c92084da31faf0  -
$ kubectl -n istio-system get secret istiod-service-account-token -o jsonpath='{.data.root-cert\.pem}' | md5sum
4b95d2ba22ce8971c7c92084da31faf0  -

The CA certificate should match. If they do not, restart the istiod pods.

$ kubectl -n istio-system patch deployment istiod \
    -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"
deployment.extensions "istiod" patched

ReplicaFailure set to FailedCreate while creating bookinfo sample with istio

2 Answers