check if Kubernetes deployment was sucessful in CI/CD pipeline

2
votes

I have an AKS cluster with Kubernetes version 1.14.7.

I have build CI/CD pipelines to deploy newly created images to the cluster.

I am using kubectl apply to update a specific deployment with the new image. sometimes and for many reasons, the deployment fails, for example ImagePullBackOff.

is there a command to run after the kubectl apply command to check if the pod creation and deployment was successful?

4
azurekubernetesdeploymentcontinuous-integrationazure-aks
Check the events section after running the command: kubectl -n [your namespace] describe pod [your pod name] and check what was the reason of the ImagePullBackOff and in my opinion try first to fix the issue you have. - Adam S.

4 Answers

3
votes

For this purpose Kubernetes has kubectl rollout and you should use option status.

By default 'rollout status' will watch the status of the latest rollout until it's done. If you don't want to wait for the rollout to finish then you can use --watch=false. Note that if a new rollout starts in-between, then 'rollout status' will continue watching the latest revision. If you want to pin to a specific revision and abort if it is rolled over by another revision, use --revision=N where N is the revision you need to watch for.

You can read the full description here

If you use kubect apply -f myapp.yaml and check rollout status you will see:

$ kubectl rollout status deployment myapp
Waiting for deployment "myapp" rollout to finish: 0 of 3 updated replicas are available…
Waiting for deployment "myapp" rollout to finish: 1 of 3 updated replicas are available…
Waiting for deployment "myapp" rollout to finish: 2 of 3 updated replicas are available…
deployment "myapp" successfully rolled out
0
votes

You can parse the output through jq:

kubectl get pod -o=json | jq '.items[]|select(any( .status.containerStatuses[]; .state.waiting.reason=="ImagePullBackOff"))|.metadata.name'
0
votes

It looks like kubediff tool is a perfect match for your task:

Kubediff is a tool for Kubernetes to show you the differences between your running configuration and your version controlled configuration.

The tool can be used from the command line and as a Pod in the cluster that continuously compares YAML files in the configured repository with the current state of the cluster. 

$ ./kubediff
Usage: kubediff [options] <dir/file>...

Compare yaml files in <dir> to running state in kubernetes and print the
differences.  This is useful to ensure you have applied all your changes to the
appropriate environment.  This tools runs kubectl, so unless your
~/.kube/config is configured for the correct environment, you will need to
supply the kubeconfig for the appropriate environment.

kubediff returns the status to stdout and non-zero exit code when difference is found. You can change this behavior using command line arguments.

You may also want to check the good article about validating YAML files:

0
votes

There is another way to wait for deployment to become available with a configured timeout like

kubectl wait --for=condition=available --timeout=60s deploy/myapp

otherwise kubectl rollout status can be used but it may stuck forever in some rare cases and will require manual cancellation of pipeline if that happens.