Wait for canal pod before scheduling pods on new node

1
votes

When creating/adding a node to kubernetes, we also have to create a Canal pod.

Currently, kubernetes does not wait for the Canal pod to be ready before trying to schedule pods, resulting in failures (error below)

Error syncing pod, skipping: failed to "CreatePodSandbox" for "nginx-2883150634-fh5s2_default(385d61d6-6662-11e7-8989-000d3af349de)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx-2883150634-fh5s2_default(385d61d6-6662-11e7-8989-000d3af349de)\" failed: rpc error: code = 2 desc = NetworkPlugin cni failed to set up pod \"nginx-2883150634-fh5s2_default\" network: failed to find plugin \"loopback\" in path [/opt/loopback/bin /opt/cni/bin]"

Once the Canal pod is up-and-running, simply deleting the failing pod(s) will fix the issue.

My question is: what would be the right way to tell kubernetes to wait for the network pod to be ready before trying to schedule pods on the node?

  • Should I taint the node to only allow Canal, and untaint once it is ready?
  • Should I script the deleting of failed pods once Canal is ready?
  • Is there a configuration or a way to do it that eliminate the issue?
1
kubernetes
Can you describe more of the steps that lead up to this? Maybe actual kubectl commands or at least the workflow and why you need to do that workflow? Isn't Canal something that is deployed to all clusters before deploying other Pods? – Andy Shinn
> Isn't Canal something that is deployed to all clusters before deploying other Pods? I thought so, but not with 1.6.6 Following your comment, I've updated to the latest version of Canal. Problem fixed. Here's what happens now: 1: new pod is created 2: k8s tries to schedule new pod, fails 3: new node is created 4: k8s tries to schedule new pod on new node, fails due to Canal not being ready 5: Canal is ready 6: ?? retries to schedule new pod on new node, it works. – Mia
In my opinion, these kind of failures are normal in these kind of scenarios. Design for retrying and failures instead of declaring explicit dependencies. – Andy Shinn

1 Answers

1
votes

This is common issue, so I'll post the answer anyway.

The behaviour is normal especially in a self-hosted k8s cluster. In a self-hosted environment, all deployments incl the control plane elements (eg. kube-apiserver, canal) are scheduled at the same time.

The failed pods should eventually start properly once the control plane is running. k8s will keep restarting failed pods until it comes up properly.

To make Canal start first, the manifest can be deployed in the k8s node together with the other control plane manifests (eg. kube-apiserver, kube-controller-manager). It's usually found in /etc/kubernetes/manifests but the path is completely arbitrary. However, if Canal takes too long to be ready, the same error will appear.