cant schedule prometheus-server pod on kubernetes, pod reports taints but nodes dont have any taint

0
votes

i am getting 

0/7 nodes are available: 2 node(s) had taints that the pod didn't tolerate, 5 node(s) had volume node affinity conflict.

for my prometheus server pod but if i check each nodes there are no taints. and there is enough cpu and memory to be allocated.. what am i missing here?

i tried deleting the pods and even the deployment object but the error still persists

all nodes have 0 taints.. this is a fresh prometheus install on a new kubernetes cluster the yaml files that i have used to work until now when i needed to deploy a new kubernetes cluster

1
kubernetesprometheuskubernetes-pod
For example master nodes have special taints on them which prevent usual pods from running on them. Do you have 2 master nodes? – Vasili Angapov
Can you show info of nodes (Specifically their taints and affinity details)? – Vishal Biyani

1 Answers

1
votes

0/7 nodes are available: 2 node(s) had taints that the pod didn't tolerate, 5 node(s) had volume node affinity conflict.

The message is specific: it's not the taints that are keeping your prometheus pods off of your workers, it's the volume that is the problem. If you are in AWS, it's because your volume is in an availability zone that your workers are not (so, a us-west-2a volume and us-west-2c workers, for example)

The shortest path to success in your situation may be to either recreate the volume in the correct A.Z. if it was empty, or manually create a new volume and copy the data into an A.Z. that matches your workers, or (of course) spin up a new worker in the A.Z. that matches the volume

all nodes have 0 taints..

Is for sure not true for two reasons: because the scheduler clearly says there are two Nodes with taints, and because unless you specifically stripped them off, the masters are almost always(?) provisioned with node.kubernetes.io/master:NoSchedule taints explicitly to keep workloads off of them