Kubernetes pod secrets /var/run/secrets missing

3
votes

TL;DR:

Kubernetes pods created via terraform don't have a /var/run/secrets folder, but a pod created according to the hello-minikube tutorial does - why is that, and how can I fix it?

Motivation: I need traefik to be able to talk to the k8s cluster.

Details

I have set up a local Kubernetes cluster w/ minikube and set up terraform to work with that cluster.

To set up traefik, you need to create an Ingress and a Deployment which are not yet supported by terraform. Based on the workaround posted in that issue, I use an even simpler module to execute yaml files via terraform:

# A tf-module that can create Kubernetes resources from YAML file descriptions.
variable "name" {}
variable "file_name" { }
resource "null_resource" "kubernetes_resource" {
  triggers {
    configuration = "${var.file_name}"
  }
  provisioner "local-exec" {
    command = "kubectl apply -f ${var.file_name}"
  }
}

The resources created in this way show up correctly in the k8s dashboard.

However, the ingress controller's pod logs:

time="2017-12-30T13:49:10Z" 
level=error 
msg="Error starting provider *kubernetes.Provider: failed to create in-cluster
    configuration: open /var/run/secrets/kubernetes.io/serviceaccount/token:
    no such file or directory"

(line breaks added for readability)

/bin/bashing into the pods, I realize none of them have a path /var/run/secrets, except for the hello-minikube pod from the minikube tutorial, which was created with just two kubectl commands:

$ kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080
$ kubectl expose deployment hello-minikube --type=NodePort

Compared to the script in the terraform issue, I removed kubectl params like --certificate-authority=${var.cluster_ca_certificate}, but then again, I didn't provide this when setting up hello-minikube either, and the original script doesn't work as-is, since I couldn't figure out how to access the provider details from ~/.kube/config in terraform.

I tried to find out if hello-minikube is doing something fancy, but I couldn't find its source code.

Do I have to do something specific to make the token available? According to traefic issue 611, the InCluster-configuration should be automated, but as it currently stands I'm stuck.

Versions

Host system is a Windows 10 machine

> minikube version
minikube version: v0.24.1

> kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-11-29T22:43:34Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

Related

There are some related questions and github issues, but they didn't help me fix the problem yet either.

1
kubernetesterraformminikubetraefik
Foremost, thank you for an amazing question write up; I would use this question as a template for how others should ask! Can you check the automountServiceAccountToken field in the PodSpec and see if it is true? The only other constructive question I know is whether its serviceAccountName points to an existing S.A.; I would hope a bogus one would bomb the deploy, but don't know for sure what will happen in that case - mdaniel
Thank you so much!! Setting automountServiceAccountToken immediately fixed the problem. I was staring right at it but I just didn't see what was going on ;) Care to post this as an answer? The fact that it just deployed without even complaining certainly made this harder to pinpoint, but I guess that's a problem of the traefik image? - mnemosyn
I guess that's a problem of the traefik image unknown, but it certainly seems reasonable to check their issue tracker and see if we can't improve the experience for the next person, as that's 100% something the image could guard against - mdaniel

1 Answers

7
votes

Foremost, thank you for an amazing question write up; I would use this question as a template for how others should ask!

Can you check the automountServiceAccountToken field in the PodSpec and see if it is true?

The only other constructive question I know is whether its serviceAccountName points to an existing S.A.; I would hope a bogus one would bomb the deploy, but don't know for sure what will happen in that case