Hashicorp Vault as a StatefulSet on Kubernetes

Question

I am trying to run Vault as a StatefulSet on Kubernetes.

I have a working consul cluster based on this: https://github.com/kelseyhightower/consul-on-kubernetes

My sts file for Vault looks like this:

kind: StatefulSet
metadata:
  name: vault
spec:
  serviceName: vault
  replicas: 2
  template:
    metadata:
      labels:
        app: vault
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - vault
              topologyKey: kubernetes.io/hostname
      containers:
        - name: vault
          image: "vault:0.9.0"
          ports:
          - containerPort: 8200
            name: http
          - containerPort: 8201
            name: backend
          args:
            - "server -config=/vault/config/vault-server.json"
          securityContext:
            capabilities:
              add:
                - IPC_LOCK
          volumeMounts:
            - name: config
              mountPath: /vault/config
            - name: tls
              mountPath: /etc/tls
      volumes:
        - name: config
          configMap:
            name: vault
        - name: tls
          secret:
            secretName: vault

My config file looks like this

{
    "disable_mlock": true,
    "listener": [
        {
            "tcp": {
                "tls_disable": true
            }
        }
    ],
    "storage": {
        "consul": {
            "address": "consul.default.svc.cluster.local:8500",
            "path": "vault",
            "token": "7e21f292-e7e7-f879-210c-4af2ae483cac"
        }
    }
}

When I apply the StatefulSet, I get a bind error

Error initializing listener of type tcp: listen tcp 127.0.0.1:8200: bind: address already in use

I have tried adding a listener with 127.0.0.1 and 0.0.0.0 with different ports. The pod is reading the config file because I was getting TLS warnings until I disabled.

Any ideas on what is bound to localhost on the pod? Any troubleshooting help would be appreciated

if you run docker run -p 8200:8200 -p 8201:8201 vault:0.9.0 server -config=/and/so/forth locally, does it start up as you would expect? — mdaniel
and do you get that error on all pods, or just one? and all the time, or intermittently? — mdaniel
also, its a bit weird to have "disable_mlock": true, when the container is explicitly granted IPC_LOCK (but I doubt that has anything at all to do with your bind question) — mdaniel
Runs as expected if using just Docker, both pods act the same, it's redundant, but I don't think it's the issue — jsmickey

jsmickey jsmickey · Accepted Answer · 2017-11-28T12:59:45

The issue was the Docker container starts vault in dev mode

From https://github.com/hashicorp/docker-vault/blob/master/0.X/Dockerfile#L69

# By default you'll get a single-node development server that stores everything
# in RAM and bootstraps itself. Don't use this configuration for production.
CMD ["server", "-dev"]

I added/changed the cmd and argument lines in the statefulSet yaml to

command: ["vault", "server"]
args:
  - "-config=/vault/config/vault-server.json"

This gets rid of dev mode and uses server mode.

Please note this is not a production ready example, it is just for learning

Hashicorp Vault as a StatefulSet on Kubernetes

2 Answers