Using sysctls in Google Kubernetes Engine (GKE)

Question

I'm running a k8s cluster - 1.9.4-gke.1 - on Google Kubernetes Engine (GKE).

I need to set sysctl net.core.somaxconn to a higher value inside some containers.

I've found this official k8s page: Using Sysctls in a Kubernetes Cluster - that seemed to solve my problem. The solution was to make an annotation on my pod spec like the following:

annotations:
  security.alpha.kubernetes.io/sysctls: net.core.somaxconn=1024

But when I tried to create my pod:

Status:         Failed
Reason:         SysctlForbidden
Message:        Pod forbidden sysctl: "net.core.somaxconn" not whitelisted

So I've tried to create a PodSecurityPolicy like the following:

---
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: sites-psp
  annotations:
    security.alpha.kubernetes.io/sysctls: 'net.core.somaxconn'
spec:
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
  - '*'

... but it didn't work either.

I've also found that I can use a kubelet argument on every node to whitelist the specific sysctl: --experimental-allowed-unsafe-sysctls=net.core.somaxconn

I've added this argument to the KUBELET_TEST_ARGS setting on my GCE machine and restarted it. From what I can see from the output of ps command, it seems that the option was successfully added to the kubelet process on the startup:

/home/kubernetes/bin/kubelet --v=2 --kube-reserved=cpu=60m,memory=960Mi --experimental-allowed-unsafe-sysctls=net.core.somaxconn --allow-privileged=true --cgroup-root=/ --cloud-provider=gce --cluster-dns=10.51.240.10 --cluster-domain=cluster.local --pod-manifest-path=/etc/kubernetes/manifests --experimental-mounter-path=/home/kubernetes/containerized_mounter/mounter --experimental-check-node-capabilities-before-mount=true --cert-dir=/var/lib/kubelet/pki/ --enable-debugging-handlers=true --bootstrap-kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --anonymous-auth=false --authorization-mode=Webhook --client-ca-file=/etc/srv/kubernetes/pki/ca-certificates.crt --cni-bin-dir=/home/kubernetes/bin --network-plugin=kubenet --volume-plugin-dir=/home/kubernetes/flexvolume --node-labels=beta.kubernetes.io/fluentd-ds-ready=true,cloud.google.com/gke-nodepool=temp-pool --eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5% --feature-gates=ExperimentalCriticalPodAnnotation=true

The problem is that I keep receiving a message telling me that my pod cannot be started because sysctl net.core.somaxconn is not whitelisted.

Is there some limitation on GKE so that I cannot whitelist a sysctl? Am I doing something wrong?

chrishiestand chrishiestand · Accepted Answer · 2018-10-05T21:34:14

Until sysctl support becomes better integrated you can put this in your pod spec

spec:
  initContainers:
    - name: sysctl-buddy
      image: busybox:1.29
      securityContext:
        privileged: true
      command: ["/bin/sh"]
      args:
        - -c
        - sysctl -w net.core.somaxconn=4096 vm.overcommit_memory=1
      resources:
        requests:
          cpu: 1m
          memory: 1Mi

Using sysctls in Google Kubernetes Engine (GKE)

2 Answers