EKS, Windows node. networkPlugin cni failed

0
votes

Per https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html, I ran the command, eksctl utils install-vpc-controllers --cluster <cluster_name> --approve

My EKS version is v1.16.3. I tries to deploy Windows docker images to a windows node. I got error below.

Warning FailedCreatePodSandBox 31s kubelet, ip-west-2.compute.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ab8001f7b01f5c154867b7e" network for pod "mrestapi-67fb477548-v4njs": networkPlugin cni failed to set up pod "mrestapi-67fb477548-v4njs_ui" network: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address

$ kubectl logs vpc-resource-controller-645d6696bc-s5rhk -n kube-system
I1010 03:40:29.041761       1 leaderelection.go:185] attempting to acquire leader lease  kube-system/vpc-resource-controller...
I1010 03:40:46.453557       1 leaderelection.go:194] successfully acquired lease kube-system/vpc-resource-controller
W1010 23:57:53.972158       1 reflector.go:341] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: watch of *v1.Pod ended with: too old resource version: 1480444 (1515040)

It complains too old resource version. How do I upgrade the version?

3
kubernetes
What version of the vpc-resource-controller do you use? - Jonas
$ kubectl logs vpc-resource-controller-645d6696bc-s5rhk -n kube-system I1010 03:40:29.041761 1 leaderelection.go:185] attempting to acquire leader lease kube-system/vpc-resource-controller... I1010 03:40:46.453557 1 leaderelection.go:194] successfully acquired lease kube-system/vpc-resource-controller W1010 23:57:53.972158 1 reflector.go:341] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: watch of *v1.Pod ended with: too old resource version: 1480444 (1515040) - Melissa Jenner
that part does not describe a related problem, I have posted an answer - Jonas

3 Answers

0
votes

To troubleshoot this I would...

  1. First list the components to make sure they're running:
$kubectl get pod -n kube-system | grep vpc
vpc-admission-webhook-deployment-7f67d7b49-wgzbg   1/1     Running   0          38h
vpc-resource-controller-595bfc9d98-4mb2g           1/1     Running   0          29
  1. If they are running check their logs
kubectl logs <vpc-yadayada> -n kube-system

  1. Make sure the instance type you are using has enough available IPs per ENI because in the Windows world only one ENI is used and is limited to the max available IP's per ENI minus one for the Primary IP address. I have run into this error before where I have exceeded the number of IP's available to my ENI.

  2. Confirm that the selector of your pod is right

nodeSelector:
  kubernetes.io/os: windows
  kubernetes.io/arch: amd64

As an anecdote, I have done the steps mentioned under the To enable Windows support for your cluster with a macOS or Linux client section of the doc you linked on a few clusters to date, and they have worked well.

0
votes

What is your output for

kubectl describe node <windows_node>

?

if it's like :

  vpc.amazonaws.com/CIDRBlock:           0
  vpc.amazonaws.com/ENI:                 0
  vpc.amazonaws.com/PrivateIPv4Address:  0

then you need to re-create the nodegroup with different instance type...

then try to deploy this :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: windows-server-iis-test
  namespace: default
spec:
  selector:
    matchLabels:
      app: windows-server-iis-test
      tier: backend
      track: stable
  replicas: 1
  template:
    metadata:
      labels:
        app: windows-server-iis-test
        tier: backend
        track: stable
    spec:
      containers:
      - name: windows-server-iis-test
        image: mcr.microsoft.com/windows/servercore:1809
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: IfNotPresent
        command:
        - powershell.exe
        - -command
        - "Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/2.0.1.6/ServiceMonitor.exe' -OutFile 'C:\\ServiceMonitor.exe'; echo '<html><body><br/><br/><marquee><H1>Hello EKS!!!<H1><marquee></body><html>' > C:\\inetpub\\wwwroot\\default.html; C:\\ServiceMonitor.exe 'w3svc'; "
        resources:
          limits:
            cpu: 256m
            memory: 256Mi
          requests:
            cpu: 128m
            memory: 100Mi
      nodeSelector:
        kubernetes.io/os: windows
---
apiVersion: v1
kind: Service
metadata:
  name: windows-server-iis-test
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: windows-server-iis-test
    tier: backend
    track: stable
  sessionAffinity: None
  type: ClusterIP

kubectl proxy

open browser http://localhost:8001/api/v1/namespaces/default/services/http:windows-server-iis-test:80/proxy/default.html will shown webpage with Hello EKS text

0
votes
  1. I removed the windows nodes, re-created windows nodes with different instance type. But, it did not work.
  2. Removed windows nodes group, re-created windows nodes group. It did not work.
  3. Finally, I removed entire EKS cluster, re-created eks cluster. The command, kubectl describe node <windows_node> gives me the output below.
  vpc.amazonaws.com/CIDRBlock           0         0
  vpc.amazonaws.com/ENI                 0         0
  vpc.amazonaws.com/PrivateIPv4Address  1         1

Deployed windows-server-iis.yaml. It works as expected. The root cause of the problem is mystery.