Kubernetes Cluster on AWS with Kops - NodePort Service Unavailable

Question

I am having difficulties accessing a NodePort service on my Kubernetes cluster.

Goal

set up ALB Ingress controller so that i can use websockets and http/2

setup NodePort service as required by that controller

Steps taken

Previously a Kops (Version 1.6.2) cluster was created on AWS eu-west-1. The kops addons for nginx ingress was added as well as Kube-lego. ELB ingress working fine.

Setup the ALB Ingress Controller with custom AWS keys using IAM profile specified by that project.

Changed service type from LoadBalancer to NodePort using kubectl replace --force

> kubectl describe svc my-nodeport-service
Name:                   my-node-port-service
Namespace:              default
Labels:                 <none>
Selector:               service=my-selector
Type:                   NodePort
IP:                     100.71.211.249
Port:                   <unset> 80/TCP
NodePort:               <unset> 30176/TCP
Endpoints:              100.96.2.11:3000
Session Affinity:       None
Events:                 <none>

> kubectl describe pods my-nodeport-pod
Name:           my-nodeport-pod
Node:           <ip>.eu-west-1.compute.internal/<ip>
Labels:         service=my-selector
Status:         Running
IP:             100.96.2.11
Containers:
  update-center:
    Port:               3000/TCP
    Ready:              True
    Restart Count:      0

(ssh into node)
$ sudo netstat -nap | grep 30176
tcp6       0      0 :::30176                :::*                    LISTEN      2093/kube-proxy

Results

Curl from ALB hangs

Curl from <public ip address of all nodes>:<node port for service> hangs

Expected

Curl from both ALB and directly to the node:node-port should return 200 "Ok" (the service's http response to the root)

Update: Issues created on github referencing above with some further details in some cases:

Jonathan Wickens Jonathan Wickens · Accepted Answer · 2017-08-08T07:26:09

By default Kops does not configure the EC2 instances to allows NodePort traffic from outside.

In order for traffic outside of the cluster to reach the NodePort you must edit the configuration for your EC2 instances that are your Kubernetes nodes in the EC2 Console on AWS.

Once in the EC2 console click "Security groups." Kops should have annotated the original Security groups that it made for your cluster as nodes.<your cluster name> and master.<your cluster name>

We need to modify these Security Groups to forward traffic from the default port range for NodePorts to the instances.

Click on the security group, click on rules and add the following rule.

Port range to open on the nodes and master: 30000-32767

This will allow anyone on the internet to access a NodePort on your cluster, so make sure you want these exposed.

Alternatively instead of allowing it from any origin you can allow it only from the security group created by for the ALB by the alb-ingress-controller. However, since these can be re-created it will likely be necessary to modify the rule on modifications to the kubernetes service. I suggest specifying the NodePort explicitly to it is a predetermined known NodePort rather than a randomly assigned one.

Kubernetes Cluster on AWS with Kops - NodePort Service Unavailable

2 Answers