How do Kubernetes NodePort services with Service.spec.externalTrafficPolicy=Local route traffic?

2
votes

There seems to be two contradictory explanations of how NodePort services route traffic. Services can route traffic to one of the two, not both:

  1. Nodes (through the kube-proxy) According to kubectl explain Service.spec.externalTrafficPolicy and this article that adds more detail, packets incoming to NodePort services with Service.spec.externalTrafficPolicy=Local set get routed to a kube-proxy, which then routes the packets to the corresponding pods its running.
    • This kube-proxy networking documentation further supports this theory adding that endpoints add a rule in the service's IPtable that forwards traffic to nodes through the kube-proxy.
  2. Pods: services update their IPtables from endpoints, which contain the IP addresses for the pods they can route to. Furthermore, if you remove your service's label selectors and edit endpoints you can change where your traffic is routed to.

If one of these is right, then I must be misunderstanding something.

  • If services route to nodes, then why can I edit endpoints without breaking the IPtables?
  • If services route to pods, then why would services go through the trouble of routing to nodes when Service.spec.externalTrafficPolicy is set?
2
kuberneteskubernetes-service

2 Answers

5
votes

A Service is a virtual address/port managed by kube-proxy. Services forward traffic to their associated endpoints, which are usually pods but as you mentioned, can be set to any destination IP/Port.

A NodePort Service doesn't change the endpoint side of the service, the NodePort allows external traffic into Service via a port on a node.

Breakdown of a Service

kube-proxy can use 3 methods to implement the forwarding of a service from Node to destination.

  • a user proxy
  • iptables
  • ipvs

Most clusters use iptables, which is what is described below. I use the term "forward" instead of "route" because services use Network Address Translation (or the proxy) to "forward" traffic rather than standard network routing.

The service ClusterIP is a virtual entity managed by kube-proxy. This address/port combination is available on every node in the cluster and forwards any local (pod) service traffic to the endpoints IP and port. 

                                         / Pod (remote node)
Pod -- ClusterIP/Port -- KUBE-SVC-NAT  --  Pod
                                         \ Pod (remote node)

A service with a NodePort is the same as above, with the addition of a way to forward external traffic into the cluster via a Node. kube-proxy manages an additional rule to watch for external traffic and forward it into the same service rules. 

Ext --     NodePort   \                / Pod (remote node)
                        KUBE-SVC-NAT --  Pod
Pod -- ClusterIP/Port /                \ Pod (remote node)

The externalTrafficPolicy=Local setting makes a NodePort service use only a local Pod to service the incoming traffic. This avoids a network hop which removes the need to rewrite the source of the packet (via NAT). This results in the real network IP arriving at the pod servicing the connection, rather than one of the cluster nodes being the source IP. 

Ext --     NodePort   \                  Pod (remote node)
                        KUBE-SVC-NAT --  Pod (local)
Pod -- ClusterIP/Port /                  Pod (remote node)

iptables

I recommend attempting to trace a connection from source to destination for a service or nodeport on a host. It requires a bit of iptables knowledge but I think it's worthwhile

To list all the services ip/ports that will be forwarded:

iptables -vnL -t nat KUBE-SERVICES

To list all the nodeports that will be forwarded:

iptables -vnL -t nat KUBE-NODEPORTS

Once you have the rule you can jump through KUBE-SVC-XXX "target" rules in the full output.

iptables -vnL -t nat | less

0
votes

externalTrafficPolicy: Cluster will not used on ClusterIP, try to remove and apply it, it"ll work