Nginx real_ip_header through a GCE ingress?

Question

My goal is to filter access by IP address of an angular app deployed on Kubernetes Engine served by nginx through a GCE ingress.

But on my nginx the remote_addr is not right.

$LB_IP is the ip defined here : kubernetes.io/ingress.global-static-ip-name: app-angular

I'm using set_real_ip_from on nginx to set the ip from X-Forwarded-For

set_real_ip_from $LB_IP;
real_ip_header X-Forwarded-For;

The original ip comes in the X-Forwarded-For header as expected from the google doc : https://cloud.google.com/compute/docs/load-balancing/http/#components

I can see the X-Forwarded-For contains the $CLIENT_IP but the remote_addr is not correct, and by the way my filter on IP is not working. Any idea ?

My nginx logs :

10.40.40.40 - - [07/Feb/2018:11:29:48 +0000] "GET /styles.bundle.css HTTP/1.1" 200 35908 "http://MY_URL/home" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36" "$CLIENT_IP, $LB_IP"

But i want :

$CLIENT_IP - - [07/Feb/2018:11:29:48 +0000] "GET /styles.bundle.css HTTP/1.1" 200 35908 "http://MY_URL/home" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36" "$CLIENT_IP, $LB_IP"

My Ingress configuration :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: app-angular
spec:
  rules:
    - host: MY_URL
      http:
        paths:
        - backend:
            serviceName: app-backend
            servicePort: 80

My backend configuration :

apiVersion: v1
kind: Service
metadata:
  name: app-backend
spec:
  type: NodePort
  selector:
    app: app-angular
  ports:
  - port: 80
    targetPort: 80

My deployment configuration :

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: app-angular
spec:
  replicas: 2
  revisionHistoryLimit: 2
  template:
    metadata:
      labels:
        app: app-angular
        tier: frontend
    spec:
      containers:
      - name: app-angular
        image: gcr.io/MY_PROJECT/app-angular:MY_TAG
        imagePullPolicy: Always
        ports:
          - containerPort: 80

My nginx configuration :

server {
  listen 80;

  root /usr/share/nginx/html;

  location / {

    set_real_ip_from $LB_IP;
    real_ip_header X-Forwarded-For;

    allow 130.211.0.0/22; #Google IP
    allow $CLIENT_IP; # FILTERED IP
    allow 10.0.0.0/8; #internal network
    deny all;

    index index.html index.htm;
    try_files $uri $uri/ /index.html =404;
  }

}

Any idea ?

Unknown Unknown · Accepted Answer · 2018-02-07T18:52:10

They are two solutions (Thks to gcbirzan for helping me on GCP slack):

1) Update My nginx configuration with the good IPS from IP addresses ranges :

IP addresses ranges can be found here : https://console.cloud.google.com/networking/networks/list. You can just add set_real_ip_from for a region or all needed regions. Don't forget real_ip_recursive on;

server {
  listen 80;

  root /usr/share/nginx/html;

  location / {

    set_real_ip_from 10.128.0.0/20;
    ...
    real_ip_header X-Forwarded-For;
    real_ip_recursive on;

    allow 130.211.0.0/22; #Google IP
    allow $CLIENT_IP; # FILTERED IP
    allow 10.0.0.0/8; #internal network
    deny all;

    index index.html index.htm;
    try_files $uri $uri/ /index.html =404;
  }

}

2) Update My backend configuration with :

externalTrafficPolicy: Local

apiVersion: v1
kind: Service
metadata:
  name: app-backend
spec:
  type: NodePort
  selector:
    app: app-angular
  ports:
  - port: 80
    targetPort: 80
  externalTrafficPolicy: Local

Update nginx configuration :

now the IP shown in $remote_addr will be set with the Load balancer IP for your client requests and with google Infra IPS : 130.211.0.0/22,35.191.0.0/16

Don't forget real_ip_recursive on;

server {
  listen 80;

  root /usr/share/nginx/html;

  location / {

    set_real_ip_from $LB_IP;
    set_real_ip_from 130.211.0.0/22;
    set_real_ip_from 35.191.0.0/16;
    real_ip_header X-Forwarded-For;
    real_ip_recursive on;

    allow 130.211.0.0/22; #Google IP
    allow 35.191.0.0/16; #Google IP
    allow $CLIENT_IP; # FILTERED IP
    allow 10.0.0.0/8; #internal network
    deny all;

    index index.html index.htm;
    try_files $uri $uri/ /index.html =404;
  }

}

Nginx real_ip_header through a GCE ingress?

3 Answers