LoadBalancer Healthy Fails when run Traefik in Google Cloud Kubernetes

2
votes

I'm trying to configure Traefik in Google Cloud Kubernetes, I'm creating a cluster, and configuring the cluster with the files below.

My code is based from here:

run.sh

#!/usr/bin/env bash

PROJECT="my-beautiful-project" # Replace This
CLUSTER_NAME="cluster-traefik-x"

# Create Cluster
gcloud container clusters create $CLUSTER_NAME --zone europe-west4-c

# Connect to Cluster:
gcloud container clusters get-credentials $CLUSTER_NAME --zone europe-west4-c --project $PROJECT

# Reserve IPs
gcloud compute addresses create my-web-static-ip-rui --global
gcloud compute addresses create my-web-static-ip-rui-dashboard --global

# Setup Traefik
kubectl apply -f 10-traefik-service-acc.yaml
kubectl apply -f 20-traefik-cr.yaml
kubectl apply -f 30-traefik-crb.yaml

kubectl apply -f 40-traefik-deployment.yaml
kubectl apply -f 50-traefik-svc.yaml

# gcloud container clusters get-credentials $CLUSTER_NAME --zone europe-west4-c --project $PROJECT && kubectl port-forward --namespace kube-system $(kubectl get pod --namespace kube-system --selector="k8s-app=traefik-ingress-lb" --output jsonpath='{.items[0].metadata.name}') 8080:8080 # DASHBOARD
# gcloud container clusters get-credentials $CLUSTER_NAME --zone europe-west4-c --project $PROJECT && kubectl port-forward --namespace kube-system $(kubectl get pod --namespace kube-system --selector="k8s-app=traefik-ingress-lb" --output jsonpath='{.items[0].metadata.name}') 8081:80 # HTTP

kubectl apply -f 60-traefik-webui-svc.yaml
# gcloud container clusters get-credentials $CLUSTER_NAME --zone europe-west4-c --project $PROJECT && kubectl port-forward --namespace kube-system $(kubectl get pod --namespace kube-system --selector="k8s-app=traefik-ingress-lb" --output jsonpath='{.items[0].metadata.name}') 8082:8080 # DASHBOARD

kubectl apply -f 70-traefik-ingress.yaml
kubectl apply -f 75-traefik-ingress-lb-rui.yaml
kubectl apply -f 210-ws-rui-deployment.yaml
kubectl apply -f 220-ws-rui-svc.yaml
kubectl apply -f 230-ws-rui-ingress.yaml

# curl http://127.0.0.1:8081/hello # Should fail
# curl -H host:api.my-domain.com http://127.0.0.1:8081/hello # Returns ok

10-traefik-service-acc.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress
  namespace: kube-system

20-traefik-cr.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch

30-traefik-crb.yaml

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress
subjects:
- kind: ServiceAccount
  name: traefik-ingress
  namespace: kube-system

40-traefik-deployment.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80 # LOADBALANCER
        - name: admin
          containerPort: 8080 # DASHBOARD
        args:
        - --api
        - --kubernetes
        - --logLevel=DEBUG # INFO | DEBUG

50-traefik-svc.yaml

kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web # LOADBALANCER
    - protocol: TCP
      port: 8080
      name: admin # DASHBOARD
  type: NodePort # -> https://docs.traefik.io/user-guide/kubernetes/
# I read in internet some examples with NodePort and some with LoadBalancer. 
# I think that the most correct is NodePort
#  type: LoadBalancer
#  loadBalancerIP: 130.211.20.21 # Use when type is LoadBalancer # THIS DOES NOT WORK WITH RESERVED IPs

60-traefik-webui-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  type: NodePort
#  type: ClusterIP # ClusterIP is the default ServiceType. The examples that I saw don't have anything, then is ClusterIP by default, but Kubernetes Ingress says: `error while evaluating the ingress spec: service "kube-system/traefik-web-ui" is type "ClusterIP", expected "NodePort" or "LoadBalancer"` then, I choose NodePort
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8080 # DASHBOARD

70-traefik-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
#    kubernetes.io/ingress.class: traefik # SHOULD I USE THIS HERE?
    kubernetes.io/ingress.global-static-ip-name: "my-web-static-ip-rui-dashboard"
spec:
  rules:
    - host: dashboard.api.my-domain.com
      http:
        paths:
        - path: /
          backend:
            serviceName: traefik-web-ui
            servicePort: web

75-traefik-ingress-lb-rui.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-lb-ingress
  namespace: kube-system
  annotations:
#    kubernetes.io/ingress.class: traefik # SHOULD I USE THIS HERE?
    kubernetes.io/ingress.global-static-ip-name: "my-web-static-ip-rui"

spec:
  rules:
    - host: api.my-domain.com
      http:
        paths:
        - path: /
          backend:
            serviceName: traefik-ingress-service
            servicePort: web

210-ws-rui-deployment.yaml


kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: ws-rui-hello-world
  labels:
    app: animals
    animal: bear
spec:
  replicas: 2
  selector:
    matchLabels:
      app: animals
      task: bear
  template:
    metadata:
      labels:
        app: animals
        task: bear
        version: v0.0.1
    spec:
      containers:
      - name: bear
#        image: supergiantkir/animals:bear
        image: registry.hub.docker.com/ruimartinsptl/python-user-web-service-lixo
        ports:
        - containerPort: 80

220-ws-rui-svc.yaml


apiVersion: v1
kind: Service
metadata:
  name: ws-rui-hello-world
spec:
  type: NodePort
  ports:
  - name: http
    targetPort: 80
    port: 80
  selector:
    app: animals
    task: bear

230-ws-rui-ingress.yaml


apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ws-rui-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: api.my-domain.com
    http:
      paths:
      # - path: / # SHOULD I USE also `/`?
      #   backend:
      #     serviceName: ws-rui-hello-world
      #     servicePort: http
      - path: /hello
        backend:
          serviceName: ws-rui-hello-world
          servicePort: http

But, I'm having some problems:

  • I can map the service ports in localhost, and run curl -H host:api.my-domain.com http://127.0.0.1:8081/hello or curl -H host:dashboard.api.my-domain.com http://127.0.0.1:8080/dashboard. But I can't acess to services from outside.

This is my Kubernetes Cluster:

Deployments were well deployd:

enter image description here

The IPs were well created enter image description here

The Services are well created, but ingress fails in health check, and is not a problem because of domain, because I already tried with my domains. enter image description here

This is the Health Check: enter image description here

The port 31398 is with health check problems: enter image description here

This is the health check that was created authomatically: enter image description here

But If I do a Port Forwarding from this port to my localhost, its work:

enter image description here

enter image description here

But when I try to access to realdomain, I receive always an 404 error. Someone can helpme please?

Someone that already configured Traefik in google cloud platform?

All tips are welcome :) ????

UPDATE: Solution

Traefik service should be LoadBalancer type instead of NodePort (file 50-traefik-svc.yaml), then I don't need to create the ingress for Traefik (file 70-traefik-ingress.yaml can be deteled)

I also need to insert the loadBalancerIP: xxx.xxx.xxx.xxx in 50-traefik-svc.yaml instead of kubernetes.io/ingress.global-static-ip-name: "my-web-static-ip-rui-dashboard" in 70-traefik-ingress.yaml. (This IP sometimes takes a long time until be set ~15 minutes)

Thanks to all people that tried to help me :)

2
kubernetesgoogle-cloud-platformtraefiktraefik-ingress
Thank you for providing very clear details about the issue. I have a guess but maybe you can help confirm it by editing the GCP health check to see what hostname it uses? I'm thinking maybe the health check doesn't pass any Host header and fails because the Ingress requires it. – Andy Shinn

2 Answers

0
votes

Clearly you know what you're doing, and I don't know specifics about the tool.

But I've seen this issue pop up occasionally... and this is especially relevant if you're running in some sort of "managed" environment where security people are automatically "fixing" security breaches.

1) Is anyone deleting the rules that allow the GCP health checker to get access to your IPs? (That rule would have been auto created, but sometimes "security people" remove them

2) Has anyone removed the 0.0.0.0/0 rule from the exposed API. (Guessing not since you've got a health check failure).

Check the log files for any PATCH to firewall rules.

0
votes

Your python application listening either /rui or /rui/hello paths, while you are trying to reach /hello , that's why it redirects to the default backed 404.

So there are two options:

a) either use a correct path

http://127.0.0.1:8080/rui/hello

b) or set rewrite-target in your ingress object

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ws-rui-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/rewrite-target: /rui/hello
spec:
  rules:
  - host:
    http:
      paths:
      - path: /hello
        backend:
          serviceName: ws-rui-hello-world
          servicePort: http