Why ingress always return 502 Error to specific service nodeport only?

1
votes

I have an alpine docker image to run my raw PHP website on an apache server(PHP 7.4) EXPOSE 80.
I want to run the image on Kubernetes(GKE) with an ingress controller.
I'm pushing the image with gcloud command to the google container registry.
Both the deployment and service have no errors and created successfully as NodePort.
The ingress I deployed is from google tutorials(https://cloud.google.com/community/tutorials/nginx-ingress-gke)
In my ingress now there is:

  • 34.68.78.46.xip.io/
  • 34.68.78.46.xip.io/hello
  • 34.68.78.46.xip.io/jb(/|$)(.*)

The /hello is the same configuration of the tutorial and it is working fine.
The /jb is the same configuration as I mentioned below and always returning 502 Error.
Ingress details in the GCP console show no warnings or errors

I have checked:
Kubernetes GKE Ingress : 502 Server Error
GKE Ingress: 502 error when downloading file
502 Server Error Google kubernetes

Here is the deployment file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jomlahbazar-deployment
spec:
  selector:
    matchLabels:
      greeting: jomlah
      department: bazar
  replicas: 1
  template:
    metadata:
      labels:
        greeting: jomlah
        department: bazar
    spec:
      containers:
      - name: jomlah
        image: "us.gcr.io/third-nature-273904/jb-img-1-0:v1"
        ports:
        - containerPort: 80
        env:
        - name: "PORT"
          value: "80"

Here is the service file:

apiVersion: v1
kind: Service
metadata:
  name: jomlahbazar-service
spec:
  type: NodePort
  selector:
    greeting: jomlah
    department: bazar
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

Here in the ingress file:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/add-base-url : "true"
spec:
  rules:
  - host: 34.68.78.46.xip.io
    http:
      paths:
      - path: /
        backend:
          serviceName: jomlahbazar-service
          servicePort: 80
      - path: /hello
        backend:
          serviceName: hello-app
          servicePort: 8080
      - path: /jb(/|$)(.*)
        backend:
          serviceName: jomlahbazar-service
          servicePort: 80

Here is the ingress description:

Name:             ingress-resource
Namespace:        default
Address:          34.68.78.46
Default backend:  default-http-backend:80 (10.20.1.6:8080)
Rules:
  Host                Path  Backends
  ----                ----  --------
  34.68.78.46.xip.io
                      /              jomlahbazar-service:80 (<none>)
                      /hello         hello-app:8080 (10.20.2.61:8080)
                      /jb(/|$)(.*)   jomlahbazar-service:80 (<none>)
Annotations:
  kubernetes.io/ingress.class:                       nginx
  nginx.ingress.kubernetes.io/add-base-url:          true
  nginx.ingress.kubernetes.io/rewrite-target:        /$2
  nginx.ingress.kubernetes.io/ssl-redirect:          false
  nginx.ingress.kubernetes.io/use-regex:             true
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/add-base-url":"true","nginx.ingress.kubernetes.io/rewrite-target":"/$2","nginx.ingress.kubernetes.io/ssl-redirect":"false","nginx.ingress.kubernetes.io/use-regex":"true"},"name":"ingress-resource","namespace":"default"},"spec":{"rules":[{"host":"34.68.78.46.xip.io","http":{"paths":[{"backend":{"serviceName":"jomlahbazar-service","servicePort":80},"path":"/"},{"backend":{"serviceName":"hello-app","servicePort":8080},"path":"/hello"},{"backend":{"serviceName":"jomlahbazar-service","servicePort":80},"path":"/jb(/|$)(.*)"}]}}]}}

Events:
  Type    Reason          Age                 From                      Message
  ----    ------          ----                ----                      -------
  Normal  AddedOrUpdated  36m (x6 over 132m)  nginx-ingress-controller  Configuration for default/ingress-resource was added or updated

The output of kubectl get ing ingress-resource -o yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/add-base-url":"true","nginx.ingress.kubernetes.io/rewrite-target":"/$2","nginx.ingress.kubernetes.io/ssl-redirect":"false","nginx.ingress.kubernetes.io/use-regex":"true"},"name":"ingress-resource","namespace":"default"},"spec":{"rules":[{"host":"34.68.78.46.xip.io","http":{"paths":[{"backend":{"serviceName":"jomlahbazar-service","servicePort":80},"path":"/"},{"backend":{"serviceName":"hello-app","servicePort":8080},"path":"/hello"},{"backend":{"serviceName":"jomlahbazar-service","servicePort":80},"path":"/jb(/|$)(.*)"}]}}]}}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/add-base-url: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
  creationTimestamp: "2021-02-11T06:00:07Z"
  generation: 5
  name: ingress-resource
  namespace: default
  resourceVersion: "2195351"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/ingress-resource
  uid: 74dc822f-91cb-4902-991b-1ad298f44ae6
spec:
  rules:
  - host: 34.68.78.46.xip.io
    http:
      paths:
      - backend:
          serviceName: jomlahbazar-service
          servicePort: 80
        path: /
      - backend:
          serviceName: hello-app
          servicePort: 8080
        path: /hello
      - backend:
          serviceName: jomlahbazar-service
          servicePort: 80
        path: /jb(/|$)(.*)
status:
  loadBalancer:
    ingress:
    - ip: 34.68.78.46
1
dockerkubernetesgoogle-kubernetes-enginekubernetes-ingressgke-networking
There is misconfiguration in your deployment. Instead of env.name.value: "80" you should have env.name.value: "8080". I will elaborate soon as answer. – PjoterS
This is definitely an error yes. I changed it but still, I get the 502 error. Should I delete the deployment and service then create again with different names or should configuring them do just fine? Because I can still see the same backend services. – Ibrahim Abou Khalil
I fixed what you noted and edited the question info again, the 502 error is still showing. – Ibrahim Abou Khalil

1 Answers

1
votes

I've run some tests on my GKE cluster. I've replicate your behavior using 2 hello world applications v1 and v2.

Scenario 1

HW 1

    spec:
      containers:
      - name: hello1
        image: gcr.io/google-samples/hello-app:1.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080

SVC HW1

spec:
  type: NodePort
  selector:
    key: app
  ports:
    - port: 80
      targetPort: 8080

HW 2

    spec:
      containers:
      - name: hello2
        image: gcr.io/google-samples/hello-app:2.0
        env:
        - name: "PORT"
          value: "80"

SVC HW2

spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: hello2

Ingress

spec:
  rules:
  -  http:
      paths:
      - path: /hello2
        backend:
          serviceName: h2
          servicePort: 80
      - path: /hello
        backend:
          serviceName: fs
          servicePort: 80

Outputs:

$ curl 34.117.70.75/hello
Hello, world!
Version: 1.0.0
Hostname: fd-c6d79cdf8-7rmmd

$ curl 34.117.70.75/hello2

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>

In this scenario, deployment was configured to create pod which will listen on port: 80. As you skipped configureing containerPort in deployment, Kubernetes automatically used the same port in containerPort as was set in port. You can verify it using netstat command.

$ kubectl exec -ti h2-deploy-6dbf5b7899-g7rbj -- bin/sh
/ # netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 :::80                   :::*                    LISTEN      1/hello-app
/ #

In your service you set containerPort: 8080, so service expect traffic to go trought port 8080. As pod is listening only on 80 and traffic goes trought 8080 you are getting 502 error.

Scenario 2

After change value from "80" to "8080" and apply new configuration.

$ curl 34.117.70.75/hello
Hello, world!
Version: 1.0.0
Hostname: fd-c6d79cdf8-7rmmd

$ curl 34.117.70.75/hello2
Hello, world!
Version: 2.0.0
Hostname: h2-deploy-5f5ccfbf9f-fjhrb

Netstat:

$ kubectl exec -ti f2-deploy-5f5ccfbf9f-fjhrb -- bin/sh
/ # netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 :::8080                 :::*                    LISTEN      1/hello-app

Solution

Solution 1

You should change your application deployment to:

        env:
        - name: "PORT"
          value: "8080"

and apply new configuration.

Solution 2 Use containerPort

    spec:
      containers:
      - name: jb
        image: "us.gcr.io/third-nature-273904/jb-img-1-0:v3"
        ports:
        - containerPort: 8080

Note

Please note, if you created your own image and used EXPOSE in your Dockerfile, you should configure your Deployment to use this specific port.

Let me know if you still have issue.