Kubernetes HTTP liveness probe fails with “connection refused” even though URL works without it

2
votes

ENVIRONMENT:

Kubernetes version: v1.16.3  
OS: CentOS 7  
Kernel: Linux k8s02-master01 3.10.0-1062.4.3.el7.x86_64 #1 SMP Wed Nov 13 23:58:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

WHAT HAPPENED:

I have a Wordpress Deployment running a container built from a custom Apache/Wordpress image. The image exposes port 8080 instead of 80 (Dockerfile below). The Pod is exposed to the world through Traefik reverse proxy. Everything works fine without any liveness or readiness checks. Pod gets ready and Wordpress is accessible from https://www.example.com/.

I tried adding liveness and readiness probes and they both repeatedly fail with "connection refused". When I remove both probes and reapply the Deployment, it works again. It works until the probe hits the failure threshhold, at which point the container goes into an endless restart loop and becomes unaccessible.

POD EVENTS:

Events:
  Type     Reason     Age                   From                        Message
  ----     ------     ----                  ----                        -------
  Normal   Scheduled  <unknown>             default-scheduler           Successfully assigned development/blog-wordpress-5dbcd9c7c7-kdgpc to gg-k8s02-worker02
  Normal   Killing    16m (x2 over 17m)     kubelet, gg-k8s02-worker02  Container blog-wordpress failed liveness probe, will be restarted
  Normal   Created    16m (x3 over 18m)     kubelet, gg-k8s02-worker02  Created container blog-wordpress
  Normal   Started    16m (x3 over 18m)     kubelet, gg-k8s02-worker02  Started container blog-wordpress
  Normal   Pulled     13m (x5 over 18m)     kubelet, gg-k8s02-worker02  Container image "wordpress-test:test12" already present on machine
  Warning  Unhealthy  8m17s (x35 over 18m)  kubelet, gg-k8s02-worker02  Liveness probe failed: Get http://10.244.3.83/: dial tcp 10.244.3.83:80: connect: connection refused
  Warning  BackOff    3m27s (x27 over 11m)  kubelet, gg-k8s02-worker02  Back-off restarting failed container

POD LOGS:

WordPress not found in /var/www/html - copying now...
WARNING: /var/www/html is not empty! (copying anyhow)
Complete! WordPress has been successfully copied to /var/www/html
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.244.3.83. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.244.3.83. Set the 'ServerName' directive globally to suppress this message
[Wed Dec 11 06:39:07.502247 2019] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.3.11 configured -- resuming normal operations
[Wed Dec 11 06:39:07.502323 2019] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
10.244.3.1 - - [11/Dec/2019:06:39:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:39:33 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:39:48 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:40:03 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
10.244.3.1 - - [11/Dec/2019:06:40:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"

DOCKERFILE ("wordpress-test:test12"):

FROM wordpress:5.2.4-apache

RUN sed -i 's/Listen 80/Listen 8080/g' /etc/apache2/ports.conf;
RUN sed -i 's/:80/:8080/g' /etc/apache2/sites-enabled/000-default.conf;
# RUN sed -i 's/#ServerName www.example.com/ServerName localhost/g' /etc/apache2/sites-enabled/000-default.conf;

EXPOSE 8080

CMD ["apache2-foreground"]

DEPLOYMENT:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog-wordpress
  namespace: development
  labels:
    app: blog

spec:
  selector:
    matchLabels:
      app: blog
      tier: wordpress
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 2
  template:
    metadata:
      labels:
        app: blog
        tier: wordpress
    spec:
      volumes:
        - name: blog-wordpress
          persistentVolumeClaim:
            claimName: blog-wordpress
      containers:
        - name: blog-wordpress
          # image: wordpress:5.2.4-apache
          image: wordpress-test:test12
          securityContext:
            runAsUser: 65534
            allowPrivilegeEscalation: false
            capabilities:
              add:
                - "NET_ADMIN"
                - "NET_BIND_SERVICE"
                - "SYS_TIME"
          resources:
            requests:
              cpu: "250m"
              memory: "64Mi"
            limits:
              cpu: "500m"
              memory: "128Mi"
          ports:
            - name: liveness-port
              containerPort: 8080
          readinessProbe:
            initialDelaySeconds: 15
            httpGet:
              path: /index.php
              port: 8080
            timeoutSeconds: 15
            periodSeconds: 15
            failureThreshold: 5
          livenessProbe:
            initialDelaySeconds: 10
            httpGet:
              path: /index.php
              port: 8080
            timeoutSeconds: 10
            periodSeconds: 15
            failureThreshold: 5
          env:
            # Database
            - name: WORDPRESS_DB_HOST
              value: blog-mysql
            - name: WORDPRESS_DB_NAME
              value: wordpress
            - name: WORDPRESS_DB_USER
              valueFrom:
                secretKeyRef:
                  name: blog-mysql
                  key: username
            - name: WORDPRESS_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: blog-mysql
                  key: password
            - name: WORDPRESS_TABLE_PREFIX
              value: wp_
            - name: WORDPRESS_AUTH_KEY
              valueFrom:
                secretKeyRef:
                  name: blog-wordpress
                  key: auth-key
            - name: WORDPRESS_SECURE_AUTH_KEY
              valueFrom:
                secretKeyRef:
                  name: blog-wordpress
                  key: secure-auth-key
            - name: WORDPRESS_LOGGED_IN_KEY
              valueFrom:
                secretKeyRef:
                  name: blog-wordpress
                  key: logged-in-key
            - name: WORDPRESS_NONCE_KEY
              valueFrom:
                secretKeyRef:
                  name: blog-wordpress
                  key: nonce-key
            - name: WORDPRESS_AUTH_SALT
              valueFrom:
                secretKeyRef:
                  name: blog-wordpress
                  key: auth-salt
            - name: WORDPRESS_SECURE_AUTH_SALT
              valueFrom:
                secretKeyRef:
                  name: blog-wordpress
                  key: secure-auth-salt
            - name: WORDPRESS_LOGGED_IN_SALT
              valueFrom:
                secretKeyRef:
                  name: blog-wordpress
                  key: logged-in-salt
            - name: WORDPRESS_NONCE_SALT
              valueFrom:
                secretKeyRef:
                  name: blog-wordpress
                  key: nonce-salt
            - name: WORDPRESS_CONFIG_EXTRA
              value: |
                define('WPLANG', 'fr_FR');
                define('WP_CACHE', false);
                define('WP_MEMORY_LIMIT', '64M');
          volumeMounts:
            - name: blog-wordpress
              mountPath: "/var/www/html/wp-content"

DEPLOYMENT SERVICE:

apiVersion: v1
kind: Service
metadata:
  name: blog-wordpress
  namespace: development
  labels:
    app: blog

spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  selector:
    app: blog
    tier: wordpress
  type: ClusterIP

TRAEFIK INGRESSROUTE:

##
# HTTP
##

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: blog
  namespace: development
spec:
  entryPoints:
    - http
  routes:
  - match: Host(`example.com`)
    kind: Rule
    services:
    - name: blog-wordpress
      port: 80
    middlewares:
      - name: redirect-to-https
        namespace: kube-system

---

##
# HTTPS
##

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: blog-https
  namespace: development
spec:
  entryPoints:
    - https
  routes:
  - match: Host(`example.com`) && PathPrefix(`/`)
    kind: Rule
    services:
    - name: blog-wordpress
      port: 80

  tls:
    certResolver: letsencrypt

Thank you!

3
wordpressapachedockerkubernetestraefik
From your Pod event i can see that the probe is being checked on port 80 (dial tcp 10.244.3.83:80: connect: connection refused) while in deployment it`s shows 8080. Can you verify that? – acid_fuji

3 Answers

2
votes

For anyone interested I've managed to solve this issue.

I was getting a 301 redirect response from Wordpress due to Wordpress forcing my domain name example.com. Solved this issue by disabling Wordpress canonical redirection feature for the specific request http://POD_IP:8080/index.php.

Here's how:

Added the Pod IP address as an environment variable:

- name: K8S_POD_IP
  valueFrom:
    fieldRef:
      fieldPath: status.podIP

Created a Wordpress plugin with a custom redirect_canonical filter that prevents Wordpress from redirecting http://POD_IP:8080/index.php:

<?php
/**
 * Plugin Name: Kubernetes Liveness Probe Exception
 */

add_filter('redirect_canonical', function($redirect_url, $requested_url) {
    $K8S_POD_IP = getenv('K8S_POD_IP');
    $LIVENESS_URL = "http://" . $K8S_POD_IP . ":8080/index.php";

    if ($requested_url == $LIVENESS_URL) {
        return $requested_url;
    }

    return $redirect_url;
}, 10, 2);
1
votes
10.244.3.1 - - [11/Dec/2019:06:39:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"

You're getting a 301 redirect response from Apache. You need to be getting a 2xx to be considered a success.

To check what path it's redirecting you to try curl --location --verbose http://url/index.php

If you can't figure out a way around Apache or Wordpress's redirection, you could consider a tcpSocket probe rather than httpGet

0
votes

I'd think WP redirects you to a “clean” url of /. Remove the index.php part