Can't nginx proxy pass to kibana in kubernetes

1
votes

I'm trying to use a proxy pass with nginx to a Kibana pod using a basic auth.

Worked for testing (it's another k8s cluster, but pretty similar, using same namespace, kube-dns, env inside the pods matches and they see each other) Context: I deploy this via helm at k8s in AWS, the nginx has a Kubernetes LB service type (which basically it's an ELB at AWS with its cname at route53).

If I point nginx pod to kibana-app.kube-system.svc.cluster.local:5601 I see the request at kibana pod from nginx, but returning 404 while trying to go to server.basePath: /api/v1/proxy/namespaces/kube-system/services/kibana-app/

I can access kibana-app pod by getting the url from "kubectl cluster-info" and then checking the logs, the request goes like this:

"method":"get","statusCode":200,"req":{"url":"/app/kibana"
"x-forwarded-uri":"/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana

Can't find what's going wrong while trying to reach Kibana path from nginx (after doing a basic auth)

    server {
    listen       80;
    server_name  localhost;

    access_log  /var/log/nginx/host.access.log;

    location / {
        auth_basic "simple auth";
        auth_basic_user_file /var/kibana_config/htpasswd;
        try_files KIBANA @kibana-app;
    }

    location @kibanaapp {
        return 301 http://kiban-app-url-from-route53/server.basePath;
    }

    location /api {
            proxy_pass https://api.awszone.mydomain/api;
        proxy_set_header Authorization "Basic ";
    }
}

Also tried to move the proxy_pass statement, removing the return and just doing a proxy_pass from where kibana's pod is listening but either doesn't work, the request never gets to the pod or when the request gets to kibana-app pod, it returns a 404.

Any thoughts?

Thanks!

Update :

I'm almost there, now I can see the "kibana is loading screen" but never finish loading the bundles, json and stuff, nginx pod log:

GET /api/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/commons.style.css

same request at kibana pod returning 404:

"statusCode":404,"req":{"url":"/app/kibana/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/commons.bundle.js?v=10146","method":"get","headers":{"host":"kibana.app.env.com","referer":"http://kibana.app.env.com/api "referer":"http://kibana.app.env.com/api"},"res":{"statusCode":404,"responseTime":2,"contentLength":9},"message":"GET /app/kibana/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/commons.bundle.js?v=10146

my nginx conf:

server {
    listen 80;
    server_name localhost;
    access_log  /var/log/nginx/host.access.log;

    location / {
        auth_basic "simple auth";
        auth_basic_user_file /var/kibana_config/htpasswd;
        try_files KIBANA @kibana-app;
    }

    location @kibana-app {
        return 301 kibana.app.env.com/server.basePath;
    }

    location /api {
        proxy_pass http://kibana-logging.kube-system.svc.cluster.local:5601;
        proxy_set_header HOST $host;
        proxy_set_header Referer $http_referer;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Authorization "simple auth ";
    }

}

"kibana.app.env.com" it's just the FQDN that kubernetes creates at route53 as a CNAME to an ELB which hits the nodes from where nginx/kibana pods are. That's the url I use at the browser and it should reach nginx, ask me for basic authorization and then take me to kibana pod with server.basePath: /api/v1/proxy/namespaces/kube-system/services/kibana-logging Please, ask me something if I'm not being clear, sorry that I can't just copy/paste everything.

2
nginxkuberneteskibana

2 Answers

0
votes

Not sure how this is working on the other cluster. So the base path that you mentioned: /api/v1/proxy/namespaces/kube-system/services/kibana-app/ seems like a kube-apiserver base path, and that's the path that a proxy setup using kubectl proxy would do to talk to your applications and services in the cluster.

If you really want to talk from nginx to Kibana inside the cluster you would have to add the kibana-app.kube-system.svc.cluster.local:5601 endpoint to your nginx backend.

0
votes

Finally, it's working:

    server {
        listen 80;
        server_name localhost;
        access_log  /var/log/nginx/host.access.log;

        location / {
            auth_basic "simple auth";
            auth_basic_user_file /var/kibana_config/htpasswd;
            try_files KIBANA @kibana-app;
        }

        location @kibana-app {
            return 301 /api/v1/proxy/namespaces/kube-system/services/kibana-logging/;
        }

        location /api/v1/proxy/namespaces/kube-system/services/kibana-logging/ {
            proxy_set_header Authorization "simple auth ";
            proxy_pass http://kibana-logging.kube-system.svc.cluster.local:5601/;
            proxy_set_header HOST $host;
            proxy_set_header Referer $http_referer;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_cache_bypass $http_upgrade;
        }
}

Going to the URL that K8s created at AWS as an ELB (kibana-app.env.com) redirects to /api/v1/proxy/namespaces/kube-system/services/kibana-logging/ which proxy_pass to kibana pod : http://kibana-logging.kube-system.svc.cluster.local:5601