K8S NGINX ingress reverse proxy rules for kibana etc

Question

I know there are lots of questions on this topic but none so far have helped. The #1 issue is my own understanding. I've set up an Nginx Ingress Controller in K8S and am now trying to add the ingress rules for a Kibana cluster service, which is working fine with port forwarding.

I cannot get the reverse proxy to work at all. I would appreciate 2 things:

1. The K8S ingress rules to get this to work
2. A really good "dummies" guide to setting up reverse proxies via ingress. Most guides I find are "this is a RP, hey, we can redirect /app1 to /app2... isn't that great?" They really don't have any detail beyond the very basic. Alternatively, they are NGINF conf based and this doesn't map to K8S ingress rules in the same way

What's happening is that I can get the browser /kibana to redirect to my K8S service but then Kibana replies with it's own redirects to /app/home and then the process breaks down with a 404 (instead if /kibana/app/home as it needs to be). I don't know how to handle this flow in the ingress rules - it's not as simple as redirect X to Y.

I've tried a number of ingress rules, but it's basically infinite monkeys until something works. I really want to understand it properly but would appreciate an answer to this specific issue too.

Ingress so far

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-kibana
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /kibana(/|$)(.*)
        pathType: Prefix
        backend:
          serviceName: elasticsearch-kibana
          servicePort: 5601

K8S 1.18.3

Using the bitnami charts, I've created kibana (and ES) via Helm in my logging namespace:

helm install elasticsearch bitnami/elasticsearch --set sysctlImage.enabled=false,master.replicas=1,coordinating.replicas=1,data.replicas=1,ingest.replicas=1,global.kibanaEnabled=true -n logging

My ingress controller also via helm in my ingress namespace

helm install ingress bitnami/nginx-ingress-controller  -n ingress

The ingress rules applies in the same logging namespace as kibana

kubectl apply -f ingress-rules.yaml -n logging

Answer 1

I managed to make it work with server.basePath parameter.

Here is what the docs say about it:

basePath enables you to specify a path to mount Kibana at if you are running behind a proxy. Use the server.rewriteBasePath setting to tell Kibana if it should remove the basePath from requests it receives, and to prevent a deprecation warning at startup. This setting cannot end in a slash (/).

Also, since you are doing the rewrite at the ingress level I also had to disabled the rewriteBasePath (it should be false be default of kibana 7 and older). Now the complete config looks like the following:

  kibana.yml: |
    pid.file: /opt/bitnami/kibana/tmp/kibana.pid
    server.host: "::"
    server.port: 5601
    elasticsearch.hosts: [http://elasticsearch-coordinating-only:9200]
    server.basePath: /kibana 
    server.rewriteBasePath: false

As you can see below, the request is not redirected to /app/home but to /kibana/app/home instead, which is exactly what we want:

➜  curl $(minikube ip)/kibana -v      
*   Trying 192.168.49.2...
* TCP_NODELAY set
* Connected to 192.168.49.2 (192.168.49.2) port 80 (#0)
> GET /kibana HTTP/1.1
> Host: 192.168.49.2
---
< HTTP/1.1 302 Found
< Date: Mon, 22 Feb 2021 11:19:45 GMT
----
< location: /kibana/app/home
< kbn-name: elasticsearch-kibana-7f9f447b9c-mnxrs
----

Answer 2

Find and change file kibana.yml

server.basePath: /your/path

kibana.yml: |
    server.basePath: /dashboards/kibana
    server.rewriteBasePath: true
    elasticsearch.hosts: http://elasticsearch-svc:9200
    server.host: "0.0.0.0"

    logging.silent: false
    logging.quiet: true
    logging.verbose: false


    xpack:    
      security:
        enabled: false
      infra: 
        sources:
          default:
            logAlias: "logstash-*"
            fields:
              timestamp: "@timestamp"
              message: ["message"]
              host: "host"
              container: "syslog_program"

psicopante