Running Meteor app with Nginx SSL proxy on Kubernetes

Question

I have a Meteor app deployed using Kubernetes on Google Cloud, configured with Nginx acting as SSL termination. Everything working ok.

However, it appears that if two different clients connect to two different SSL containers, updates don't show up on the respective apps for up to 10 seconds, which makes it seem that Websockets isn't working, but polling is taking effect. I have confirmed that all clients are connected with Websockets, but since updates do not propagate immediately, perhaps Nginx isn't configured to correctly talk with the Meteor app.

Here's my SSL/Nginx service:

apiVersion: v1 kind: Service metadata: name: frontend-ssl labels: name: frontend-ssl spec: ports: - name: http port: 80 targetPort: 80 - name: https port: 443 targetPort: 443 selector: name: frontend-ssl type: LoadBalancer loadBalancerIP: 123.456.123.456 sessionAffinity: ClientIP

And here is the Meteor service:

apiVersion: v1 kind: Service metadata: name: frontend labels: name: frontend spec: ports: - port: 3000 targetPort: 3000 selector: name: flow-frontend type: LoadBalancer loadBalancerIP: 123.456.123.456 sessionAffinity: ClientIP

For SSL termination, I'm using the Kubernetes suggested SSL setup forked with Websockets additions https://github.com/markoshust/nginx-ssl-proxy

You're pointing 123.456.123.456 at both Services (or is that a typo)? I'm not familiar with meteor, but there's a chance your problem is not a meteor problem. You should have at least 2 pods (preferably managed by 2 RCs), and 2 Services, one for nginx and one for your app. Only the nginx Service needs to be Type=Loadbalacer. — Prashanth B
To narrow down the unknowns, you can force the pods to land on the same node using node selectors (github.com/kubernetes/kubernetes/blob/release-1.1/docs/…). You can also point an nginx instance directly at the service with something like proxy_pass http://svcName.svcNamespace.svc.cluster.local:svcPort in your config (assuming you have cluster DNS working and can resolve your service name with eg nslookup). Let me know how this goes. — Prashanth B
Nginx and Meteor both have separate rcs and pods. I figured only nginx needs Type=LoadBalancer, however I don't think having that in the meteor side should affect anything. They do live on the same IP, however that just makes both publicly available, each on their own port. — Mark Shust at M.academy
In response to your second comment, I don't want pods landing on the same instances, because that sort of defeats the purpose of running kubernetes. I want to potentially run my meteor app on 100 pods located across 100 different instances. I'm in fact using proxy_pass http://target_service:port to route requests from nginx to meteor. — Mark Shust at M.academy
FYI, I did remove type=LoadBalancer from my meteor service. Things run the same, however. — Mark Shust at M.academy

MrE MrE · Accepted Answer · 2016-04-02T17:00:53

In your NginX config, did you make sure to use the ip_hash flag to direct websockets to the same server each time? also you need to make sure the websocket Upgrade headers are forwarded:

upstream meteorapp{
     ip_hash;
     server   hostname:port
}
server {
    # your server stuff here
    # 
    location / {
        proxy_pass                  http://meteorapp;
        proxy_set_header            Host $host;
        proxy_set_header            X-Real-IP $remote_addr;
        proxy_http_version          1.1;
        proxy_set_header            Upgrade $http_upgrade;
        proxy_set_header            Connection "upgrade";

        proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header            X-Forwarded-Proto http;
        proxy_redirect              http:// $scheme://;
    }
}

Running Meteor app with Nginx SSL proxy on Kubernetes

2 Answers