Cluster communication and firewalls in Google Container Engine

Question

I'm trying to set up the following environment on Google Cloud and have 3 major problems with it:

Database Cluster

3 nodes
one port open to world, a few ports open to the compute cluster

Compute Cluster - 5 nodes - communicated with the database cluster - two ports open to the world - runs Docker containers

a) The database cluster runs fine, I have the configuration port open to world, but I don't know how to limit the other ports to only the compute cluster?

I managed to get the first Pod and Replication-Controller running on the compute cluster and created a service to open the container to the world:

controller:

{
  "id": "api-controller",
  "kind": "ReplicationController",
  "apiVersion": "v1beta1",
  "desiredState": {
    "replicas": 2,
    "replicaSelector": {
      "name": "api"
    },
    "podTemplate": {
      "desiredState": {
        "manifest": {
          "version": "v1beta1",
          "id": "apiController",
          "containers": [{
            "name": "api",
            "image": "gcr.io/my/api",
            "ports": [{
              "name": "api",
              "containerPort": 3000
            }]
          }]
        }
      },
      "labels": {
        "name": "api"
      }
    }
  }
}

service:

{
  "id": "api-service",
  "kind": "Service",
  "apiVersion": "v1beta1",
  "selector": {
    "name": "api"
  },
  "containerPort": "api",
  "protocol": "TCP",
  "port": 80,
  "selector": { "name": "api" },
  "createExternalLoadBalancer": true
}

b) The container exposes port 3000, the service port 80. Where's the connection between the two?

The firewall works with labels. I want 4-5 different pods running in my compute cluster with 2 of them having open ports to the world. There can be 2 or more containers running on the same instance. The labels however are specific to the nodes, not the containers.

c) Do I expose all nodes with the same firewall configuration? I can't assign labels to containers, so not sure how to expose the api service for example?

Is there a reason you are running two clusters? It seems like it would be easier to run them in the same cluster with resource limits and not have to worry too much about the networking. — Ian Lewis
I need various machine times (large for database, medium/small for services), Container Engine doesn't have that feature. — Patrick
I would just use large size for the cluster and run everything in that cluster (per my answer) since that's the easiest approach. — Ian Lewis
Putting everything on large machines/instances would be unnecessarily expensive in running the cluster. Given there's a bunch of small node.js scripts running against a big database cluster. — Patrick

Ian Lewis Ian Lewis · Accepted Answer · 2015-08-03T06:34:29

I'll try my best to answer all of your questions as best I can.

First off, you will want to upgrade to using v1 of the Kubernetes API because v1beta1 and v1beta3 will no longer be available after Aug. 5th: https://cloud.google.com/container-engine/docs/v1-upgrade

Also, Use YAML. It's so much less verbose ;)

--

Now on to the questions you asked:

a) I'm not sure I completely understand what you are asking here but it sounds like running the services in the same cluster (with resource limits) would be way easier than trying to deal with cross cluster networking.

b) You need to specify a targetPort so that the service knows what port to use on the container. This should match port 3000 that you have in your resource controller. See the docs for more info.

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata: {
    "labels": [{
      "name": "api-service"
    }],
  },
  "spec": {
    "selector": {
      "name": "api"
    },
    "ports": [{
      "port": 80,
      "targetPort": 3000
    }]
    "type": "LoadBalancer" 
  }
}

c) Yes. In Kubernetes the kube-proxy accepts traffic on any node and routes it to the appropriate node or local pod. You don't need to worry about mapping the load balancer to, or writing firewall rules for those specific nodes that happen to be running your pods (it could actually change if you do a rolling update!). kube-proxy will route traffic to the right place even if your service is not running on that node.

Cluster communication and firewalls in Google Container Engine

1 Answers