How to implement logging with Fluentd/Elasticsearch/Ikebana on Kubernetes 1.1?

Question

I am trying to implement logging on my AWS/CoreOS/Kubernetes (1.1) setup, which I've set up without kube-up. So far I've installed fluentd as a static pod on all nodes and the Fluentd-Elasticsearch addon replication controllers and services. However, it doesn't yet work. Specifically, Kibana crashes like this:

ELASTICSEARCH_URL=http://elasticsearch-logging.kube-system:9200
{"@timestamp":"2016-03-14T22:54:04.906Z","level":"error","message":"Service Unavailable","node_env":"production","error":{"message":"Service Unavailable","name":"Error","stack":"Error: Service Unavailable\n  at respond (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:235:15)\n  at checkRespForFailure (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:203:7)\n  at HttpConnector.<anonymous> (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/connectors/http.js:156:7)\n  at IncomingMessage.bound (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)\n  at IncomingMessage.emit (events.js:117:20)\n  at _stream_readable.js:944:16\n  at process._tickCallback (node.js:442:13)\n"}}
{"@timestamp":"2016-03-14T22:54:04.908Z","level":"fatal","message":"Service Unavailable","node_env":"production","error":{"message":"Service Unavailable","name":"Error","stack":"Error: Service Unavailable\n  at respond (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:235:15)\n  at checkRespForFailure (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:203:7)\n  at HttpConnector.<anonymous> (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/connectors/http.js:156:7)\n  at IncomingMessage.bound (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)\n  at IncomingMessage.emit (events.js:117:20)\n  at _stream_readable.js:944:16\n  at process._tickCallback (node.js:442:13)\n"}}

What should I do?

FWIW, Elasticsearch is connectable at http://elasticsearch-logging.kube-system:9200/, although it does return status 503. That could be the issue for all I know.

# curl http://elasticsearch-logging.kube-system:9200/
{
  "status" : 503,
  "name" : "Puppet Master",
  "cluster_name" : "kubernetes-logging",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
    "build_timestamp" : "2015-04-27T09:21:06Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

fluentd-es.yaml

apiVersion: v1
kind: Pod
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
spec:
  containers:
  - name: fluentd-elasticsearch
    image: gcr.io/google_containers/fluentd-elasticsearch:1.13
    resources:
      limits:
        cpu: 100m
    args:
    - -q
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
  terminationGracePeriodSeconds: 30
  volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers

es-controller.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: elasticsearch-logging-v1
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 2
  selector:
    k8s-app: elasticsearch-logging
    version: v1
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - image: gcr.io/google_containers/elasticsearch:1.7
        name: elasticsearch-logging         
        resources:
          limits:
            cpu: 100m
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: es-persistent-storage
          mountPath: /data
      volumes:
      - name: es-persistent-storage
        emptyDir: {}

es-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "Elasticsearch"
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch-logging

kibana-controller.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: kibana-logging-v1
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kibana-logging
    version: v1
  template:
    metadata:
      labels:
        k8s-app: kibana-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: kibana-logging
        image: gcr.io/google_containers/kibana:1.3
        resources:
          limits:
            cpu: 100m
        env:
          - name: "ELASTICSEARCH_URL"
            value: "http://elasticsearch-logging.kube-system:9200"
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP

kibana-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "Kibana"
spec:
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging

Answer 1

The 503 from Elasticsearch is likely the issue. It should be returning a 200 if all is well. Your first step should be to look at the Elasticsearch logs. You can do this with the kubectl logs POD command. Your es-controller and es-service YAML appears to be correct.

One thing to note is that the fluentd-elasticsearch container you are using provides a configuration suitable for systems using syslog, whereas CoreOS uses systemd/journald. That configuration will likely still provide you logs from your docker containers, as long as you are using the default json-file log driver, but will not provide system logs. To get the system logs, you have to use other means, e.g., https://github.com/reevoo/fluent-plugin-systemd or https://github.com/mheese/journalbeat.