PUT Elasticsearch Ingest Pipeline by default

Question

We currently use Elasticsearch for storage of Spring Boot App logs that are sent by Filebeat and use Kibana to visualise this.

Our entire architecture is dockerized inside a docker-compose file. Currently, our when we start the stack, we have to wait for Elasticsearch to start, then PUT our Ingest Pipeline, then restart Filebeat, and only then do our logs show up properly ingested in Kibana.

I'm quite new to this, but I was wondering if there is no way to have Elasticsearch save ingest pipelines so that you do not have to load them every single time? I read about mounting volumes or running custom scripts to wait for ES and PUT when ready, but all of this seems very cumbersome for a use case that to me seems like the default?

Answer 1

We used a similar approach to ozlevka, by running a script during the build process of our custom Elasticsearch image.

This is our script:

#!/bin/bash
# This script sets up the Elasticsearch docker instance with the correct pipelines and templates

baseUrl='localhost:9200'
contentType='Content-Type:application/json'

# filebeat
ingestUrl=$baseUrl'/_ingest/pipeline/our-pipeline?pretty'
payload='/usr/share/elasticsearch/config/our-pipeline.json'

/usr/share/elasticsearch/bin/elasticsearch -p /tmp/pid > /dev/null & 
# wait until Elasticsearch is up
# you can get logs if you change /dev/null to /dev/stderr
while [[ "$(curl -s -o /dev/null -w ''%{http_code}'' -XPUT $ingestUrl -H$contentType -d@$payload)" != "200" ]]; do
    echo "Waiting for Elasticsearch to start and posting pipeline..."
    sleep 5
done

kill -SIGTERM $(cat /tmp/pid) 
rm /tmp/pid
echo -e "\n\n\nCompleted Elasticsearch Setup, refer to logs for details"

Answer 2

I suggest using the startscript into filebeat container.

The script will ping elasticsearch be ready, after this create pipeline and start filebeat.

#!/usr/bin/env bash -e

START_FILE=/tmp/.es_start_file
http () {
    local path="${1}"
    curl -XGET -s -k --fail http://${ELASTICSEARCH_HOST}:{$ELASTICSEARCH_PORT}${path}
}

pipeline() {
    curl -XPUT -s -k --fail http://${ELASTICSEARCH_HOST}:{$ELASTICSEARCH_PORT}/_ingest/pipeline/$PIPELINE_NAME -d @pipeline.json
}

while true; do
    if [ -f "${START_FILE}" ]; then
        pipeline
        /usr/bin/filebeat -c filebeat.yaml &
        exit 0    
    else
        echo 'Waiting for elasticsearch cluster to become green'
        if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
            touch ${START_FILE}
        fi    
    fi
done

This method will be good for docker-compose and docker swarm. For k8s preferable create readiness probe.