I have configured ElasticSearch on a Kubernetes cluster. In Kubernetes cluster for application, I have fluentd configured, using THIS helm chart, with the following parameters:
spec:
containers:
- env:
- name: FLUENTD_ARGS
value: --no-supervisor -q
- name: OUTPUT_HOST
value: x.x.x.x
- name: OUTPUT_PORT
value: "80"
- name: OUTPUT_PATH
value: /elastic
- name: LOGSTASH_PREFIX
value: logstash
- name: OUTPUT_SCHEME
value: http
- name: OUTPUT_SSL_VERIFY
value: "false"
- name: OUTPUT_SSL_VERSION
value: TLSv1_2
- name: OUTPUT_TYPE_NAME
value: _doc
- name: OUTPUT_BUFFER_CHUNK_LIMIT
value: 2M
- name: OUTPUT_BUFFER_QUEUE_LIMIT
value: "8"
- name: OUTPUT_LOG_LEVEL
value: info
In ElasticSearch cluster I have nginx-ingress controller configured and I want fluentd to send logs to Elasticsearch via this nginx ingress. In "OUTPUT_HOST" I am using nginx-ingress public IP. In "OUTPUT_PORT" I have used "80" as nginx is listening on 80.
I am getting the following error in fluentd:
2019-11-06 07:16:46 +0000 [warn]: [elasticsearch] failed to flush the buffer.
retry_time=40 next_retry_seconds=2019-11-06 07:17:18 +0000 chunk="596a7f6afffad60f2b28a5e13f"
error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster (
{:host=>\"x.x.x.x\", :port=>80, :scheme=>\"http\", :path=>\"/elastic\"}): [405]
{\"error\":\"Incorrect HTTP method for uri [/] and method [POST], allowed: [HEAD, GET, DELETE]\",\"status\":405}"
I can guess by looking at the log, it is considering "/elastic" as index.
A mentioned HERE is used the annotation "nginx.ingress.kubernetes.io/rewrite-target: /" but problem persists.
After this, I changed nginx-ingress to listen to calls at "/" instead of "/elastic". changed "OUTPUT_PATH" in fluentD config too.
I could see the error I was getting earlier is gone but I would still like to use "/elastic" instead of "/". I am not sure what nginx config I need to change to achieve this. Please help me here.
After this, I got a "request entity too large" error which was resolved by adding - "nginx.ingress.kubernetes.io/proxy-body-size: 100m" in annotations. By default its 1M and for fluentD by default it's 2M. It was bound to fail.
Now I am getting errors like:
2019-11-06 10:01:08 +0000 [warn]: dump an error event:
error_class=Fluent::Plugin::ConcatFilter::TimeoutError error="Timeout flush: kernel:default" location=nil tag="kernel"
time=2019-11-06 10:01:08.267224927 +0000 record=
{
"transport"=>"kernel",
"syslog_facility"=>"0",
"syslog_identifier"=>"kernel",
"boot_id"=>"6e4ca7b1c1a11b74151a12979",
"machine_id"=>"89436ac666fa120304f2077f3bf2",
"priority"=>"6",
"hostname"=>"gke-dev--node-pool",
"message"=>"
cbr0: port 9(vethe75a241b) entered disabled statedevice vethe75a241b left promiscuous mode
cbr0: port 9(vethe75a241b) entered disabled state
IPv6: ADDRCONF(NETDEV_UP): veth630f6cb0: link is not ready
IPv6: ADDRCONF(NETDEV_CHANGE): veth630f6cb0: link becomes ready
cbr0: port 9(veth630f6cb0) entered blocking state
cbr0: port 9(veth630f6cb0) entered disabled statedevice veth630f6
cb0 entered promiscuous mode
cbr0: port 9(veth630f6cb0) entered blocking state
cbr0: port 9(veth630f6cb0) entered forwarding state",
"source_monotonic_timestamp"=>"61153348174"
}
