GCP Dataflow logging to fluentd

Question

We would like to consolidate our logs from our application and GCP services into one logging system. The applications running in K8s forward to a fluentd instance, which then passes the logs on to our Logging system. (Humio). For GCP services, especially Dataflow, the logging is towards Stackdriver. (Google cloud logging).

An experiment to have a custom logback appender in Dataflow didn't succeed. DF apparently doesn't allow custom appenders.

What are the options for streaming stackdriver logs outbound to a fluentd server? Anyone experience with this?

Thank You

Answer 1

Your best bet is going to be to create something like a PubSub sink for those logs and have something else pull those logs from the sink and write them to your ultimate destination, I think.