In Google cloud dataflow/Beam we can collect application metrics using this Metrics API and it can be queried in the application using another API. But I need to persist these metrics in Bigquery/Prometheus for the realtime monitoring and alerting mechanism. Please suggest how to achieve this in Google cloud dataflow.
1 Answers
1
votes
The best way of collecting metrics from Dataflow is by using Stackdriver Monitoring, which allows for access Dataflow job metrics such as Job Status, Element Counts, System Lag (for streaming jobs), and User Counters from the Stackdriver dashboards. Additional advantage of Stackdriver is alerting capabilities, which will notify you of various conditions, such as long streaming system lag or failed jobs.
Here is a handy list of metrics available for Dataflow, e.g. job/is_failed — easy to set up alerts on failed jobs.
If you prefer a graphical UI for monitoring your metrics, check out this blog post.
Please, be aware that product is in Beta state. I hope you find the above pieces of information useful.
