How can we visualize the Dataproc job status in Google Cloud Plarform?

0
votes

How can we visualize (via Dashboards) the Dataproc job status in Google Cloud Platform?

We want to check if jobs are running or not, in addition of their status like running, delay, blocked. On top of it we want to set alerting (Stackdriver Alerting) as well.

1
google-cloud-platformgoogle-cloud-dataprocstackdrivergoogle-cloud-stackdriver

1 Answers

0
votes

In this page, you have all the metrics available in Stackdriver

https://cloud.google.com/monitoring/api/metrics_gcp#gcp-dataproc

You could use cluster/job/submitted_count, cluster/job/failed_count and cluster/job/running_count to create the dashboard and metrics

Also, you could use cluster/job/completion_time to warn about long-running jobs and cluster/job/duration to check if jobs are enqueued in PENDING status for a long time.