Diagnosing failed Cloud Dataflow pipeline

5
votes

I had a Cloud Dataflow pipeline fail after about 14 worker-hours with the following cryptic log message:

Mar 29, 2016, 8:18:16 PM (3253bcfbb8c9c2a7): Workflow failed. Causes: (2bfe8449fe3ba464): S745 (STAGE REDACTED) Causes: (1a6d5387c382ba3a): A work item was attempted 4 times without success. Each time the worker eventually lost contact with the service. The work item was attempted on: (WORKERS REDACTED)

I glanced quickly at worker logs and it wasn't immediately obvious what was happening either. Is there supposed to be something to those cause codes?

The troubleshooting guide wasn't particular elucidating here, either. My best guess was that it fell under the "shuffle-bound" category (the job is very shuffle intensive), but none of the errors listed are present in the logs.

Thanks!

1
google-cloud-dataflow

1 Answers

4
votes

I looked up your job by the error IDs and it seems that the work items were repeatedly failing due to out of memory errors (the Java process was killed by OOM killer, unfortunately without getting a chance to write a heap dump - search for "oom-killer" in the cloud logs to find the relevant entries).

Unfortunately all I can suggest with this information is, consider using a bigger instance type or optimizing memory usage of your transforms (e.g. make sure they're not buffering a lot of data in memory).