Dataflow pipelines failing due to "staged package jar"

Question

A dataflow job (id 2017-01-15_19_15_22-2979660111155895381) failed with the following error, and we have no clue why it happened. We run ~100 dataflow jobs daily from the same machine, and around the same time (both before and after) other jobs launched successfully, so I don't think it is due to the google cloud service account (although the error message suggests so). Will the dataflow team be able to dig and let us know what happened?

(ce2f98d99f483fed): Staged package commons-codec-1.6-WXD1SIO0gxskuX8RJbon5g.jar at location 'gs://.../commons-codec-1.6-WXD1SIO0gxskuX8RJbon5g.jar' is inaccessible.
2017-01-15 (19:15:30) Workflow failed. Causes: (ce2f98d99f48346e): One or more access checks for temp location or staged f...
(edd80087cf3fc85d): Workflow failed. Causes: (ce2f98d99f48346e): One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions.

edit : in the past 48 hours, we've seen 5-6 different jobs failing because of a similar issue (staged jar file is not accessible or finalizing 'the write step to gcs' failed). I couldn't find anything related to this on SO other than an old post (Google Cloud Dataflow jobs failing, inaccessible jars & 410 gone errors).

We've had these (transient) errors in the past too. It's normally signifies a problem with GCS. What we tend to do when it comes up, is to delete all the files from the staging bucket and upload everything again. You could try that and see if it helps. — Graham Polley
Thanks for sharing your experience. It appears that we experienced the same issue, and after cleaning up the staging bucket, the issue went away. — Haden Hooyeon Lee

Graham Polley Graham Polley · Accepted Answer · 2017-01-21T22:01:18

This is a transient error that we've come across with GCS too. As a workaround, what you can do is delete all the contents of your staging bucket and upload everything again.

We perform a clean out of the bucket with each new release of our pipeline(s) even though this should not be necessary.

Dataflow pipelines failing due to "staged package jar"

1 Answers