GCP Dataflow Streaming Job getting Processing stuck in step s03 for at least 05m00s without outputting or completing in state finish

0
votes

I am getting this error daily in my python dataflow job.

I am using Apache beam 2.15 (same with 2.17 as well) Python 3.7.

enter image description here

2020-01-28 17:08:53.801 GMTError message from worker: Processing stuck in step s03 for at least 10m00s without outputting or completing in state finish at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57) at org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285) at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85) at org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1295) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:149) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1028) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

1
python-3.xgoogle-cloud-platformgoogle-cloud-dataflowapache-beamdataflow
Take a look at this post stackoverflow.com/questions/54990412/… – rmesteves

1 Answers

0
votes

It is useful to note that the Processing stuck messages do not indicate that your job is stuck. Rather, they indicate that the Dataflow worker has been performing a single operation for over 5 minutes.

If you see this message every now and then, but your pipeline continues to make progress overall, then you've got nothing to worry about.

If you see this message continuously, and your pipeline stops making any progress at all, then that's an indication of actual stuckness.