Apache Beam: PubsubReader fails with NPE

Question

I have a a beam pipeline that reads from PubSub and write to BigQuery after applying some transformation. The pipeline fails consistently with a NPE. I am using beam SDK version 0.6.0. Any Idea on what I could be doing wrong? I am trying to run the pipeline with a DirectRunner.

java.lang.NullPointerException
at org.apache.beam.sdk.io.PubsubUnboundedSource$PubsubReader.ackBatch(PubsubUnboundedSource.java:640)
at org.apache.beam.sdk.io.PubsubUnboundedSource$PubsubCheckpoint.finalizeCheckpoint(PubsubUnboundedSource.java:313)
at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.getReader(UnboundedReadEvaluatorFactory.java:174)
at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:127)
at org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
at org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Thomas Groh Thomas Groh · Accepted Answer · 2017-03-24T17:13:10

This problem exists because of a Bug (BEAM-1656) in the DirectRunner and a precondition within PubsubCheckpoint. The bug in the DirectRunner was fixed in pull request 2237, which is merged into the Github master branch, but after 0.6.0 release.

Updating to the 0.7.0 nightly build or building from github HEAD will solve this problem when using the DirectRunner.

To update to the current nightly build you will have to add the following repositories to your project's pom.xml. The earliest version of the beam-runners-direct-java module containing the fix is 0.7.0-20170316.070901-9, but not all modules are built with this specific version so you may have to either specify individually compatible versions or use 0.7.0-SNAPSHOT

    <repositories>
      <repository>
        <id>apache.snapshots</id>
        <name>Apache Development Snapshot Repository</name>

 <url>https://repository.apache.org/content/repositories/snapshots/</url>
        <releases>
          <enabled>false</enabled>
        </releases>
        <snapshots>
          <enabled>true</enabled>
        </snapshots>
      </repository>

    </repositories>

Apache Beam: PubsubReader fails with NPE

1 Answers