Undebuggable Dataflow Exception

Question

We have created a very simple job in Dataflow. It reads from BigQuery and stores into RedisCluster periodically. The pipeline code is as below.

    // initialize pipeline
    PipelineOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().create();
    Pipeline pipeline = Pipeline.create(options);

    // ... read from bigquery
    final String query = "..."; // actual query snipped
    PCollection<TableRow> bigQueryRes = pipeline.apply("Reading from BigQuery table",
            BigQueryIO.readTableRows().fromQuery(query).usingStandardSql());

    // ... store to Redis Cluster
    bigQueryRes.apply("Storing to RedisCluster",
            ParDo.of(new RedisClusterSinkDoFn()));

    // run pipeline
    pipeline.run();

When we directly execute the job directly through command line it executes perfectly.

But when we generate a template, from same JAR, execute through Cloud Function using the launch api we find the below exception:

 java.io.FileNotFoundException: No files matched spec: gs://dataflow-bucket/tmp/BigQueryExtractTemp/0e22ca710a6b45f7aa134cff02b4ae9b/000000000057.avro

    at org.apache.beam.sdk.io.FileSystems.maybeAdjustEmptyMatchResult (FileSystems.java:166)
    at org.apache.beam.sdk.io.FileSystems.match (FileSystems.java:153)
    at org.apache.beam.sdk.io.FileBasedSource.createReader (FileBasedSource.java:332)
    at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$1.iterator (WorkerCustomSources.java:362)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop (ReadOperation.java:179)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start (ReadOperation.java:160)
    at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute (MapTaskExecutor.java:77)
    at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork (BatchDataflowWorker.java:395)
    at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork (BatchDataflowWorker.java:364)
    at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork (BatchDataflowWorker.java:292)

We are not sure what is causing this issue as neither the Dataflow logs nor BigQuery is being helpful. The only difference I could tell is that when the job succeeds - there is a folder "gs://dataflow-bucket/tmp/BigQueryExtractTemp" which is created with split output of the query result. The folder is absent the job fails.

Any thoughts on where to start debugging for this?

Thanks in anticipation.

Can you run "gcloud auth list" in the command line to see which account you are using? Make sure to use the same account or to use an account with the proper permissions. — Nathan Nasser

Caio Riva Caio Riva · Accepted Answer · 2018-12-19T16:33:49

I've been investigating this issue and discovered that the jobId (for the BigQuery job) used in the BigQueryIO is generated on the template creation. This means that for a second execution for the same template it will generate a 409 error, saying that a BigQuery job with the same jobId already exists. Resulting in this "java.io.FileNotFoundException: No files matched spec:" error as no temporary file for the BigQuery input data has been created.

Fortunately, this bug has been fixed for Apache Beam 2.x versions. Google documentation says:

"If you want to run a batch pipeline that reads from BigQuery, you must use .withTemplateCompatibility() on all BigQuery reads"

So to fix it add the .withTemplateCompatibility() to your BigQueryIO read.

Hope that helps you!

Undebuggable Dataflow Exception

1 Answers