Apache Beam Java 2.26.0: BigQueryIO 'No rows present in the request'

Question

Since the Beam 2.26.0 update we ran into errors in our Java SDK streaming data pipelines. We have been investigating the issue for quite some time now but are unable to track down the root cause. When downgrading to 2.25.0 the pipeline works as expected.

Our pipelines are responsible for ingestion, i.e., consume from Pub/Sub and ingest into BigQuery. Specifically, we use the PubSubIO source and the BigQueryIO sink (streaming mode). When running the pipeline, we encounter the following error:

{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "No rows present in the request.",
    "reason" : "invalid"
  } ],
  "message" : "No rows present in the request.",
  "status" : "INVALID_ARGUMENT"
}

Our initial guess was that the pipeline's logic was somehow bugged, causing the BigQueryIO sink to fail. After investigation, we concluded that the PCollection feeding the sink is indeed containing correct data.

Earlier today I was looking in the changelog and noticed that the BigQueryIO sink received numerous updates. I was specifically worried about the following changes:

BigQuery’s DATETIME type now maps to Beam logical type org.apache.beam.sdk.schemas.logicaltypes.SqlTypes.DATETIME
Java BigQuery streaming inserts now have timeouts enabled by default. Pass --HTTPWriteTimeout=0 to revert to the old behavior

With respect to the first update, I made sure to disable all DATETIME in the resulting TableRow objects. In this specific scenario, the error still stands.

For the second change, I'm unsure how to pass the --HTTPWriteTimeout=0 flag to the pipeline. How is this best achieved?

Any other suggestions as to the root cause of this issue?

Thanks in advance!

I did! Initially ignored the issue and assumed it would be fixed in the upcoming version. Issue persists in 2.28.0. — LaurensVijnck
This is for the Java SDK, right? I'm guessing so based on the changes you called out, but it might be good to specify at the beginning of the question. — Daniel Oliveira
The BigQueryOptions is defined in github.com/apache/beam/blob/master/sdks/java/io/… and can be used like beam.apache.org/releases/javadoc/2.28.0/org/apache/beam/sdk/…. Does setting this flag help? — Cubez

LaurensVijnck LaurensVijnck · Accepted Answer · 2021-03-18T10:02:17

We have finally been able to fix this issue and rest assured it has been a hell of a ride. We basically debugged the entire BigQueryIO connector and came to the following conclusions:

The TableRow objects that are being forwarded to BigQuery used to contain enum values. Due to these not being serializable, an empty payload is forwarded to BigQuery. In my opinion, this error should be made more explicit (why was this suddenly changed anyway?).
- The issue was solved by adding the @value annotation to each enum entry (com.google.api.client.util.Value).
The same TableRow object also contained values of the type byte[]. This value was injected in a BigQuery column with the bytes type. While this was working without explicitly computing a base64 before, it was now yielding errors.
- The issue was solved by computing a base64 ourselves (this setup is also discussed in the following post).

Apache Beam Java 2.26.0: BigQueryIO 'No rows present in the request'

2 Answers