Defining TableSchema in PubSubToBigQuery template

Question

I'm having trouble specifying a TableSchema in Google's Dataflow template for for reading a pub-sub subscription and streaming into BigQuery.
Template provided here - https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java

Per documentation here - https://beam.apache.org/documentation/io/built-in/google-bigquery/#creating-a-table-schema

I attempted to add TableSchema to match my destination in BigQuery

/** Define Table Schema */
TableSchema schema = new TableSchema()
    .setFields(
        ImmutableList.of(
            new TableFieldSchema()
                .setName("device_id")
                .setType("STRING")
                .setMode("NULLABLE"),
            new TableFieldSchema()
                .setName("timestamp")
                .setType("TIMESTAMP")
                .setMode("NULLABLE"),
            new TableFieldSchema()
                .setName("orientation")
                .setType("FLOAT")
                .setMode("NULLABLE"),
            new TableFieldSchema()
                .setName("orientation")
                .setType("STRING")
                .setMode("NULLABLE"),
            new TableFieldSchema()
                .setName("light_level")
                .setType("FLOAT")
                .setMode("NULLABLE"),
            new TableFieldSchema()
                .setName("temperature")
                .setType("FLOAT")
                .setMode("NULLABLE"),
            new TableFieldSchema()
                .setName("button_pressed")
                .setType("BOOLEAN")
                .setMode("NULLABLE"),
            new TableFieldSchema()
                .setName("city")
                .setType("STRING")
                .setMode("NULLABLE"),
                new TableFieldSchema()
                .setName("region")
                .setType("string")
                .setMode("NULLABLE"),
            new TableFieldSchema()
                .setName("lat")
                .setType("STRING")
                .setMode("NULLABLE"),
            new TableFieldSchema()
                .setName("long")
                .setType("string")
                .setMode("NULLABLE")));

Here's the error I get when trying to compile.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.2:compile (default-compile) on project google-cloud-teleport-java: Compilation failure [ERROR] /home/jquattlebaum/DataflowTemplates/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java:[131,5] cannot find symbol [ERROR] symbol: class TableSchema [ERROR]
location: class com.google.cloud.teleport.templates.PubSubToBigQuery

I see in your code your are using both lowercase string and upper case STRING. can you try only uppercase and advice if you see any change — Tamir Klein

Anton Anton · Accepted Answer · 2019-05-20T16:26:01

Something like this usually happens when you don't have a dependency setup correctly. I.e. your project (template) has to declare a dependency that has the definition of TableSchema and then you have to add a corresponding import statement in your java file (PubSubToBigQuery.java).

Dependency declaration likely happens in pom.xml and if you haven't modified it much, it should already include the correct dependency for TableSchema, the google-api-services-bigquery library: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/pom.xml#L258.

If it does, then make sure you have import com.google.api.services.bigquery.model.TableSchema at the top of PubSubToBigQuery.java.

Then make sure you do mvn clean or clean the project from the IDE before building it again.

If this does not help I would look for a typo somewhere.

Defining TableSchema in PubSubToBigQuery template

1 Answers