GCP Data Fusion multiple table import

Question

I'm trying to use Multiple Database Tables and BigQuery Multi Table Data Fusion plugin to import multiple table in one pipeline

But when I try to execute I get the following error

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BigQuery Multi Table has no outputs. Please check that the sink calls addOutput at some point.

I'm using Data Fusion version 6.1.4 Multiple Database Tables version 1.2.0 and BigQuery Multi Table version 0.14.8.

Any suggestion on what may be the problem?

Edit:

following the configuration of multiple table database source

{
            "name": "Multiple Database Tables",
            "plugin": {
                "name": "MultiTableDatabase",
                "type": "batchsource",
                "label": "Multiple Database Tables",
                "artifact": {
                    "name": "multi-table-plugins",
                    "version": "1.2.0",
                    "scope": "USER"
                },
                "properties": {
                    "splitsPerTable": "1",
                    "referenceName": "multiTable",
                    "connectionString": "${secure(connection)}",
                    "jdbcPluginName": "netezza",
                    "user": "${secure(username)}",
                    "password": "${secure(password)}",
                    "whiteList": "categoria_l,cliente_l,regione_l"
                }
            },
            "outputSchema": [
                {
                    "name": "etlSchemaBody",
                    "schema": ""
                }
            ]
        },

After further test the problem is that the source response is empty because data fusion is not reading view from source database but only tables

Did your pipeline finish successfully, populating the tables to Bigquery sink? — Nick_Kh
It seems that BigQuery Multi Table plugin doesn't produce any Output schema but basically spreads Bigquery Job to insert/update columns across the tables. — Nick_Kh
The data previously is stored in temporary GCS bucket, and then imported to Bigquery tables. — Nick_Kh
@Nick_Kh no if i run the pipeline fails with the error reported in the question — Paolo Fusari

xgMz xgMz · Accepted Answer · 2020-10-16T16:11:55

It seems like the Multiple Database Tables source produced no records ("Out 0"). I'd check there first. You can do a quick check using the Preview mode. Plugin doc here.

GCP Data Fusion multiple table import

1 Answers