0
votes

I'm running a cloud dataflow job to import multiple text files (.csv) from GCS into cloud spanner.

The job is partially working, about 6 million out of a 1 billion rows are imported, but then the job fails on this error:

Error message from worker: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: Error to parseRow. row: CSVRecord [comment='null', recordNumber=1, values=[source_set_id_hash, rank, run_id, source_set_id, recommended_set_id, score, updated_at, version]], table: CREATE TABLE `set_recs_similar_content` (
    `source_set_id_hash`                    STRING(MAX) NOT NULL,
    `version`                               STRING(MAX) NOT NULL,
    `rank`                                  INT64 NOT NULL,
    `recommended_set_id`                    INT64 NOT NULL,
    `run_id`                                STRING(MAX) NOT NULL,
    `score`                                 FLOAT64 NOT NULL,
    `source_set_id`                         INT64 NOT NULL,
    `updated_at`                            TIMESTAMP NOT NULL,
) PRIMARY KEY (`source_set_id_hash` ASC, `version` ASC, `rank` ASC)

Is this because it's reading the first row of the CSV and expecting it to match the format?

The relevant section from my manifest.json file is below:

      "columns": [
        {"column_name": "source_set_id_hash", "type_name": "STRING"},
        {"column_name": "rank", "type_name": "INT64"},
        {"column_name": "run_id", "type_name": "STRING"},
        {"column_name": "source_set_id", "type_name": "INT64"},
        {"column_name": "recommended_set_id", "type_name": "INT64"},
        {"column_name": "score", "type_name": "FLOAT64"},
        {"column_name": "updated_at", "type_name": "TIMESTAMP"},
        {"column_name": "version", "type_name": "STRING"}
      ]

All files in GCS are formatted identically, so it seems odd that 6% of the job would complete, but then fail.

Relevant documentation link: https://cloud.google.com/dataflow/docs/guides/templates/provided-batch#gcstexttocloudspanner

Thanks

1
The "values" part of the error message usually contains values, not column names like values=[source_set_id_hash, rank, so I think your hypothesis is correct. Are the column names quoted in your files? - Peter Kim

1 Answers

2
votes

It turns out that the header should not be present in your CSV files.

From the template source file:

 * <p>Text file must NOT have a header.

Please retry after removing the header from all of your files. Below line from the error message suggests that the first row was a header:

recordNumber=1, values=[source_set_id_hash, rank, run_id, source_set_id, recommended_set_id, score, updated_at, version]