I'm running a cloud dataflow job to import multiple text files (.csv) from GCS into cloud spanner.
The job is partially working, about 6 million out of a 1 billion rows are imported, but then the job fails on this error:
Error message from worker: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: Error to parseRow. row: CSVRecord [comment='null', recordNumber=1, values=[source_set_id_hash, rank, run_id, source_set_id, recommended_set_id, score, updated_at, version]], table: CREATE TABLE `set_recs_similar_content` (
`source_set_id_hash` STRING(MAX) NOT NULL,
`version` STRING(MAX) NOT NULL,
`rank` INT64 NOT NULL,
`recommended_set_id` INT64 NOT NULL,
`run_id` STRING(MAX) NOT NULL,
`score` FLOAT64 NOT NULL,
`source_set_id` INT64 NOT NULL,
`updated_at` TIMESTAMP NOT NULL,
) PRIMARY KEY (`source_set_id_hash` ASC, `version` ASC, `rank` ASC)
Is this because it's reading the first row of the CSV and expecting it to match the format?
The relevant section from my manifest.json file is below:
"columns": [
{"column_name": "source_set_id_hash", "type_name": "STRING"},
{"column_name": "rank", "type_name": "INT64"},
{"column_name": "run_id", "type_name": "STRING"},
{"column_name": "source_set_id", "type_name": "INT64"},
{"column_name": "recommended_set_id", "type_name": "INT64"},
{"column_name": "score", "type_name": "FLOAT64"},
{"column_name": "updated_at", "type_name": "TIMESTAMP"},
{"column_name": "version", "type_name": "STRING"}
]
All files in GCS are formatted identically, so it seems odd that 6% of the job would complete, but then fail.
Relevant documentation link: https://cloud.google.com/dataflow/docs/guides/templates/provided-batch#gcstexttocloudspanner
Thanks
values=[source_set_id_hash, rank, so I think your hypothesis is correct. Are the column names quoted in your files? - Peter Kim