Error while trying to load multiple CSVs to Google BigQuery

1
votes

I'm trying to load multiple CSVs (31) into a new table in BigQuery, and I get the following error:

BigQuery error in load operation: Error processing job 'job_name': Too many errors encountered. Failure details: - /gzip/subrange//bigstore/bucket_name/one_of_the_csvs.tsv.gz: CSV table references column position 75, but line starting at position:8369304 contains only 75 columns

I have 76 columns, this might be related to the 75 in the error message. It might be that the creation of the CSVs was wrong (although this code worked correctly, with some modificatios, several months ago)

What am I missing here..? I couldn't find info about this error..

2
csvgoogle-bigquery
Have you tried importing with the option "allow jagged rows"? - matt_black
"allow jagged rows" solved it for me, though i somewhat guess the error came from a trailing new line - Hirnhamster

2 Answers

2
votes

Seems this error message is clear: you have a jagged line somewhere in the csv file. Have you validated the content? If you are sure it's good, then probably you can attach the file that triggers the issue and someone could have a look.

0
votes

just use any csv validator to find the problem. In my case - there was 1 line, that included other 2 linesenter image description here