Issue while writing into Date datatype in Big Query using Spark Java

0
votes

I am trying to store date datatype column in BigQuery via Spark

cast(from_unixtime(eventtime*60) as date)  as createdDate

I have tried to_date as well like below, but no luck

to_date(from_unixtime(eventtime*60))  as createdDate

Now I am trying to save this dataset using Spark-BigQuery connector, it is giving me error that field createdDate has changed type from DATE to INTEGER. But when I try to print the schema in spark, its correctly saying that the column data types is Date.

 |-- createdDate: date (nullable = false)

Not sure why its failing while loading data into BigQuery. And the same things works if I change the data types from Date to Timestamp. Please advice.

1
google-bigquery

1 Answers

0
votes

The resolution is to use intermediateFormat as Orc. With intermediate format as Avro it is not working, and we can't use parquet(default) format as we have array data type in our table where Big Query create intermediate format like explained here. Save Array<T> in BigQuery using Java