Does Spark saveastable infer schema from dataframe

1
votes

I am using Spark 2.0. Requirement is to create a new table from selecting values into dataframe. While writing out the df as

df.write.saveasTable(hive_table_name, format='parquet',mode='overwrite')

Error is: client cannot authenticate via:[TOKEN, KERBEROS] Host Details : local host is: "some-ip" and destination host is:"some-other-ip"

Also if the table does not exist in hive, will spark.write.saveasTable create a new table in hive and auto infer the schema?

2
apache-sparkpysparkapache-spark-sqlspark-dataframepyspark-sql

2 Answers

0
votes

I am not sure about the error you are receiving, but yes, Spark will infer the schema automatically if you try to create a table that doesn't exist.

Hope that helps!

Subhash

0
votes

Ok, some learnings over the past weeks,

saveasTable saves a table to the hdfs file system. Without a schema explicitly created on Hive to consume the parquet file, the schema inference from spark, while creating the dataframe is not used by hive to reflect the existing columns of a table on Hive.

Schema inference is only for JSON, CSV and not for .dat files or compressed textfiles. These files would have to be dealt with delimiters and the dataframe would have to be renamed with first row as column title and then saved to the disk.