Unable to save data frame as Hive table, throwing file not found exception

0
votes

When I am trying to save data frame as Hive table in pyspark

df_writer.saveAsTable('hive_table', format='parquet', mode='overwrite')

I am getting following error:

Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://hostname:8020/apps/hive/warehouse/testdb.db/hive_table at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)

I have the path till 'hdfs://hostname:8020/apps/hive/warehouse/testdb.db/'

Please provide your inputs

1
apache-sparkhivepysparkpyspark-sql

1 Answers

0
votes

Try using DataFrameWriter as 

df.write.mode(SaveMode.Append).insertInto(s"${dbName}.${t.table}")