Spark returns Empty DataFrame but Populated in Hive

Question

I have a table in hive

db.table_name

When I run the following in hive I get results back

SELECT * FROM db.table_name;

When I run the following in a spark-shell

spark.read.table("db.table_name").show

It shows nothing. Similarly

sql("SELECT * FROM db.table_name").show

Also shows nothing. Selecting arbitrary columns out before the show also displays nothing. Performing a count states the table has 0 rows.

Running the same queries works against other tables in the same database.

Spark Version: 2.2.0.cloudera1

The table is created using

table.write.mode(SaveMode.Overwrite).saveAsTable("db.table_name")

And if I read the file using the parquet files directly it works.

spark.read.parquet(<path-to-files>).show

EDIT: I'm currently using a workaround by describing the table and getting the location and using spark.read.parquet.

Are you using HiveContext (if Spark 1.6) or have you called .enableHiveSupport() (if Spark 2.x)? — Vladislav Varslavans
Using spark2-shell which I believe enables hive support on when setting up the session. — A Spoty Spot
Are logs available? Have you looked there for possible errors/warnings? — Vladislav Varslavans
Since it's client mode all errors/warnings are logged to the shell. No logs get displayed. — A Spoty Spot

Angel F O Angel F O · Accepted Answer · 2018-03-12T21:28:29

Have you refresh metadata table? Maybe you need to refresh table to access to new data.

spark.catalog.refreshTable("my_table")