Delete partition directories from HDFS, would it reflect in hive table?

1
votes

Lets say I created a hive table with partition column as year, month and day and if i delete the partition from hdfs, then result get reflected in hive table or not

2
hivehive-partitions
What do you mean by "delete the partition"? Delete all directories + files? delete only the sub-directories and put the files in the top directory? – David דודו Markovitz
Delete the all directories + files – Dipak
Possible duplicate of How to update partition metadata in Hive , when partition data is manualy deleted from HDFS – Remus Rusanu

2 Answers

1
votes

Yes. The partition data will be gone.
The metastore will still hold the partition information (metadata) and you can see it using show partition mytable.
You can find the partitions need to be dropped using msck repair mytable.
You can drop the partitions using alter table mytable drop partition (...)

0
votes

Hive table will still show the partitions, you will have to either drop the partitions deleted on HDFS manually (or drop and re-create table) and run MSCK.

Commands:

If you intend to alter the table and drop all deleted partitions-

ALTER TABLE table_name DROP [IF EXISTS] PARTITION partition_spec[, PARTITION partition_spec, ...]
  [IGNORE PROTECTION] [PURGE];            -- (Note: PURGE available in Hive 1.2.0 and later, IGNORE PROTECTION not available 2.0.0 and later)

I would go with drop and re-create table then run MSCK.

To add all existing partitions to table-

msck repair table <table_name>

Alternatively, you could drop all partitions using ALTER TABLE and then run the MSCK command.