Finding Hive/Impala table's compression details

Question

I convert tables from one format to another, from uncompressed to compressed (Snappy, Gzip etc).

I thought I could rely on describe [formatted|extended] tblname until I read this. DESCRIBE Statement

It states

The Compressed field is not a reliable indicator of whether the table contains compressed data. It typically always shows No, because the compression settings only apply during the session that loads data and are not stored persistently with the table metadata.

How do I find out if a table is compressed and what codec is used? I don't mind using Spark to get that info.

Answer 1

Answering my question:

For Avro data files : avro-tools getmeta filename

For Parquet data files : parquet-tools meta filename

Answer 2

As you state, the 'describe formatted' and 'show create table' methods are not always guaranteed to contain the proper compression format information.

The most credible way to identify compression codec and storage format is to go to the HDFS location of the table files, and look at their extension:

hdfs dfs -ls -r /hdfspath/

An ORC file compressed in snappy for example should end in .snappy.orc.

Answer 3

First thought is to check this Hive/MR properties:

hive.exec.compress.output=
mapreduce.output.fileoutputformat.compress=
mapreduce.output.fileoutputformat.compress.codec=   
mapreduce.output.fileoutputformat.compress.type=