How does Hive know how the data file is compressed when doing query

0
votes

I create a hive table with create table t(name string, age int) stored as parquet. Then I insert some data, some are using gzip compression, and some are using snappy compression.That is, there are gzip files and snappy files in the corresponding HDFS directory. When I do the query, I see that the data in the gzip and snappy are all queried out, my question is how does Hive detect which compression codec is used when it processing the file?

3
hadoophivecompression

3 Answers

2
votes

Simply to check metadata and get compression info is not enough. You can easily mix compressed and uncompressed files in one table directory and it will work. So compression codec is defined for each file in the file itself.

In which part of the file is this information about codec is written?

It depends on file type.

Table metadata contain information about file type (can be specified as STORED AS ... in table DDL )

So, first of all Hive reads the metadata to get the file type.

Depending on the file type (text, orc, parquet, etc) the codec information is stored in the file header or footer.

Each file type has it's corresponding reader. The reader knows how to read codecs information. For example the ORC reader reads the last 16k bytes of the file with the hope that it will contain both the Footer and Postscript sections. PostScript section in ORC contains codecs info.

Text files contain codec information in the header, and so on... So, the short answer is: The corresponding file reader knows where to get information about compression codec.

0
votes

Hive detects it the same way as you describe the table :

DESCRIBE [EXTENDED|FORMATTED] table_name (details https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL)

above command will show all table properties whether it is compressed or which type of compression it is using and tons of other properties related to your hive table.

so while reading your query "HQL engine" reads the metadata of your table first as HIVE follows the Schema on read phenomenon (more details http://www.centurylink.com/business/enterprise/blog/thinkgig/data-lakes-schema-on-read-vs-schema-on-write/)

0
votes

When a Table is created in hive we define below fields.

ROW FORMAT SERDE

STORED AS file_format

LOCATION hdfs_path

File_format Column location and separator..

Look table definition. Added compression / decompression