default storage file format in hadoop/hdfs

Question

I am setting up a new hadoop cluster (experimental at this stage).

I want it to be configured such that whenever a file is copied onto the cluster (either through copyFromLocal or using sqoop etc), hadoop/hdfs should store the data in parquet file format.

Am I expecting right about this ? is it possible ?

I thought there should be a configuration parameter somewhere at the hdfs level, where i could specify which format to use while storing data, somehow not able to find that. Wondering if i m missing something here.

Ben Watson Ben Watson · Accepted Answer · 2015-11-06T11:14:33

No you're right - there's no HDFS-level configuration. You'd have to set the storage format each time you operate on some data. Imagine the damage that would be done if every file was automatically converted into Parquet. All the temporary files created by applications, any Hive/Pig scripts and any lookup files would be ruined.

To save the output of a Sqoop command into Parquet:

sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files

will write the data into Parquet format.

There's no way to do this with a copyFromLocal.

To move data that's already on the HDFS into Parquet, load the data into an external Hive table in its original format, create a Parquet table and then load the data into it, i.e.

//Overlay a table onto the input data on the HDFS
CREATE EXTERNAL TABLE input (
  id int,
  str string
STORED AS <the-input-data-format>
LOCATION 'hdfs://<wherever-you-put-the-data>';

//Create a Parquet-formatted table
CREATE TABLE parquet (
  id int,
  str string
STORED AS PARQUET;

//Write your input data into the Parquet table - this will format the data into Parquet
INSERT INTO TABLE parquet
SELECT * FROM input;

default storage file format in hadoop/hdfs

1 Answers