insert data in HDFS using Hive

0
votes

let's assume that we have an external Hive table pointing on CSVs files on an hdfs directory.

so what happened when inserting a new line on this table using hive :

  1. will the insert cause a whole rewrite of the table ?
  2. or a a whole rewrite of the hdfs block where the data is located in ?
  3. or will simply append the new line at the end of the file ?

Same question for the update operation

thanks in advance !

1
hadoophivehdfs
Forget about UPDATEs. There's a half-baked attempt at ACID transactions, but it requires ORC tables and a background compaction process. - Samson Scharfrichter
AFAIK an INSERT OVERWRITE will nix all the files in directory then create 1..N new files (one per reducer if the SELECTrequires a reducer step, one per mapper otherwise -- unless you tweak the "merge small files" parameters to reduce fragmentation). And an INSERT will just add 1..N new files. - Samson Scharfrichter
Please note that HDFS files are immutable, there is no appending or rewriting (OK, now you can do it but that's a deviant feature for queer edge cases; as far as Hive is concerned HDFS files are immutable) - Samson Scharfrichter

1 Answers

1
votes

Answering your question, thinking that you are using an insert statement and not using INSERT OVERWRITE with files.

  1. No, insert will create a new file with the data you have inserted
  2. No, only new file will be inserted
  3. No appends are done to the existing files

Even if you use INSERT INTO and insert some files, then those new files will come and sit in the particular directory in HDFS without impacting existing files.

If you are using INSERT OVERWRITE all the files present in the directory of the given table will be deleted and new files will be placed in that directory.