Loading new files using Pig LOAD statement

Question

I wanted to load data from HDFS to HBSE table sing PIG script.

I have hadfs folder structure as below:

-rw-r--r--  1 user supergroup   63 2014-05-15 20:28 dataparse/good/goodrec_051520142028
-rw-r--r--  1 user supergroup   72 2014-05-15 20:30 dataparse/good/goodrec_051520142030
-rw-r--r--  1 user supergroup   110 2014-05-15 20:32 dataparse/good/goodrec_051520142032

In the above all filenames are attached with the timestamp.

Below is my PIG script to load from HDFS to HBASE:

G = LOAD '/user/user/dataparse/good/' USING PigStorage(',') as (c1:chararray, c2:chararray,c3:chararray,c4:chararray,c5:chararray);
STORE G INTO 'hbase://test' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('t1:name t1:state t1:phone_no t1:gender');

The script is working fine and the data from all the 3 files are written to the Hbase "test" table.

Suppose after some time if some more files comes to HDFS with the same structure and when i run the pig script it will LOAD all the files in the "good" directory along with the already read file. So how can i load only those files which are new files. Already loaded files should not be loaded again into my HBASE table.

How can i do this?

Thanks, Sapthashree

Sudarshan Sudarshan · Accepted Answer · 2014-05-15T16:34:11

I think you have a few options here.

Using globs

Using a shell script pick up the "new" files, Use the glob feature so that multiple files can be fed into the script. A related use case is here
If the files have a date and timestamp in the filename then you can use globs directly, look here to inspiration

Using big guns

If using globs is failing you, then you need to bring out the big guns, use a custom load function put in the logic to identify "new files" in it and you should be good to go. Details here

Loading new files using Pig LOAD statement

2 Answers