Apache pig load multiple files

Question

I have the following folder structure containing my content adhering to the same schema -

/project/20160101/part-v121
/project/20160105/part-v121
/project/20160102/part-v121
/project/20170104/part-v121

I have implemented a pig script which uses JSONLoader to load & processes individual files. However I need to make it generic to read all the files under the dated folder.

Right now I have managed to extract the file paths using the following -

hdfs -ls hdfs://local:8080/project/20* > /tmp/ei.txt
cat /tmp/ei.txt | awk '{print $NF}' | grep part > /tmp/res.txt

Now I need to know how do I pass this list to pig script so that my program runs on all the files.

Murali Rao Murali Rao · Accepted Answer · 2017-03-10T23:17:40

We can use regex path in LOAD statement.

In your case the below statement should help, let me know if you face any issues.

A = LOAD 'hdfs://local:8080/project/20160102/*' USING JsonLoader();

Assuming .pig_schema (produced by JsonStorage) in the input directory.

Ref : https://pig.apache.org/docs/r0.10.0/func.html#jsonloadstore

Apache pig load multiple files

1 Answers