Apache Pig - Create unique output folder using UDF

2
votes

I am using Apache Pig from Hue to perform ETL operations on files using the script etl-op.pig. The output is stored into the specified folder in HDFS using the following line:

STORE outval INTO '/user/root/Pig-Output

However next time when the script is run, it says the output folder already exists and doesn't create a separate folder.

Is there any way to create a Java UDF in Pig using Hue so that a unique identifier can be generated and appended to the 'Pig-Output' folder name present in the script ?

1
javahadoopapache-pighue
What type of identified do you need? A popular way is to just add a variable, e.g. ''STORE outval INTO '/user/root/Pig-Output/$output" - Romain
@Romain: What value will $output contain ? - User456898
You will be prompted to provide the value at submission. - Romain
@Romain: Thanks, got it :) - User456898

1 Answers

1
votes

You can do it without UDF: Define a variable like the current unix timestamp:

%default TS `date  +%s`

And than use it as e.g. a postfix of your folder:

STORE outval INTO '/user/root/Pig-Output_$TS' ...