I am trying to access a file (sample.txt) in UDF. I want to put that file in distributed cache and use it from there. I am using amazon EMR to run the Pig job. I am copying the file (sample.txt) to HDFS using EMR bootstrap-action while creating cluster.
bootstrap.sh(copies file from s3 to hdfs)
hadoop fs -copyToLocal s3n://s3_path/sample.txt /mnt/sample.txt
UsingSample.java(UDF that uses sample.txt)
public class UsingSample extends EvalFunc<String>{
public String useSampleText(String str) throws Exception{
File sampleFile = new File(“./sample”);
//do something with sampleFile
}
@Override
public String exec(Tuple input) throws IOException {
if (input == null || input.size() == 0)
return null;
String str = (String) input.get(0);
String result = "";
try {
result = useSampleText(str);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return result;
}
public List<String> getCacheFiles() {
List<String> list = new ArrayList<String>(1);
list.add("/mnt/sample.txt#sample"); // not sure if the path I am passing is correct
return list;
}
}
create_cluster.sh(script that creates the cluster and executes the Pig script)
aws emr create-cluster
--auto-terminate
--name "sample cluster"
--ami-version 3.8.0
--enable-debugging
--applications Name=Pig
--use-default-roles
--instance-type m1.large
--instance-count 3
--steps Type=PIG,Name="Pig Program",ActionOnFailure=CONTINUE,Args=[-f,$S3_PIG_SCRIPT_URL,-p,INPUT=$INPUT,-p,OUTPUT=$OUTPUT]
--bootstrap-action Path=s3://s3_bootstrapscript_path/bootstrap.sh
The error which I am getting is FileNotFound exception when trying to access sample.txt in getCacheFiles().
I am using:
Hadoop 2.4
Pig 0.12
Please help.
