Using a "declare" to define a s3 url to load in PIG

Question

I would like to use a declare to define a URL that will be used in several places in my PIG code, like this:

 %declare OUTPUT_S3_DIR s3n://mystuff/shouldgohere/

When I do that I get the following error:

ERROR 2999: Unexpected internal error. java.net.URISyntaxException: Illegal character in scheme name at index 0: 's3n://mystuff/shouldgohere/'

I have tried using all of these:

 %declare OUTPUT_S3_DIR s3n://mystuff/shouldgohere/
 %declare OUTPUT_S3_DIR s3n://mystuff/shouldgohere
 %declare OUTPUT_S3_DIR 's3n://mystuff/shouldgohere/'
 %declare OUTPUT_S3_DIR 's3n://mystuff/shouldgohere'

and it does not seem to work when I use it like so:

rmf '$OUTPUT_S3_DIR';
STORE my_data INTO '$OUTPUT_S3_DIR' USING PigStorage(',');

I am pretty frustrated with this since it is obviously a simple syntax problem.

ftrotter ftrotter · Accepted Answer · 2014-04-08T00:09:33

Apparently, the error I made was in the rmf command. It needs to be

rmf $OUTPUT_S3_DIR;

and not

rmf '$OUTPUT_S3_DIR';

Hopefully someone else will not need to suffer...

Using a "declare" to define a s3 url to load in PIG

1 Answers