Pig Script issues

1
votes

I am using pig with Hcatalog to load data from hive external table I enter grunt using pig -useHCatalog and execute the following:

register 'datafu'

define Enumerate datafu.pig.bags.Enumerate('1');

imported_data  = load 'hive external table' using org.apache.hive.hcatalog.pig.HCatLoader() ;


converted_data = foreach imported_data generate name,ip,domain,ToUnixTime(ToDate(dateandtime,'MM/dd/yyyy hh:mm:ss.SSS aa'))as unix_DateTime,date;


grouped = group converted_data by (name,ip,domain);

result = FOREACH grouped {
             sorted = ORDER converted_data BY unix_DateTime;
             sorted2 = Enumerate(sorted);
             GENERATE FLATTEN(sorted2);
};

All commands run and provide desired result.

Problem: I made a pig script with above commands named as pigFinal.pig and executed the following in local mode coz script in local filesystem.

pig -useHCatalog -x local '/path/to/pigFinal.pig';

Exception

Failed to generate logical plan. Nested exception: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve datafu.pig.bags.Enumerate using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.] at org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:1507) at org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:9372) at org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:11051) at org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:10810) at org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:10159) at org.apache.pig.parser.LogicalPlanGenerator.nested_command(LogicalPlanGenerator.java:16315) at org.apache.pig.parser.LogicalPlanGenerator.nested_blk(LogicalPlanGenerator.java:16116) at org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:16024) at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15849) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1933) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188) ... 17 more

Where do i need register datafu jar for pig scripts?I guess this is the issue. Please help

2
hadoopapache-pig

2 Answers

0
votes

You have to ensure the jar file is located in the same folder as your pigscript or ensure that the correct path is provided in the pigscript while registering the jar file. So in your case

Modify this

register 'datafu'

To

-- If,lets say datafu-1.2.0.jar is your jar file and is located in the same folder as your pigscript then in your pigscript at the top have this
REGISTER datafu-1.2.0.jar 

-- Else,lets say datafu-1.2.0.jar is your jar file and is located in the folder /usr/hadoop/lib then in your pigscript at the top have this
REGISTER /usr/hadoop/lib/datafu-1.2.0.jar
0
votes
pig -useHCatalog \
    -x local \
    -Dpig.additional.jars="/local/path/to/datafu.jar:/local/path//other.jar" \ 
    /path/to/pigFinal.pig;

OR

in your pig script use fully qualified path

register /local/path/to/datafu.jar;