PIG command execution

1
votes

I am learning Hadoop by myself so I am not sure if what I asking is even a problem. When I run the command pig -x local to run it locally, i get the following message:


    15/10/05 15:23:28 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
    15/10/05 15:23:28 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
    2015-10-05 15:23:28,830 [main] INFO  org.apache.pig.Main - Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35
    2015-10-05 15:23:28,831 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/nkhl/pig_1444038808829.log
    2015-10-05 15:23:29,050 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /home/nkhl/.pigbootup not found
    2015-10-05 15:23:29,333 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
    2015-10-05 15:23:29,334 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
    2015-10-05 15:23:29,335 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
    2015-10-05 15:23:29,562 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

It looks different on my online tutor's screen so I am a little confused.

What concerns me most is the deprecation part. Can someone help me with that please? What is it trying to say? Don't get me wrong, everything works fine. The GRUNT shell loads up, and things execute fine. I just wanted to know what that meant.

It's an Ubuntu machine.

Thanks!

2
hadoopapache-pigubuntu-14.04
Are you asking about the deprecated warnings in general or you wanted to understand what each values do in the 'deprecated' warnings? If it is in general, then probably your online tutor was using a previous version which made use of certain variables. These variables have been deprecated or retired and replaced by new variable names as suggested in the warnings. However, the old variable names are still accepted and not an error yet. - proutray
Thanks for that. So there's no problems (whatsoever) with these lines, is there? - Anonymous Person
In my opinion, consider them as warnings. - proutray

2 Answers

2
votes

Running pig as local is great AFAIK if you are using for some quick testing.Like displaying the sysout in UDF etc. The above warnings you can safely ignore.It is saying that some of the variables set in conf-site.xml are deprecated.

You can switch off those parameters by editing the

log4j.logger.org.apache.hadoop.conf.Configuration.deprecation

in log4j.properties file.

1
votes

You have some Hadoop-related variables set, such as HADOOP_HOME or HADOOP_PREFIX or HADOOP_CONF_DIR, which aren't needed if you are running Pig in local mode.

unset HADOOP_HOME
unset HADOOP_PREFIX
unset HADOOP_CONF_DIR

Deprecations aren't scary. They are a reminder that the code is calling on something that will eventually go away in a future version. These specific deprecations are caused by differences between Hadoop 1 vs Hadoop 2. Pig is compatible with both versions. If you happened to be using Hadoop 1.2.1 instead of 2.x, you wouldn't see the warnings. This is because Pig is checking the Hadoop 1 values first.

If you're interested in learning more, you can check out the Pig source code. https://github.com/apache/pig/blob/release-0.15.0/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java#L219-L222