Spring + Maven + Hadoop

Question

I have a problem in Spring + Maven + Hadoop enviroment. I am using Apache Hadoop, but don't use spring-hadoop that is provided from SpringSource. I added following dependency to Maven for using hadoop in java.

<dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>0.20.2</version>
        <exclusions>
            <exclusion>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>jetty</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>jetty-util</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>jsp-2.1</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>jsp-api-2.1</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>servlet-api-2.1</artifactId>
            </exclusion>
            <exclusion>
                <groupId>javax.servlet</groupId>
                <artifactId>servlet-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>javax.servlet.jsp</groupId>
                <artifactId>jsp-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>tomcat</groupId>
                <artifactId>jasper-compiler</artifactId>
            </exclusion>
            <exclusion>
                <groupId>tomcat</groupId>
                <artifactId>jasper-runtime</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

And I executed hadoop mapreduce job. Mapreduce job works well.

But I can't see executed jobs from "Running Jobs" List in Hadoop Web UI (http://xxx.xxx.xxx.xxx:50030/jobtracker.jsp)

I started without hadoop command(ex-bin/hadoop -jar ....jar ), but just java application. Could this be the reason?

Is your hadoop environment running outside the mapreduce job? or are you spinning up a pseudo environment? Is your job tracker running? ps Thanks for posting your pom it helped me with another issue ;) — Miguel Pereira
How is this a maven question? Rather find out what the problem is, before we can help you. — Niels Bech Nielsen

dmitry dmitry · Accepted Answer · 2014-03-13T04:39:08

Hadoop provides API to run MR jobs locally, without setting up Hadoop cluster. If you don't see your job running/completed in Hadoop web UI - it seems that you run MR job locally.

To run remotely - specify appropriate job tracker host, in your case http://xxx.xxx.xxx.xxx:9001

Spring + Maven + Hadoop

1 Answers