Issue in Map Reduce Program

1
votes

I am executing a hadoop Map-Reduce Job for simple word count problem using Putty. I have configured Hadoop on a VM and I have verified all components of Hadoop are running using jps.

When I am executing the code using command hadoop jar Untitled.jar

I am getting error

15/06/20 19:36:48 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
15/06/20 19:37:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library
15/06/20 19:37:09 WARN snappy.LoadSnappy: Snappy native library not loaded
15/06/20 19:37:09 INFO mapred.FileInputFormat: Total input paths to process : 0
15/06/20 19:37:10 INFO mapred.JobClient: Running job: job_201506201820_0004
15/06/20 19:37:11 INFO mapred.JobClient:  map 0% reduce 0%
15/06/20 19:37:12 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000001_0, Status : FAILED
Error initializing attempt_201506201820_0004_m_000001_0:
ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
        at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
        at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603)
        at java.lang.Thread.run(Thread.java:745)

15/06/20 19:37:13 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_0&filter=stdout
15/06/20 19:37:13 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_0&filter=stderr
15/06/20 19:37:14 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000001_1, Status : FAILED
Error initializing attempt_201506201820_0004_m_000001_1:
ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
        at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
        at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603)
        at java.lang.Thread.run(Thread.java:745)

15/06/20 19:37:15 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_1&filter=stdout
15/06/20 19:37:15 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_1&filter=stderr
15/06/20 19:37:16 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000001_2, Status : FAILED
15/06/20 19:37:16 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_2&filter=stdout
15/06/20 19:37:16 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_2&filter=stderr
15/06/20 19:37:17 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000000_0, Status : FAILED
Error initializing attempt_201506201820_0004_m_000000_0:
ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
        at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
        at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603)
        at java.lang.Thread.run(Thread.java:745)

15/06/20 19:37:17 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_0&filter=stdout
15/06/20 19:37:17 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_0&filter=stderr
15/06/20 19:37:18 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000000_1, Status : FAILED
Error initializing attempt_201506201820_0004_m_000000_1:
ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
        at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
        at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603)
        at java.lang.Thread.run(Thread.java:745)

15/06/20 19:37:19 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_1&filter=stdout
15/06/20 19:37:19 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_1&filter=stderr
15/06/20 19:37:20 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000000_2, Status : FAILED
Error initializing attempt_201506201820_0004_m_000000_2:
ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
        at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
        at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603)
        at java.lang.Thread.run(Thread.java:745)

15/06/20 19:37:20 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_2&filter=stdout
15/06/20 19:37:20 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_2&filter=stderr
15/06/20 19:37:21 INFO mapred.JobClient: Job complete: job_201506201820_0004
15/06/20 19:37:21 INFO mapred.JobClient: Counters: 4
15/06/20 19:37:21 INFO mapred.JobClient:   Job Counters
15/06/20 19:37:21 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
15/06/20 19:37:21 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
15/06/20 19:37:21 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=0
15/06/20 19:37:21 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
15/06/20 19:37:21 INFO mapred.JobClient: Job Failed: JobCleanup Task Failure, Task: task_201506201820_0004_m_000000
java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
        at WordCount.main(WordCount.java:32)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

What am I missing.

2
hadoopmapreduce
Can you add jps results, datanode and namenode logs please? - chomp

2 Answers

1
votes

you don't seem to be providing any input path. Look at the error:

15/06/20 19:37:09 INFO mapred.FileInputFormat: Total input paths to process : 0

0
votes

From logs, you can see the issue occured at first map execution it self, as 0% for both map and reduce processes. Next is the actual error. "No such file". The only time we deal with map reduce and file is input and output. Given we are at start of map process, I think the issue could be with input path and its permissions. Please look into them. Also the output directory should not be existed before running the job. It creates I suppose. Happy coding