Wordcount C++ Hadoop pipes does not work

1
votes

I am trying to run the example of wordcount in C++ like this link describes the way to do : Running the WordCount program in C++. The compilation works fine, but when I tried to run my program, an error appeared :

bin/hadoop pipes -conf ../dev/word.xml -input testtile.txt -output wordcount-out
11/06/06 14:23:40 WARN mapred.JobClient: No job jar file set. User classes may not be
found. See JobConf(Class) or JobConf#setJar(String).
11/06/06 14:23:40 INFO mapred.FileInputFormat: Total input paths to process : 1
11/06/06 14:23:40 INFO mapred.JobClient: Running job: job_201106061207_0007
11/06/06 14:23:41 INFO mapred.JobClient: map 0% reduce 0%
11/06/06 14:23:53 INFO mapred.JobClient: Task Id : attempt_201106061207_0007_m_000000_0, Status : FAILED
java.io.IOException
at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188) at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194) at org.apache.hadoop.mapred.pipes.Application.(Application.java:149) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201106061207_0007_m_000000_0: Server failed to authenticate. Exiting

I am running Hadoop on Fedora on two nodes, and I followed instructions for configurations from that link : Running Hadoop on multi-node cluster. I tried the wordcount example of Hadoop with that command :

bin/hadoop jar hadoop-examples-0.20.203.0.jar wordcount testtile.txt wordcount-out

And that command works fine. That is why I don't understand why my program did not work. So I hope that someone have an idea about what I am doing wrong, or if someone had already resolve this error.

1
c++hadoopcluster-computingword-count

1 Answers

1
votes

I do not know if I have to answer to my question in this way, or edit my question. Anyway I find the solution and I just want to tell it for everyone who will get the same error.

After few days of research and try, I understand that Fedora and C++ on 64bits for Hadoop is not a good match. I tried to compile the Hadoop wordcount C++ with ant like explained in the wiki. But ant gets me some error about : libssl and stdint.

First, if you are on Fedora you have to add -lcrypto to the LIBS variables in the .configure. That is cause the dependency on libcrypto must now be explicitely stated on these platform when linking to libssl.(see bug on Fedora).

Second issue : ant produces a lot of error about C++ files : to resolve that you just have to add an include : stdint.h on the top of the file.

Then the build success. I tried then to run wordcount example on my Hadoop cluster and it works, while mine did not. I expected that issue come from the library that I just corrected and I was right : I tried to run Hadoop example with library from the hadoop install directory and it did not work and I get the same error.

That could be explained by the fact that ant recompile the C++ library needed for Hadoop(with correction that I did) and used it, instead library provides in the Hadoop install directory.