Failed to load native-lzo library when setting up Lzo on Cloudera Hadoop

3
votes

I just followed the steps in the Cloudera Document, and had GPL Extras Parcel installed on the cluster as well as configured HDFS service via Cloudera Manager. But an error ocurred when trying to read .lzo files on HDFS: 

$hadoop fs -text /tmp/Lzo/log.txt.lzo 
INFO lzo.GPLNativeCodecLoader: Loaded native gpl library
WARN lzo.LzoCompressor: java.lang.NoSuchFiledError: lzoCompressLevelFunc
ERROR lzo.LzoCodec: Failed to load/initialize native-lzo library
-text: Fatal internal error
java.lang.RuntimeException: native-lzo library not available

I've read a dozen of posts, and known that it's caused by a failure on JNI loading lzo library, but none of them could properly solve my problem. Following are the efforts I've made:
1.All datanodes had installed lzop.
2.JAVA_LIBRARY_PATH in mapred-site.xml was set to /opt/cloudera/parcels/CDH/lib/hadoop/lib/native, which contains liblzo2.* files.
3.HADOOP_CLASSPATH was set to /usr/local/lib which contains hadoop-lzo.jar files.

What else can I do? Any suggestions would be appreciated!

2
hadoopcloudera-cdhlzo
Check stackoverflow.com/questions/23441142/… – BruceWayne
Thanks for suggestion! My cluster is a little bit different from that, cause I'm using CDH 5.6 which doesn't match any version of the HADOOP_LZO parcels from the repository, but instead a GPL EXTRAS parcel which contains hadoop-lzo-0.4.15-cdh5.6.0.jar. I'm wondering whether they are equal? – Paul Lam

2 Answers

2
votes

Problem solved! It's caused by java.lang.NoSuchFiledError. Hadoop-lzo-0.4.15 do not have the field lzoCompressLevelFunc, and when I switched to hadoop-lzo0.4.20 the WARN and the ERROR were gone.

0
votes

I got this issue as well today and spent quite a long time to figure out what the actual root cause is.

So I want to summarize the issue here: In short, the jar and the native library needs to be compatible, and the best way to ensure this is to generate (build) them from the same version of source code.

What I suffer yesterday is that I'm using hadoop-gpl-compression.jar, but the native library I'm using is built from hadoop-lzo. So there is a compatibility issue here. What I did after is to build the native library from hadoop-gpl-compression project instead of hadoop-lzo, then it works.

If you are using the jar built from hadoop-lzo, then you should also use the native library built from that library, and it is better to use the same version of source to build the native library.

If you are using the jar built from hadoop-gpl-compression project, then you should also use the native library built from it, and it is better to use the same version of source to build the native library.