Snappy not available error on EMR

0
votes

I'm using Spark on EMR with one master and one core node, I'm facing the folowing issue:

java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.

I set the java.library.path to /usr/lib/hadoop/lib/native/:/usr/lib/hadoop-lzo/lib/native/

checknative gives me this on master and core node:

16/04/28 15:03:27 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
16/04/28 15:03:27 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so

EMR VERSION: emr-4.6.0

Hadoop distribution:Amazon 2.7.2

Applications:Spark 1.6.1

1
hadoopamazon-web-servicesapache-sparksnappy

1 Answers

0
votes

I finally found the solution, I set the LD_LIBRARY_PATH and it works now !!

export LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native/

I did this operation on all nodes. I hove it will help someone :) !