The native hadoop library is imperative for gzip to work. Supported Platforms. The native hadoop library is supported on.nix platforms only. The library does not to work with Cygwin or the Mac OS X platform. The native hadoop library is mainly used on the GNU/Linus platform and has been tested on these distributions: RHEL4/Fedora; Ubuntu; Gentoo.
- Hadoop Native Libraries. Hadoop Streaming. Hadoop Archives. HDFS User Guide. HDFS Architecture. Unfortunately it is known not to work on Cygwin and Mac OS X and has mainly been used on the GNU/Linux platform. The native hadoop library is not built.
- This patch makes the autoconf script work on Mac OS X. LZO needs to be installed (including the optional shared libraries) for the compile to succeed. You'll want to regenerate the configure script using autoconf after applying this patch.
- Nov 21, 2015 How-to Install Apache Hadoop on Mac OS X 10.11 El Capitan Step-by-Step Guide November 21, 2015 Uncategorized Comments: 0 Hadoop 2.X QuickStart on Mac OS X 10.11 El Capitan.
build-native-mac.sh
| brew install snappy |
| CPPFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib ant compile-native |
| chmod 644 build/native/Mac_OS_X-x86_64-64/lib/* |
| ls -l /usr/local/lib/libsnappy.dylib |
| ls -l build/native/Mac_OS_X-x86_64-64/lib/libhadoop.dylib |
Hadoop Native Library Mac Os X 10
native-mac.diff
| commit a5681a94b83e7314a46863f811226e4e51967595 |
| Author: David Phillips <david@acz.org> |
| Date: Wed Oct 10 18:34:24 2012 -0700 |
| Compile native libraries on Mac OS X |
| diff --git a/src/native/Makefile.am b/src/native/Makefile.am |
| index c70de7d..6268877 100644 |
| --- a/src/native/Makefile.am |
| +++ b/src/native/Makefile.am |
| @@ -33,9 +33,7 @@ export PLATFORM = $(shell echo $$OS_NAME | tr [A-Z] [a-z]) |
| AM_CPPFLAGS = @JNI_CPPFLAGS@ -I$(HADOOP_NATIVE_SRCDIR)/src |
| -Isrc/org/apache/hadoop/io/compress/zlib |
| - -Isrc/org/apache/hadoop/io/compress/snappy |
| - -Isrc/org/apache/hadoop/io/nativeio |
| - -Isrc/org/apache/hadoop/security |
| + -Isrc/org/apache/hadoop/io/compress/snappy |
| AM_LDFLAGS = @JNI_LDFLAGS@ -m$(JVM_DATA_MODEL) |
| AM_CFLAGS = -g -Wall -fPIC -O2 -m$(JVM_DATA_MODEL) |
| @@ -43,13 +41,7 @@ lib_LTLIBRARIES = libhadoop.la |
| libhadoop_la_SOURCES = src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c |
| src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c |
| src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c |
| - src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c |
| - src/org/apache/hadoop/security/getGroup.c |
| - src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c |
| - src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c |
| - src/org/apache/hadoop/io/nativeio/file_descriptor.c |
| - src/org/apache/hadoop/io/nativeio/errno_enum.c |
| - src/org/apache/hadoop/io/nativeio/NativeIO.c |
| + src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c |
| libhadoop_la_LDFLAGS = -version-info 1:0:0 $(AM_LDFLAGS) |
| libhadoop_la_LIBADD = -ldl -ljvm |
| diff --git a/src/native/acinclude.m4 b/src/native/acinclude.m4 |
| index 266ac87..48481b3 100644 |
| --- a/src/native/acinclude.m4 |
| +++ b/src/native/acinclude.m4 |
| @@ -13,8 +13,10 @@ AC_CACHE_CHECK([Checking for the 'actual' dynamic-library for '-l$1'], ac_cv_lib |
| ac_cv_libname_$1='`objdump -p conftest | grep NEEDED | grep $1 | sed 's/W*NEEDEDW*(.*)W*$/'1'/'`' |
| elif test ! -z '`which ldd | grep -v 'no ldd'`'; then |
| ac_cv_libname_$1='`ldd conftest | grep $1 | sed 's/^[[[^A-Za-z0-9]]]*([[[A-Za-z0-9.]]]*)[[[^A-Za-z0-9]]]*=>.*$/'1'/'`' |
| + elif test ! -z '`which otool | grep -v 'no otool'`'; then |
| + ac_cv_libname_$1='`otool -L conftest | grep $1 | sed -e 's/^[ ]*//' -e 's/ .*//' -e 's/.*/(.*)$/1/'`'; |
| else |
| - AC_MSG_ERROR(Can't find either 'objdump' or 'ldd' to compute the dynamic library for '-l$1') |
| + AC_MSG_ERROR(Can't find either 'objdump', 'ldd' or 'otool' to compute the dynamic library for '-l$1') |
| fi |
| else |
| ac_cv_libname_$1=libnotfound.so |
| diff --git a/src/native/configure.ac b/src/native/configure.ac |
| index 71ff66d..87b34a1 100644 |
| --- a/src/native/configure.ac |
| +++ b/src/native/configure.ac |
| @@ -56,7 +56,7 @@ JNI_LDFLAGS=' |
| if test $JAVA_HOME != ' |
| then |
| JNI_LDFLAGS='-L$JAVA_HOME/jre/lib/$OS_ARCH/server' |
| - JVMSOPATH=`find $JAVA_HOME/jre/ -name libjvm.so | head -n 1` |
| + JVMSOPATH=`find $JAVA_HOME/jre/ -name 'libjvm.*' | head -n 1` |
| JNI_LDFLAGS='$JNI_LDFLAGS -L`dirname $JVMSOPATH`' |
| fi |
| ldflags_bak=$LDFLAGS |
Mac Hadoop Native Library
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment