$ sudo yum install hadoop-0.20-nativeInstall on Ubuntu systems:
$ sudo apt-get install hadoop-0.20-nativeThis should create a directory under /usr/lib/hadoop/lib/native/ which contains some native hadoop libraries. Then create environment config for Flume:
$ cp /usr/lib/flume/bin/flume-env.sh.template /usr/lib/flume/bin/flume-env.shAnd update the last line in the file to be: For 32-bit platform
$ export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-i386-32For 64-bit platform
$ export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-amd64-64Next update the flume’s configuration file under “/etc/flume/conf/flume-site.xml” on the collector node to:
And then finally restart the flume-node:flume.collector.dfs.compress.codec SnappyCodec Writes formatted data compressed in specified codec to dfs. Value is None, GzipCodec, DefaultCodec (deflate), BZip2Codec, SnappyCodec or any other Codec Hadoop is aware of
$ /etc/init.d/flume-node restartYou next file update in HDFS will look something like the following:
-rw-r--r-- 3 flume supergroup 0 2011-10-21 14:01 /data/traffic/Y2011_M9_W37_D254/R0_P0/C1_20111021-140124175+1100.955183363700204.00000244.snappy.tmp -rw-r--r-- 3 flume supergroup 35156526 2011-10-20 16:51 /data/traffic/Y2011_M9_W37_D254/R0_P0/C2_20111020-164928958+1100.780424004236302.00000018.snappy -rw-r--r-- 3 flume supergroup 830565 2011-10-20 17:15 /data/traffic/Y2011_M9_W37_D254/R0_P0/C2_20111020-171423368+1100.781918413572302.00000018.snappy -rw-r--r-- 3 flume supergroup 0 2011-10-20 17:19 /data/traffic/Y2011_M9_W37_D254/R0_P0/C2_20111020-171853599+1100.782188644505302.00000042.snappy.tmp -rw-r--r-- 3 flume supergroup 1261171 2011-10-20 17:37 /data/traffic/Y2011_M9_W37_D254/R0_P0/C2_20111020-173728225+1100.783303271088302.00000018.snappy -rw-r--r-- 3 flume supergroup 2128701 2011-10-20 17:40 /data/traffic/Y2011_M9_W37_D254/R0_P0/C2_20111020-174024045+1100.783479090669302.00000046.snappyHappy Fluming..