Hadoop使用lzo压缩格式

在hadoop中搭建lzo环境:

1. wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz
2. export CFLAGS=-m64
3. ./configure -enable-shared -prefix=/usr/local/hadoop/lzo/
4. make && make test && make install

在hadoop-env.sh中

export LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib  

将如下配置到 core-site.xml 中：

<property>

<name>io.compression.codecs</name>

<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</value>

</property>

<property>

<name>io.compression.codec.lzo.class</name>

<value>com.hadoop.compression.lzo.LzoCodec</value>

</property>

将如下配置到mapred-site.xml中：

<property>

<name>mapred.compress.map.output</name>

<value>true</value>

</property>

<property>

<name>mapred.map.output.compression.codec</name>

<value>com.hadoop.compression.lzo.LzoCodec</value>

</property>

<property>

<name>mapred.child.env</name>

<value>LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib</value>

</property>