Hadoop源代码编译

Environment

OS: Ubuntu 12.04 LTS

Kernel Version: 3.8.0-33-generic

Hadoop: 2.2.0

Pre-Compile

官方的BUILDING.txt给出的安装需求

image

安装JDK

1
sudo apt-get install openjdk-7-jdk

安装Maven

1
sudo apt-get install maven

安装Findbugs

Download: http://findbugs.sourceforge.net/

设置环境变量

1
2
3
4
vi .bashrc
#set findbugs path
export FINDBUGS_HOME=/home/killua/Dev/findbugs-2.0.3
export PATH=$PATH:$FINDBUGS_HOME/bin

安装ProtocolBuffer

Download: https://code.google.com/p/protobuf/

在安装protocolbuffer之前需要先安装g++

1
sudo apt-get install g++

安装protocolbuffer

1
2
3
4
cd  protobuf-2.5.0
./configure
make
make install

安装CMake

1
sudo apt-get install cmake

Compile

1
mvn package -Pdist -DskipTests -Dtar

image

编译后的结果可以在hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0中看到

image

常见问题

1) error: cannot access AbstractLifeCycle

在编译的过程中出现/home/killua/Workspace/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[88,11] error: cannot access AbstractLifeCycle错误

经过查证发现是hadoop2.2.0的一个bug,具体参见https://issues.apache.org/jira/browse/HADOOP-10110

解决方法:

修改hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/pom.xml,将

1
2
3
4
5
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty</artifactId>
<scope>test</scope>
</dependency>

修为

1
2
3
4
5
6
7
8
9
10
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty</artifactId>
<scope>test</scope>
</dependency>
坚持原创技术分享,您的支持将鼓励我继续创作!