Dockerfile完成Hadoop2.6的伪分布式搭建

在 《Docker中搭建Hadoop-2.6单机伪分布式集群》中在容器中操作来搭建伪分布式的Hadoop集群，这一节中将主要通过Dokcerfile 来完成这项工作。

1 获取一个简单的Docker系统镜像,并建立一个容器。

　　 这里我选择下载CentOS镜像

docker pull centos

　　通过docker tag命令将下载的CentOS镜像名称换成centos，并删除老标签

docker tag docker.io/centos centos
docker rmr docker.io/centos

2. JDK的安装和配置

　　去Oracle官网提前下载好所需的jdk。

　　建立文件夹，并将jdk copy到文件夹下

[root@centos-docker ~]# mkdir centos-jdk
[root@centos-docker ~]# mv jdk-7u79-linux-x64.tar.gz ./centos-jdk/
[root@centos-docker ~]# cd centos-jdk/
[root@centos-docker centos-jdk]# ls
jdk-7u79-linux-x64.tar.gz

　　在centos-jdk文件夹中建立Dockerfile，其内容如下：

# CentOS with JDK
# Author        amei

# build a new image with basic  centos
FROM centos
# who is the author
MAINTAINER amei

# make a new directory to store the jdk files
RUN mkdir /usr/local/java

# copy the jdk  archive to the image,and it will automaticlly unzip the tar file
ADD jdk-7u79-linux-x64.tar.gz /usr/local/java/

# make a symbol link
RUN ln -s /usr/local/java/jdk1..0_79 /usr/local/java/jdk

# set environment variables
ENV JAVA_HOME /usr/local/java/jdk
ENV JRE_HOME ${JAVA_HOME}/jre
ENV CLASSPATH .:${JAVA_HOME}/lib:${JRE_HOME}/lib
ENV PATH ${JAVA_HOME}/bin:$PATH

　　根据Dokcerfile创建新镜像：

# 注意后边的 . 不能忘了
[root@centos-docker centos-jdk]# docker build -t="centos-jdk" .
Sending build context to Docker daemon 153.5 MB
Step  : FROM centos
 ---> e8f1bdb3b6a7
.....................................
Step  : ENV PATH ${JAVA_HOME}/bin:$PATH
 ---> Running in 5ecbe2fac774
 ---> ad1110b84433
Removing intermediate container 5ecbe2fac774
Successfully built ad1110b84433

　　查看新建立的镜像

[root@centos-docker centos-jdk]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos-jdk          latest              ad1110b84433         minutes ago        MB
centos              latest              e8f1bdb3b6a7         weeks ago         196.7 MB

　　建立容器，查看新的镜像中的JDK是否正确

[root@centos-docker centos-jdk]# docker run -it centos-jdk /bin/bash
[root@b665dbff9965 /]# java -version　　　　# 出来结果表明配置没问题
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) -Bit Server VM (build 24.79-b02, mixed mode)
[root@b665dbff9965 /]# echo $JAVA_HOME
/usr/local/java/jdk

3. 在前一步基础上安装ssh

　　建立新的文件夹，并在其下建立Dokcerfile文件，其内容为：

# build a new image with centos-jdk
FROM centos-jdk
# who is the author
MAINTAINER amei

# install openssh
RUN yum -y  install openssh-server openssh-clients

#generate key files
RUN ssh-keygen -q -t rsa -b 2048 -f /etc/ssh/ssh_host_rsa_key -N ''
RUN ssh-keygen -q -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ''
RUN ssh-keygen -q -t dsa -f /etc/ssh/ssh_host_ed25519_key  -N ''

# login localhost without password
RUN ssh-keygen -f /root/.ssh/id_rsa -N ''
RUN touch /root/.ssh/authorized_keys
RUN cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

# set password of root
RUN echo "root:1234" | chpasswd

# open the port 22
EXPOSE 22
# when start a container it will be executed
CMD ["/usr/sbin/sshd","-D"]

　　利用此Dockerfile 建立镜像：

[root@centos-docker centos-jdk-ssh]# docker build -t "centos-jdk-ssh" .
Sending build context to Docker daemon  2.56 kB
Step  : FROM centos-jdk
 ---> ad1110b84433
。。。。。。。。。。。。。。。。。。。。。。。。
Successfully built 5286623a6cc0

　　验证建立好的镜像：

#在刚才的镜像之上建立容器
[root@centos-docker centos-jdk-ssh]# docker run -it centos-jdk-ssh /bin/bash
[root@118f3d29fc73 /]# /usr/sbin/sshd　　　　　　#开启sshd服务
[root@118f3d29fc73 /]# ssh root@localhost　　　 #登陆到本机
The authenticity of host 'localhost (::1)' can't be established.　　　　# 观察确实不用密码即可登陆
ECDSA key fingerprint is b7:f0:::c9:ca::8b::0d:::6f::4f:.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
[root@118f3d29fc73 ~]# exit　　　　#退出容器
logout
Connection to localhost closed.

4. 安装Hdoop2.6

　　首先先下载好hadoop安装包。

　　建立文件夹，并在文件夹下建立如下几个文件。

　　编辑core-site.xml文件

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>file:/data/hadoop/tmp</value>
        </property>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost:9000</value>
        </property>
</configuration>

　　编辑hdfs-site.xml文件

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
        <property>
                <name>dfs.replication</name>
                <value></value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/data/hadoop/dfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/data/hadoop/dfs/data</value>
        </property>
</configuration>

　　在其下建立Dokcerfile文件，其内容为：

# build a new image with  centos-jdk-ssh
FROM centos-jdk-ssh
# who is the author
MAINTAINER amei

# install some important software
RUN yum -y install net-tools  which

# copy the hadoop  archive to the image,and it will automaticlly unzip the tar file
ADD hadoop-2.6.0.tar.gz /usr/local/

# make a symbol link
RUN ln -s /usr/local/hadoop-2.6.0 /usr/local/hadoop

# copy the configuration file to image
COPY core-site.xml /usr/local/hadoop/etc/hadoop/
COPY hdfs-site.xml /usr/local/hadoop/etc/hadoop/

# change hadoop environment variables
RUN sed -i "s?JAVA_HOME=\${JAVA_HOME}?JAVA_HOME=/usr/local/java/jdk?g" /usr/local/hadoop/etc/hadoop/hadoop-env.sh

# set environment variables
ENV HADOOP_HOME /usr/local/hadoop
ENV PATH ${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

　　此时文件夹下的文件有：

[root@centos-docker centos-hadoop]# ll
total 190704
-rw-r--r--. 1 root root       403 Aug  7 06:52 core-site.xml
-rw-r--r--. 1 root root       708 Aug  7 06:52 Dockerfile
-rwxr-x---. 1 root root 195257604 Aug  7 04:44 hadoop-2.6.0.tar.gz
-rw-r--r--. 1 root root       546 Aug  7 06:25 hdfs-site.xml

　　建立镜像：

docker build -t "centos-hadoop" .

　　查看镜像：

[root@centos-docker centos-hadoop]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos-hadoop       latest              64b9d221973b         minutes ago       MB
centos-jdk-ssh      latest              5286623a6cc0        About an hour ago    MB
centos-jdk          latest              ad1110b84433         hours ago          MB

　　建立容器测试镜像：

[root@centos-docker centos-hadoop]# docker run -it centos-hadoop /bin/bash  #开启容器
[root@889d94ef9cbc /]#/usr/sbin/sshd　　　　　　　　　　　　#开启sshd服务
[root@889d94ef9cbc /]# hdfs namenode -format　　　　　　　　#格式化namenode
// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = 889d94ef9cbc/172.17.0.2
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.6.0
............................................................
16/08/06 22:56:36 INFO common.Storage: Storage directory /data/hadoop/dfs/name has been successfully formatted.
16/08/06 22:56:37 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/08/06 22:56:37 INFO util.ExitUtil: Exiting with status 0
16/08/06 22:56:37 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at 889d94ef9cbc/172.17.0.2
************************************************************/
[root@889d94ef9cbc /]# start-dfs.sh 　　# 开启hdfs
[root@889d94ef9cbc /]# jps　　　　　　#查看开启的应用程序
 SecondaryNameNode
 DataNode
 Jps
 NameNode
[root@889d94ef9cbc /]# hadoop dfsadmin -report　　#查看HDFS状况
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Configured Capacity:  (9.99 GB)
Present Capacity:  (9.08 GB)
DFS Remaining:  (9.08 GB)
DFS Used:  ( KB)
DFS Used%: 0.00%
Under replicated blocks:
Blocks with corrupt replicas:
Missing blocks: 

-------------------------------------------------
Live datanodes ():

Name: 127.0.0.1: (localhost)
Hostname: 889d94ef9cbc
Decommission Status : Normal
Configured Capacity:  (9.99 GB)
DFS Used:  ( KB)
Non DFS Used:  (933.54 MB)
DFS Remaining:  (9.08 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.87%
Configured Cache Capacity:  ( B)
Cache Used:  ( B)
Cache Remaining:  ( B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers:
Last contact: Sat Aug  :: UTC 

5. 将前边的步骤合在一起，用一个Dockerfile 来完成

　　建立一个新的文件夹，文件夹要包含建立进行所需的资源。

[root@centos-docker centos-hadoop]# ll
total
-rw-r--r--.  root root        Aug   : core-site.xml　　
-rw-r--r--.  root root        Aug   : Dockerfile
-rwxr-x---.  root root  Aug   : hadoop-2.6..tar.gz
-rw-r--r--.  root root        Aug   : hdfs-site.xml
-rwxr-x---.  root root  Aug   : jdk-7u79-linux-x64.tar.gz

　　Dockerfile中的内容为：

　　

# build a new hadoop image with basic  centos
FROM centos
# who is the author
MAINTAINER amei

# install some important softwares
RUN yum -y install openssh-server openssh-clients  net-tools  which

####################Configurate JDK################################
# make a new directory to store the jdk files
RUN mkdir /usr/local/java

# copy the jdk  archive to the image,and it will automaticlly unzip the tar file
ADD jdk-7u79-linux-x64.tar.gz /usr/local/java/

# make a symbol link
RUN ln -s /usr/local/java/jdk1..0_79 /usr/local/java/jdk

###################Configurate SSH#################################
#generate key files
RUN ssh-keygen -q -t rsa -b  -f /etc/ssh/ssh_host_rsa_key -N ''
RUN ssh-keygen -q -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ''
RUN ssh-keygen -q -t dsa -f /etc/ssh/ssh_host_ed25519_key  -N ''

# login localhost without password
RUN ssh-keygen -f /root/.ssh/id_rsa -N ''
RUN cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

###################Configurate Hadoop##############################
# copy the hadoop  archive to the image,and it will automaticlly unzip the tar file
ADD hadoop-2.6..tar.gz /usr/local/

# make a symbol link
RUN ln -s /usr/local/hadoop-2.6. /usr/local/hadoop

# copy the configuration file to image
COPY core-site.xml /usr/local/hadoop/etc/hadoop/
COPY hdfs-site.xml /usr/local/hadoop/etc/hadoop/

# change hadoop environment variables
RUN sed -i "s?JAVA_HOME=\${JAVA_HOME}?JAVA_HOME=/usr/local/java/jdk?g" /usr/local/hadoop/etc/hadoop/hadoop-env.sh

################### Integration configuration #######################
# set environment variables
ENV JAVA_HOME /usr/local/java/jdk
ENV JRE_HOME ${JAVA_HOME}/jre
ENV CLASSPATH .:${JAVA_HOME}/lib:${JRE_HOME}/lib
ENV HADOOP_HOME /usr/local/hadoop
ENV PATH ${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${JAVA_HOME}/bin:$PATH

# set password of root
RUN echo "root:1234" | chpasswd
# when start a container it will be executed
CMD ["/usr/sbin/sshd"]

以此Dockerfile来建立Hadoop镜像

docker build -t "centos-hadoop" .

6. 后话

Dockerfile和jdk,hadoop文件以及其它的配置文件都打包好放在百度云上，解压之后可以直接在目录中敲入命令  docker build -t "centos-hadoop" .  建立Hadoop镜像，不过前提是你得先有一个centos镜像。

　　http://pan.baidu.com/s/1dE8NCo5