cdh5 hadoop redhat 本地仓库配置

cdh5 hadoop redhat 本地仓库配置

cdh5 在网站上的站点位置：

http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/

在RHEL6上配置指向这个repo非常简单，只要把：

http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/cloudera-cdh5.repo

下载存储到本地即可：

/etc/yum.repos.d/cloudera-cdh5.repo

但是如果是离线情况下，网络连接不可用，就需要把整个资源镜像到本地，然后在cloudera-cdh5.repo中配置。我写了一个脚本，用于下载整个站点。虽然用wget一个命令可以搞定，为了练练shell脚本，我还是写了一个。基本思路就是分析网页，找到资源链接，存储到本地目录。脚本中：PATH_MUST_BE_EXSITED必须指向已经存在的本地目录。不废话，上代码：

#!/bin/bash
#
# @file
#   cdh5_rhel6-downloads.sh
#
# @date
#   2014-12-18
#
# @author
#   cheungmine
#
# @version
#   0.0.1pre
#
# downloads all from CDH_URL_PREFIX:
#   http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/
#
################################################################################
# specify where you want to save downloaded packages here:
#
PATH_MUST_BE_EXSITED="../libs/cdh"

# get real path from relative path
function real_path() {
    \cd "$1"
    /bin/pwd
}

# server dist resources:
#
CDH_URL_PREFIX="http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh"

CDH_GPGKEY=$CDH_URL_PREFIX"/RPM-GPG-KEY-cloudera"
CDH_REPO=$CDH_URL_PREFIX"/cloudera-cdh5.repo"

CDH5_REPODATA=$CDH_URL_PREFIX"/5/repodata/"
CDH5_RPMS_NOARCH=$CDH_URL_PREFIX"/5/RPMS/noarch/"
CDH5_RPMS_X86_64=$CDH_URL_PREFIX"/5/RPMS/x86_64/"

# source packages not used:
CDH5_SRPMS=$CDH_URL_PREFIX"/5/SRPMS/"

# get local absolute path for storing the downloaded:
CDH5_LOCALPATH=$(real_path $PATH_MUST_BE_EXSITED)
echo "**** downloaded packages will be stored in folder: "$CDH5_LOCALPATH

# first we get index pages:
#
repodata_html=$CDH5_LOCALPATH"/.repodata.index.html"
x86_64_html=$CDH5_LOCALPATH"/.x86_64.index.html"
noarch_html=$CDH5_LOCALPATH"/.noarch.index.html"

wget -c $CDH5_REPODATA -P $CDH5_LOCALPATH -O $repodata_html
wget -c $CDH5_RPMS_NOARCH -P $CDH5_LOCALPATH -O $noarch_html
wget -c $CDH5_RPMS_X86_64 -P $CDH5_LOCALPATH -O $x86_64_html

wget -c $CDH_GPGKEY -P $CDH5_LOCALPATH
wget -c $CDH_REPO -P $CDH5_LOCALPATH

# download repodata
# CDH5_REPODATA
repodata_dir=$CDH5_LOCALPATH"/5/repodata"
mkdir -p $repodata_dir
echo -e "process file: '$repodata_html'"
while read line
do
    # start with: <td><a href="
    a=`echo $line | sed -n '/<td><a href="/p'`

    if [ -n "$a" ]; then
        b=`echo $a | sed -n '/Parent Directory/p'`

        # do including: Parent Directory
        if [ -z "$b" ]; then
            # end with: </a></td>
            b=`echo $a | sed -n '/<\/a><\/td>/p'`

            if [ -n "$b" ]; then
                a=`echo $a | sed -e 's/.*<td><a href="//;s/">.*//'`
                url=$CDH5_REPODATA$a

                echo -e "download: $url"

                wget -c $url -P $repodata_dir -O $repodata_dir/$a
            fi
        fi
    fi
done < $repodata_html

# download noarch
# CDH5_RPMS_NOARCH
noarch_dir=$CDH5_LOCALPATH"/5/RPMS/noarch"
mkdir -p $noarch_dir
echo -e "process file: '$noarch_html'"
while read line
do
    # start with: <td><a href="
    a=`echo $line | sed -n '/<td><a href="/p'`

    if [ -n "$a" ]; then
        b=`echo $a | sed -n '/Parent Directory/p'`

        # do including: Parent Directory
        if [ -z "$b" ]; then
            # end with: </a></td>
            b=`echo $a | sed -n '/<\/a><\/td>/p'`

            if [ -n "$b" ]; then
                a=`echo $a | sed -e 's/.*<td><a href="//;s/">.*//'`
                url=$CDH5_RPMS_NOARCH$a

                echo -e "download: $url"

                wget -c $url -P $noarch_dir -O $noarch_dir/$a
            fi
        fi
    fi
done < $noarch_html

# download x86_64
# CDH5_RPMS_X86_64
x86_64_dir=$CDH5_LOCALPATH"/5/RPMS/x86_64"
mkdir -p $x86_64_dir
echo -e "process file: '$x86_64_html'"
while read line
do
    # start with: <td><a href="
    a=`echo $line | sed -n '/<td><a href="/p'`

    if [ -n "$a" ]; then
        b=`echo $a | sed -n '/Parent Directory/p'`

        # do including: Parent Directory
        if [ -z "$b" ]; then
            # end with: </a></td>
            b=`echo $a | sed -n '/<\/a><\/td>/p'`

            if [ -n "$b" ]; then
                a=`echo $a | sed -e 's/.*<td><a href="//;s/">.*//'`
                url=$CDH5_RPMS_X86_64$a

                echo -e "download: $url"

                wget -c $url -P $x86_64_dir -O $x86_64_dir/$a
            fi
        fi
    fi
done < $x86_64_html

# TODO: do we need to check all packages?
# remove index pages:
rm -f $repodata_html $x86_64_html $noarch_html

echo "download all packages successfully."

上面脚本可以重复多次运行，不会重复下载。PATH_MUST_BE_EXSITED里面保存了cdh5全部内容。最后把PATH_MUST_BE_EXSITED的全部内容上传到本地ftp服务器上，确保可以访问：

ftp://your-server-ip/pub/libs/cdh/

然后在需要访问的RHEL6机器上，增加一个repo文件，我的是:

# /etc/yum.repos.d/cdh5.repo

[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat	or CentOS 6 x86_64
name     = Cloudera's Distribution for Hadoop, Version 5
baseurl  = ftp://your-server-ip/pub/libs/cdh/5/
gpgkey   = ftp://your-server-ip/pub/libs/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1
enabled  = 1

好了，cdh的本地仓库建好了。只要能访问到这个ftp的服务器中存在这个 /etc/yum.repos.d/cdh5.repo 文件，就可以安装cdh hadoop的软件包。比如安装个zookeeper服务端：

#Installing the ZooKeeper Base Package
  $ yum install zookeeper

# Installing the ZooKeeper Server Package
  $ yum install zookeeper-server

# start zookeeper-server
  $ service zookeeper-server init --myid=1
Using myid of 1

zookeeper安装在/usr/lib/zookeeper，进入bin目录启动和关闭:

$ zkServer.sh start
$ zkServer.sh stop

OK！