Storm-1.0.1+ZooKeeper-3.4.8+Netty-4.1.3 HA集群安装

Storm-1.0.1+ZooKeeper-3.4.8+Netty-4.1.3 HA集群安装

    下载Storm-1.0.1
        http://mirrors.tuna.tsinghua.edu.cn/apache/storm/apache-storm-1.0.1/apache-storm-1.0.1.tar.gz

    下载ZooKeeper-3.4.8
        http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz

    下载Netty-4.1.3.Final.tar.bz2
        http://akamai.bintray.com/e7/e769383d99e244b7173b496f90f9b4a00574f8a33cdec2fe40b54f957a373e01?__gda__=exp=1469008981~hmac=7f3864e58af529209d0c0460cc3d5a73f267d7b550ad82bf14c0f645f8608e74&response-content-disposition=attachment%3Bfilename%3D%22netty-4.1.3.Final.tar.bz2%22&response-content-type=application%2Fx-bzip2

    软件包如下：
        apache-storm-1.0.1.tar.gz
        jdk-7u65-linux-x64.tar.gz
        netty-4.1.3.Final.tar.bz2
        zookeeper-3.4.8.tar.gz

    集群规划
        主机            IP                    Soft                        Process
        nimbus01        192.168.14.162        Netty、Storm                nimbus、core、logviewer
        nimbus02        192.168.14.156        Netty、Storm                nimbus、logviewer
        supervisor01    192.168.14.59        Netty、ZooKeeper、Storm        supervisor、logviewer
        supervisor02    192.168.14.119        Netty、ZooKeeper、Storm        supervisor、logviewer
        supervisor03    192.168.14.139        Netty、ZooKeeper、Storm        supervisor、logviewer

    一、配置免密码登录

    二、安装Jdk
        在所有节点上安装jdk
            cd /usr/local/soft/
            tar -zxvf jdk-7u65-linux-x64.tar.gz
            cd jdk1.7.0_65
            vim /etc/profile
                export JAVA_HOME=/usr/local/soft/jdk1.7.0_65
                export PATH=.:$JAVA_HOME/bin:$PATH
            source /etc/profile

    三、安装ZooKeeper-3.4.8
        在supervisor01节点上配置
            cd /usr/local/soft/
            tar -zxvf zookeeper-3.4.8.tar.gz
            cd zookeeper-3.4.8
            mv conf/zoo_sample.cfg conf/zoo.cfg
            vim conf/zoo.cfg
                tickTime=2000
                initLimit=10
                syncLimit=5
                dataDir=/usr/local/soft/zookeeper-3.4.8/data
                clientPort=2181
                server.1=suspervisor01:2888:3888
                server.2=suspervisor02:2888:3888
                server.3=suspervisor03:2888:3888
            mkdir data && echo "1">data/myid
        复制supervisor01节点上的ZooKeeper到supervisor02上
            scp -r /usr/local/soft/zookeeper-3.4.8 root@supervisor02:/usr/local/soft/
                echo "2" >/usr/local/soft/zookeeper-3.4.8/data/myid
        复制supervisor01节点上的ZooKeeper到supervisor03上
            scp -r /usr/local/soft/zookeeper-3.4.8 root@supervisor03:/usr/local/soft/
                echo "3" >/usr/local/soft/zookeeper-3.4.8/data/myid

    四、安装Storm-1.0.1
        在nimbus01节点上配置
            cd /usr/local/soft/
            tar -zxvf apache-storm-1.0.1.tar.gz
            cd apache-storm-1.0.1
            vim conf/storm.yaml
                # #### Storm dependents The ZooKeeper Cluster ####
                 storm.zookeeper.servers:
                     - "supervisor01"
                     - "supervisor02"
                     - "supervisor03"

                # #### Storm Nimbus Nodes HA ####
                 nimbus.seeds: ["nimbus01", "nimbus02"]

                # #### Storm Local Storage ####
                 storm.local.dir: "/usr/local/soft/apache-storm-1.0.1/localdir"

                # #### Storm Supervisor Nodes Worker Process  ####
                 supervisor.slots.ports:
                    - 6700
                    - 6701
                    - 6702
                    - 6703

                # #### Not Selector ZeroMQ, Storm Use Netty Transport Protocol ####
                 storm.messaging.transport: "org.apache.storm.messaging.netty.Context"
                 storm.messaging.netty.server_worker_threads: 1
                 storm.messaging.netty.client_worker_threads: 1
                 storm.messaging.netty.buffer_size: 5242880
                 storm.messaging.netty.max_retries: 300
                 storm.messaging.netty.max_wait_ms: 1000
                 storm.messaging.netty.min_wait_ms: 100
            mkdir localdir
        复制nimbus01节点上的Storm到nimbus02、supervisor01、supervisor02、supervisor03上
            scp -r /usr/local/soft/apache-storm-1.0.1 root@nimbus02:/usr/local/soft/
            scp -r /usr/local/soft/apache-storm-1.0.1 root@supervisor01:/usr/local/soft/
            scp -r /usr/local/soft/apache-storm-1.0.1 root@supervisor02:/usr/local/soft/
            scp -r /usr/local/soft/apache-storm-1.0.1 root@supervisor03:/usr/local/soft/

    五、启动集群
        1、ZooKeeper
            启动集群
                for i in supervisor01 supervisor02 supervisor03; do echo $i; ssh $i "source /etc/profile; /usr/local/soft/zookeeper-3.4.8/bin/zkServer.sh start"; done
            关闭集群
                for i in supervisor01 supervisor02 supervisor03; do echo $i; ssh $i "source /etc/profile; /usr/local/soft/zookeeper-3.4.8/bin/zkServer.sh stop"; done
            查看ZooKeeper进程状态
                for i in supervisor01 supervisor02 supervisor03; do echo $i; ssh $i "source /etc/profile; /usr/local/soft/zookeeper-3.4.8/bin/zkServer.sh status"; done
            查看进程
                for i in supervisor01 supervisor02 supervisor03; do echo $i; ssh $i "source /etc/profile; jps"; done

        2、启动Storm（需要确保ZooKeeper集群已经启动）
            启动nimbus（storm.yaml中nimbus.seeds: []的节点都需要启动nimbus进程）
                在nimbus01上启动如下进程
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm nimbus &
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm ui &
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm logviewer &

                在nimbus02上启动如下进程
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm nimbus &
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm logviewer &

                在supervisor01上启动如下进程
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm supervisor &
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm logviewer &

                在supervisor02上启动如下进程
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm supervisor &
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm logviewer &

                在supervisor03上启动如下进程
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm supervisor &
                    nohup /usr/local/soft/apache-storm-1.0.1/bin/storm logviewer &