一、hadoop 及 hadoop的环境搭建

一、Hadoop引言

Hadoop是在2006年雅虎从Nutch(给予Java爬虫框架)工程中剥离一套分布式的解决方案.该方案参考了Goggle的GFS(Google File System)和MapReduce论文,当时发布的版本称为Hadoop-1.x,并且在2010年雅虎对Hadoop做又一次升级,该次升级的目的是优化了Hadoop的MapReduce框架,使得Hadoop更加易用,用户只需要少许配置,就可以使用hadoop实现海量数据存储和大规模数据集的分析.一个由Apache基金会所开发的分布式系统基础架构。

HDFS: hadoop distribute filesysterm

Map Reduce:hadoop中的分布式计算框架,实现对海量数据并行分析和计算.

YARN： hadoop 的资源调度系统

Common：以上三大组件的底层支撑组件，主要提供基础工具包和 RPC 框架等

二、HDFS (存储)

I、Hadoop Eco System(hadoop の生态圈)

HDFS:分布式存储系统

mapreduce:并行计算框架

hbase:基于HDFS之上一款NoSQL数据库(名符其实海量数据存储解决方案)

hive:会一款SQL的解析引擎,可以将SQL翻译成MapReduce任务,将任务提交给MapReduce框架.

flume:分布式日志采集系统,用于搜集海量数据,并且存储到HDFS/Hbase.

Kafka:分布式消息系统,实现分布系统间解耦和海量数据的缓冲.

zookeeper:分布式协调服务,用于服务注册中心/配置中心/集群选举/状态监测/分布式锁

II、大数据分析方案哪些?

Map Reduce:代表基于磁盘离线大数据静态批处理框架-延迟较高30分钟+

Spark :代表基于内存近实时(离线)大数据静态批处理框架-几乎是Map Reduce的10~100倍速度

Storm|Spark Streaming| Flink|Kafka Stream:实时的流(流程)处理框架,达到对记录级别的数据显示毫秒级处理.

I、HDFS 环境搭建

1.安装JDK

Window 安装64 bit CentOS(需要额外开启Intel 虚拟化技术)

参考:https://blog.csdn.net/yuzongtao/article/details/44700927

安装JDK配置JAVA_HOME

[root@CentOS ~]# yum install lrzsz -y

[root@CentOS ~]# rz #回车

rz waiting to receive.

Starting zmodem transfer.  Press Ctrl+C to cancel.

Transferring jdk-8u171-linux-x64.rpm...

  100%  171154 KB    17115 KB/sec    00:00:10       0 Errors

[root@CentOS ~]# rpm -ivh jdk-8u171-linux-x64.rpm 

[root@CentOS ~]# vi /root/.bashrc

JAVA_HOME=/usr/java/latest

PATH=$PATH:$JAVA_HOME/bin

CLASSPATH=.

export JAVA_HOME

export PATH

export CLASSPATH

[root@CentOS ~]# source /root/.bashrc

[root@CentOS ~]# jps

1495 Jps

尝试[root@CentOS ~]# yum install lrzsz -y组件,如果用户将JAVA_HOME配置在系统变量中/etc/profile需要在安装hadoop时候额外配置etc/hadoop/hadoop-env.sh ,因此推荐配置在用户变量中.

配置主机名和IP映射关系/etc/hosts

[root@CentOS ~]# vi /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.169.139 CentOS

在分布式系统中很多服务都是以主机名标示节点,因此配置IP和主机名的映射关系.用户可以查看以下文件

[root@CentOS ~]# cat /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=CentOS

关闭防火墙服务

[root@CentOS ~]# service iptables stop #关闭服务

iptables: Setting chains to policy ACCEPT: filter [  OK  ]

iptables: Flushing firewall rules: [  OK  ]

iptables: Unloading modules: [  OK  ]

[root@CentOS ~]# chkconfig iptables off #关闭开机自起

[root@CentOS ~]# chkconfig --list | grep iptables

iptables        0:off   1:off   2:off   3:off   4:off   5:off   6:off

因为搭建分布式服务之间可能会产生相互的调度,为了保证正常的通信,一般需要关闭防火墙.

配置主机SSH免密码认证(秘钥)

SSH 为 Secure Shell 的缩写，SSH 为建立在应用层基础上的安全协议，专为远程登录会话和其他网络服务提供安全性的协议。

基于口令的安全验证:基于口令用户名/密码

基于密匙的安全验证:

需要依靠密匙，也就是你必须为自己创建一对密匙，并把公用密匙放在需要访问的服务器上。如果你要连接到SSH服务器上，客户端软件就会向服务器发出请求，请求用你的密匙进行安全验证。服务器收到请求之后，先在该服务器上你的主目录下寻找你的公用密匙，然后把它和你发送过来的公用密匙进行比较。如果两个密匙一致，服务器就用公用密匙加密“质询”（challenge）并把它发送给客户端软件。客户端软件收到“质询”之后就可以用你的私人密匙解密再把它发送给服务器。

[root@CentOS ~]# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Created directory '/root/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

c3:b7:c4:e3:5e:6f:db:69:48:23:1e:f7:81:9b:d1:8e root@CentOS

The key's randomart image is:

+--[ RSA 2048]----+

|                 |

|                 |

|                 |

|       . .       |

|        S =   o  |

|         = = * o |

|          + * X .|

|         . o E.=.|

|          .  .+o.|

+-----------------+

[root@CentOS ~]# ssh-copy-id CentOS

The authenticity of host 'centos (192.168.169.139)' can't be established.

RSA key fingerprint is f0:63:ed:d6:21:3b:b5:47:ad:e2:7f:98:bd:8f:54:94.

Are you sure you want to continue connecting (yes/no)? `yes`

Warning: Permanently added 'centos,192.168.169.139' (RSA) to the list of known hosts.

root@centos's password:`****`

Now try logging into the machine, with "ssh 'CentOS'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

2.HADOOP HDFS安装与配置

参考:http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html

解压并且配置HADOOP_HOME

[root@CentOS ~]# tar -zxf hadoop-2.6.0_x64.tar.gz -C /usr/

HADOOP_HOME=/usr/hadoop-2.6.0

JAVA_HOME=/usr/java/latest

PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

CLASSPATH=.

export JAVA_HOME

export PATH

export CLASSPATH

export HADOOP_HOME

[root@CentOS ~]# source /root/.bashrc

[root@CentOS ~]# echo $HADOOP_HOME

/usr/hadoop-2.6.0

HADOOP_HOME环境变量被第三方产品所依赖例如:hbase/hive/flume/Spark在集成Hadoop的时候,是通过读取> HADOOP_HOME环境变量确定HADOOP位置.

配置etc/hadoop/core-site.xml*

<property>

    <name>fs.defaultFS</name>

    <value>hdfs://CentOS:9000</value>

</property>

<property>

        <name>hadoop.tmp.dir</name>

        <value>/usr/hadoop-2.6.0/hadoop-${user.name}</value>

</property>

配置etc/hadoop/hdfs-site.xml

<property>

    <name>dfs.replication</name>

    <value>1</value>

</property>

配置etc/hadoop/slaves

CentOS

启动HDFS

1.如果是第一次启动HDFS,需要格式化namenode

[root@CentOS ~]# hdfs namenode -format

...

19/01/02 20:19:37 INFO common.Storage: Storage directory /usr/hadoop-2.6.0/hadoop-root/dfs/name has been successfully formatted.

...

格式化成功后,用户可以看到以下目录结构

[root@CentOS ~]# yum install -y tree

[root@CentOS ~]# tree /usr/hadoop-2.6.0/hadoop-root/

/usr/hadoop-2.6.0/hadoop-root/

└── dfs

    └── name

        └── current

            ├── fsimage_0000000000000000000

            ├── fsimage_0000000000000000000.md5

            ├── seen_txid

            └── VERSION

3..启动HDFS服务

[root@CentOS ~]# start-dfs.sh

Starting namenodes on [CentOS]

CentOS: namenode running as process 1846. Stop it first.

CentOS: starting datanode, logging to /usr/hadoop-2.6.0/logs/hadoop-root-datanode-CentOS.out

Starting secondary namenodes [0.0.0.0]

The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.

RSA key fingerprint is f0:63:ed:d6:21:3b:b5:47:ad:e2:7f:98:bd:8f:54:94.

Are you sure you want to continue connecting (yes/no)? yes

0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.

0.0.0.0: starting secondarynamenode, logging to /usr/hadoop-2.6.0/logs/hadoop-root-secondarynamenode-CentOS.out

[root@CentOS ~]# jps

2370 Jps

2133 DataNode

1846 NameNode

2267 SecondaryNameNode

[root@CentOS ~]# stop-dfs.sh

Stopping namenodes on [CentOS]

CentOS: stopping namenode

CentOS: stopping datanode

Stopping secondary namenodes [0.0.0.0]

0.0.0.0: stopping secondarynamenode

或者用户可以访问浏览器:http://192.168.197.140:50070

[root@CentOS ~]# hdfs dfs -put /root/jdk-8u171-linux-x64.rpm  /

[root@CentOS ~]# hdfs dfs -ls  /

Found 1 items

-rw-r--r--   1 root supergroup  175262413 2019-01-02 20:29 /jdk-8u171-linux-x64.rpm

4.开启 HDFS 的回收站

etc/hadoop/core-site.xml

<property>

        <name>fs.trash.interval</name>

        <value>1</value>

</property>

设置1分钟延迟,1分钟以后被删除文件会被系统彻底删除.防止用户误操作

[root@CentOS ~]# hdfs dfs -rm -r -f /bb.log

19/01/03 12:27:08 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 1 minutes, Emptier interval = 0 minutes.

Moved: 'hdfs://CentOS:9000/bb.log' to trash at: hdfs://CentOS:9000/user/root/.Trash/Current

[root@CentOS ~]# hdfs dfs -rm -r -f -skipTrash /aa.log

Deleted /aa.log