hadoop 运行测试

hadoop集群运行

需要提前配置配置文件

slave节点用户得是hadoop，/usr/local/src的所有文件得属于hadoop

三台虚拟机关闭setenforce与防火墙，并且配置域名解析

格式化NameNode

[hadoop@master src]$ cd /usr/local/src/hadoop/
[hadoop@master hadoop]$ ./bin/hdfs namenode -format
22/04/02 20:49:35 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/192.168.3.23
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.1

启动NameNode

[hadoop@master hadoop]$ hadoop-daemon.sh start namenode
starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.out

查看java进程

[hadoop@master hadoop]$ jps
10356 NameNode
10427 Jps
5518 SecondaryNameNode
[hadoop@master hadoop]$

slave1启动DataNode

[hadoop@slave1 src]$ hadoop-daemon.sh start datanode
starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.out
[hadoop@slave1 src]$ jps
10908 Jps
10511 DataNode

slave2启动DataNode

[hadoop@slave2 hadoop]$ hadoop-daemon.sh start datanode
starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.out
[hadoop@slave2 hadoop]$ jps
9763 DataNode
9828 Jps

在windows上做域名解析

将c:\windows\system32\drivers\etc\hosts 拖出桌面

修改，在放回去

启动SecondaryNameNode

[hadoop@master hadoop]$ hadoop-daemon.sh stop secondarynamenode
stopping secondarynamenode
[hadoop@master hadoop]$ hadoop-daemon.sh start secondarynamenode
starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.out
[hadoop@master hadoop]$ jps
13938 NameNode
12980 DataNode
14233 Jps
14190 SecondaryNameNode

master传递公钥（在hadoop用户下）

ssh-keygen

ssh-copy-id slave1

ssh-copy-id slave2

ssh-copy-id master

启动分布式操作系统和资源管理

[hadoop@master .ssh]$ start-dfs.sh
Starting namenodes on [master]
master: namenode running as process 4729. Stop it first.
192.168.3.128: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.out
192.168.3.129: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 4843. Stop it first.
[hadoop@master .ssh]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.out
192.168.3.129: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.out
192.168.3.128: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.out
[hadoop@master .ssh]$ jps
5799 Jps
4729 NameNode
5530 ResourceManager
4843 SecondaryNameNode
[hadoop@master .ssh]$ 

[hadoop@slave1 network-scripts]$ jps
4049 NodeManager
4217 Jps

slave节点出现NodeManager 主节点出现ResourceManager则成功

hadoop创建文件

[hadoop@slave1 network-scripts]$ hdfs dfs -mkdir /input
[hadoop@slave1 network-scripts]$ hdfs  dfs -ls /
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2022-04-03 11:53 /input
[hadoop@slave1 network-scripts]$

创建文件并上传

[hadoop@master network-scripts]$ mkdir ~/input2
[hadoop@master network-scripts]$ vi ~/input2/data.txt
[hadoop@master network-scripts]$ cat ~/input2/data.txt
Hello   World
Hello   Supermao
Hello   huawei
[hadoop@master ~]$ hdfs dfs -put ~/input/data.txt  /input
[hadoop@master ~]$ hdfs dfs -mkdir /mqy
[hadoop@master ~]$ hdfs dfs -put input/data.txt  /mqy
[hadoop@master ~]$ hdfs dfs -ls /mqy
Found 1 items
-rw-r--r--   2 hadoop supergroup         40 2022-04-03 14:12 /mqy/data.txt
[hadoop@master ~]$ hdfs dfs -cat /mqy/data.txt
Hello World
Hello redhat
Hello supermao

本地主机http://master:50070/explorer.html#/

获得上传的文件

停止

报错

[hadoop@master hadoop]$ hdfs dfsadmin -report
Configured Capacity: 0 (0 B)
Present Capacity: 0 (0 B)
DFS Remaining: 0 (0 B)
DFS Used: 0 (0 B)
DFS Used%: NaN%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

无法连接存储节点

删除上一次生成的/usr/local/src/hadoop/dfs/data

再次hadoop-daemon.sh start datanode

启动成功

如果启动了datanode，再次格式化namenode

那么就会因为clusterID不一致导致主控与从属无法连接

停止所有服务

stop-all.sh

删除上一次启动datanode的数据，重新启动datanode

此文件记录了datanode

hdfs-site.xml