Hadoop集群(二) HDFS搭建

HDFS只是Hadoop最基本的一个服务，很多其他服务，都是基于HDFS展开的。所以部署一个HDFS集群，是很核心的一个动作，也是大数据平台的开始。

安装Hadoop集群，首先需要有Zookeeper才可以完成安装。如果没有Zookeeper，请先部署一套Zookeeper。另外，JDK以及物理主机的一些设置等。请参考：

Hadoop集群(一) Zookeeper搭建

Hadoop集群(三) Hbase搭建

Hadoop集群(四) Hadoop升级

下面开始HDFS的安装

HDFS主机分配

192.168.67.101 c6701 --Namenode+datanode

192.168.67.102 c6702 --datanode

192.168.67.103 c6703 --datanode

1. 安装HDFS，解压hadoop-2.6.0-EDH-0u2.tar.gz

我同时下载2.6和2.7版本的软件，先安装2.6，然后在执行2.6到2.7的升级步骤

useradd hdfs

echo "hdfs:hdfs" | chpasswd

su - hdfs

cd /tmp/software

tar -zxvf hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs/

mkdir -p /data/hadoop/temp

mkdir -p /data/hadoop/journal

mkdir -p /data/hadoop/hdfs/name

mkdir -p /data/hadoop/hdfs/data

chown -R hdfs:hdfs /data/hadoop

chown -R hdfs:hdfs /data/hadoop/temp

chown -R hdfs:hdfs /data/hadoop/journal

chown -R hdfs:hdfs /data/hadoop/hdfs/name

chown -R hdfs:hdfs /data/hadoop/hdfs/data

$ pwd/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop

2. 修改core-site.xml对应的参数

$ cat core-site.xml<configuration>

<name>fs.defaultFS</name>

</property>

<name>hadoop.tmp.dir</name>

<value>/data/hadoop/temp</value>

</property>

<name>io.file.buffer.size</name>

</property>

<name>ha.zookeeper.quorum</name>

</property>

</configuration>

3. 修改hdfs-site.xml对应的参数

cat hdfs-site.xml<configuration>

<name>dfs.nameservices</name>

</property>

<name>dfs.ha.namenodes.ns</name>

</property>

<name>dfs.namenode.rpc-address.ns.nn1</name>

</property>

<name>dfs.namenode.http-address.ns.nn1</name>

</property>

<name>dfs.namenode.rpc-address.ns.nn2</name>

</property>

<name>dfs.namenode.http-address.ns.nn2</name>

</property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://c6701:8485;c6702:8485;c6703:8485/ns</value>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/data/hadoop/journal</value>

</property>

<name>dfs.ha.automatic-failover.enabled</name>

</property>

<name>dfs.client.failover.proxy.provider.ns</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

</property>

<name>dfs.namenode.name.dir</name>

<value>/data/hadoop/hdfs/name</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>/data/hadoop/hdfs/data</value>

</property>

<name>dfs.replication</name>

</property>

<name>dfs.webhdfs.enabled</name>

</property> </configuration>

4. 添加slaves文件

$ more slaves

c6701

c6702

c6703

--- 安装C6702的hdfs---

5. 创建c6702的用户，并为hdfs用户ssh免密

ssh c6702 "useradd hdfs"

ssh c6702 "echo "hdfs:hdfs" | chpasswd"

ssh-copy-id hdfs@c6702

6. 拷贝软件

scp -r /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6702:/tmp/software/.

ssh c6702 "chmod 777 /tmp/software/*"

7. 创建目录，解压软件

ssh hdfs@c6702 "mkdir hdfs"

ssh hdfs@c6702 "tar -zxvf /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs"

ssh hdfs@c6702 "ls -al hdfs"

ssh hdfs@c6702 "ls -al hdfs/hadoop*"

复制配置文件

ssh hdfs@c6702 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml"

ssh hdfs@c6702 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml"

scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml

scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml

scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves

创建hdfs需要的目录

ssh root@c6702 "mkdir -p /data/hadoop"

ssh root@c6702 " chown -R hdfs:hdfs /data/hadoop"

ssh hdfs@c6702 "mkdir -p /data/hadoop/temp"

ssh hdfs@c6702 "mkdir -p /data/hadoop/journal"

ssh hdfs@c6702 "mkdir -p /data/hadoop/hdfs/name"

ssh hdfs@c6702 "mkdir -p /data/hadoop/hdfs/data"

--- 安装C6703的hdfs---

8. 创建c6703的用户，并为hdfs用户ssh免密

ssh c6703 "useradd hdfs"

ssh c6703 "echo "hdfs:hdfs" | chpasswd"

ssh-copy-id hdfs@c6703

9. 拷贝软件

scp -r /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6703:/tmp/software/.

ssh c6703 "chmod 777 /tmp/software/*"10. 创建目录，解压软件

ssh hdfs@c6703 "mkdir hdfs"

ssh hdfs@c6703 "tar -zxvf /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs"

ssh hdfs@c6703 "ls -al hdfs"

ssh hdfs@c6703 "ls -al hdfs/hadoop*"

复制配置文件

ssh hdfs@c6703 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml"

ssh hdfs@c6703 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml"

scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml

scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml

scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves

创建hdfs需要的目录

ssh root@c6703 "mkdir -p /data/hadoop"

ssh root@c6703 " chown -R hdfs:hdfs /data/hadoop"

ssh hdfs@c6703 "mkdir -p /data/hadoop/temp"

ssh hdfs@c6703 "mkdir -p /data/hadoop/journal"

ssh hdfs@c6703 "mkdir -p /data/hadoop/hdfs/name"

ssh hdfs@c6703 "mkdir -p /data/hadoop/hdfs/data"

11. 启动HDFS，先启动三个节点的journalnode

/home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start journalnode

检查状态

$ jps3958 Jps3868 JournalNode

12. 然后启动namenode,首次启动namenode之前，先在其中一个节点(主节点)format namenode信息，信息会存在于dfs.namenode.name.dir指定的路径中

<name>dfs.namenode.name.dir</name>

<value>/data/hadoop/hdfs/name</value>

$ ./hdfs namenode -format17/09/26 07:52:17 INFO namenode.NameNode: STARTUP_MSG:/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG: host = c6701.python279.org/192.168.67.101

STARTUP_MSG: args = [-format]

STARTUP_MSG: version = 2.6.0-EDH-0u2

STARTUP_MSG: classpath = /home/hdfs/hadoop-2.6.0-EDHxxxxxxxxxx

STARTUP_MSG: build = http://gitlab-xxxxx

STARTUP_MSG: java = 1.8.0_144

************************************************************/17/09/26 07:52:17 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]17/09/26 07:52:17 INFO namenode.NameNode: createNameNode [-format]17/09/26 07:52:18 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.17/09/26 07:52:18 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.

Formatting using clusterid: CID-b2f01411-862f-44b2-a6dc-7d17bd48d52217/09/26 07:52:18 INFO namenode.FSNamesystem: No KeyProvider found.17/09/26 07:52:18 INFO namenode.FSNamesystem: fsLock is fair:true17/09/26 07:52:18 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=100017/09/26 07:52:18 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true17/09/26 07:52:18 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.00017/09/26 07:52:18 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Sep 26 07:52:1817/09/26 07:52:18 INFO util.GSet: Computing capacity for map BlocksMap17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit17/09/26 07:52:18 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB17/09/26 07:52:18 INFO util.GSet: capacity = 2^21 = 2097152 entries17/09/26 07:52:18 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false17/09/26 07:52:18 INFO blockmanagement.BlockManager: defaultReplication = 217/09/26 07:52:18 INFO blockmanagement.BlockManager: maxReplication = 51217/09/26 07:52:18 INFO blockmanagement.BlockManager: minReplication = 117/09/26 07:52:18 INFO blockmanagement.BlockManager: maxReplicationStreams = 217/09/26 07:52:18 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false17/09/26 07:52:18 INFO blockmanagement.BlockManager: replicationRecheckInterval = 300017/09/26 07:52:18 INFO blockmanagement.BlockManager: encryptDataTransfer = false17/09/26 07:52:18 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 100017/09/26 07:52:18 INFO namenode.FSNamesystem: fsOwner = hdfs (auth:SIMPLE)17/09/26 07:52:18 INFO namenode.FSNamesystem: supergroup = supergroup17/09/26 07:52:18 INFO namenode.FSNamesystem: isPermissionEnabled = true17/09/26 07:52:18 INFO namenode.FSNamesystem: Determined nameservice ID: ns17/09/26 07:52:18 INFO namenode.FSNamesystem: HA Enabled: true17/09/26 07:52:18 INFO namenode.FSNamesystem: Append Enabled: true17/09/26 07:52:18 INFO util.GSet: Computing capacity for map INodeMap17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit17/09/26 07:52:18 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB17/09/26 07:52:18 INFO util.GSet: capacity = 2^20 = 1048576 entries17/09/26 07:52:18 INFO namenode.NameNode: Caching file names occuring more than 10 times17/09/26 07:52:18 INFO util.GSet: Computing capacity for map cachedBlocks17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit17/09/26 07:52:18 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB17/09/26 07:52:18 INFO util.GSet: capacity = 2^18 = 262144 entries17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.999000012874603317/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 017/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 3000017/09/26 07:52:18 INFO namenode.FSNamesystem: Retry cache on namenode is enabled17/09/26 07:52:18 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis17/09/26 07:52:18 INFO util.GSet: Computing capacity for map NameNodeRetryCache17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit17/09/26 07:52:18 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB17/09/26 07:52:18 INFO util.GSet: capacity = 2^15 = 32768 entries17/09/26 07:52:18 INFO namenode.NNConf: ACLs enabled? false17/09/26 07:52:18 INFO namenode.NNConf: XAttrs enabled? true17/09/26 07:52:18 INFO namenode.NNConf: Maximum size of an xattr: 1638417/09/26 07:52:19 INFO namenode.FSImage: Allocated new BlockPoolId: BP-144216011-192.168.67.101-150641233975717/09/26 07:52:19 INFO common.Storage: Storage directory /data/hadoop/hdfs/name has been successfully formatted.17/09/26 07:52:20 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 017/09/26 07:52:20 INFO util.ExitUtil: Exiting with status 017/09/26 07:52:20 INFO namenode.NameNode: SHUTDOWN_MSG:/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at c6701.python279.org/192.168.67.101

************************************************************/

13. standby namenode需要先执行bootstrapstandby，输出如下

[hdfs@c6702 sbin]$ ../bin/hdfs namenode -bootstrapstandby17/09/26 09:44:58 INFO namenode.NameNode: STARTUP_MSG:/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG: host = c6702.python279.org/192.168.67.102

STARTUP_MSG: args = [-bootstrapstandby]

STARTUP_MSG: version = 2.6.0-EDH-0u2

STARTUP_MSG: classpath = /home/hdfs/haxxx

STARTUP_MSG: build = http://gitlab-xxxx

STARTUP_MSG: java = 1.8.0_144

************************************************************/17/09/26 09:44:58 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]17/09/26 09:44:58 INFO namenode.NameNode: createNameNode [-bootstrapstandby]17/09/26 09:44:59 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.17/09/26 09:44:59 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.=====================================================

About to bootstrap Standby ID nn2 from:

Nameservice ID: ns

Other Namenode ID: nn1

Other NN's HTTP address: http://c6701:50070

Other NN's IPC address: c6701/192.168.67.101:9000

Namespace ID: 793662207

Block pool ID: BP-144216011-192.168.67.101-1506412339757

Cluster ID: CID-b2f01411-862f-44b2-a6dc-7d17bd48d522

Layout version: -60=====================================================

Re-format filesystem in Storage Directory /data/hadoop/hdfs/name ? (Y or N) y17/09/26 09:45:16 INFO common.Storage: Storage directory /data/hadoop/hdfs/name has been successfully formatted.17/09/26 09:45:16 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.17/09/26 09:45:16 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.17/09/26 09:45:17 INFO namenode.TransferFsImage: Opening connection to http://c6701:50070/imagetransfer?getimage=1&txid=0&storageInfo=-60:793662207:0:CID-b2f01411-862f-44b2-a6dc-7d17bd48d52217/09/26 09:45:17 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds17/09/26 09:45:17 INFO namenode.TransferFsImage: Transfer took 0.01s at 0.00 KB/s17/09/26 09:45:17 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 351 bytes.17/09/26 09:45:17 INFO util.ExitUtil: Exiting with status 017/09/26 09:45:17 INFO namenode.NameNode: SHUTDOWN_MSG:/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at c6702.python279.org/192.168.67.102

************************************************************/

14. 检查状态，namenode还没有启动

[hdfs@c6702 sbin]$ jps4539 Jps3868 JournalNode

15. 启动standby namenode，命令和master启动的方式相同

[hdfs@c6702 sbin]$ ./hadoop-daemon.sh start namenodestarting namenode, logging to /home/hdfs/hadoop-2.6.0-EDH-0u2/logs/hadoop-hdfs-namenode-c6702.python279.org.out

16. 再次检查，namenode已经启动

[hdfs@c6702 sbin]$ jps4640 Jps4570 NameNode3868 JournalNode

17. 格式化zkfc,让在zookeeper中生成ha节点，在master上执行如下命令，完成格式化

[hdfs@c6701 bin]$ ./hdfs zkfc -formatZK17/09/26 09:59:20 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at c6701/192.168.67.101:900017/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:host.name=c6701.python279.org17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_14417/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/local/jdk1.8.0_144/jre17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/exxxx17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/lib/native17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd6417/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_6417/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hdfs17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfs/hadoop-2.6.0-EDH-0u2/bin17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=c6701:2181,c6702:2181,c6703:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@20deea7f17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Opening socket connection to server c6703.python279.org/192.168.67.103:2181. Will not attempt to authenticate using SASL (unknown error)17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Socket connection established to c6703.python279.org/192.168.67.103:2181, initiating session17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Session establishment complete on server c6703.python279.org/192.168.67.103:2181, sessionid = 0x35ebc5163710000, negotiated timeout = 500017/09/26 09:59:20 INFO ha.ActiveStandbyElector: Session connected.17/09/26 09:59:20 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns in ZK.17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Session: 0x35ebc5163710000 closed17/09/26 09:59:20 INFO zookeeper.ClientCnxn: EventThread shut down

18. 格式化完成的检查

格式成功后，查看zookeeper中可以看到 <<<<<<<<<<<命令没确认

[zk: localhost:2181(CONNECTED) 1] ls /hadoop-ha

19. 启动zkfc，这个就是为namenode使用的

./hadoop-daemon.sh start zkfcstarting zkfc, logging to /home/hdfs/hadoop-2.6.0-EDH-0u2/logs/hadoop-hdfs-zkfc-c6701.python279.org.out

$ jps4272 DataNode4402 JournalNode6339 Jps6277 DFSZKFailoverController4952 NameNode

20. 另一个节点启动zkfc，

ssh hdfs@c6702 /home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start zkfc

$ jps4981 Jps4935 DFSZKFailoverController4570 NameNode3868 JournalNode

21. 注意：进行初始化的时候，必须保证zk集群已经启动了。

1、在ZK中创建znode来存储automatic Failover的数据，任选一个NN执行完成即可：

sh bin/hdfs zkfc -formatZK

2、启动zkfs，在所有的NN节点中执行以下命令：

sh sbin/hadoop-daemon.sh start zkfc

22. 启动datanode

最后启动集群

/home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start zkfc

sh sbin/start-dfs.sh

HDFS安装过程中的重点，最后在软件启动过程中，一些初始化操作，很重要。

1. 启动全部的journalnode

2. 在namenode1上执行， hdfs namenode -format

3. 在namenode1上执行，启动namenode1，命令hadoop-daemon.sh start namenode

4. 在namenode2上执行, hdfs namenode -bootstrapstandby

5. 在namenode1上执行，格式化zkfc,在zookeeper中生成HA节点, hdfs zkfc -formatZK

6. 启动zkfc，hadoop-daemon.sh start zkfc。有namenode运行的节点，都要启动ZKFC

7. 启动 datanode

HDFS只是Hadoop最基本的一个模块，这里已经安装完成，可以为后面的Hbase提供服务了。

Hadoop集群(二) HDFS搭建的更多相关文章

Hadoop集群(三) Hbase搭建
前面已经完成Zookeeper和HDFS的安装,本文会详细介绍Hbase的安装步骤.以及安装过程中遇到问题的汇总. 系列文章: Hadoop集群(一) Zookeeper搭建 Hadoop集群(二 ...
大数据系列（1）——Hadoop集群坏境搭建配置
前言关于时下最热的技术潮流,无疑大数据是首当其中最热的一个技术点,关于大数据的概念和方法论铺天盖地的到处宣扬,但其实很多公司或者技术人员也不能详细的讲解其真正的含义或者就没找到能被落地实施的可行性方 ...
hadoop集群环境的搭建
hadoop集群环境的搭建今天终于把hadoop集群环境给搭建起来了,能够运行单词统计的示例程序了. 集群信息如下: 主机名 Hadoop角色 Hadoop jps命令结果 Hadoop用户 Had ...
搭建Hadoop集群 (二)
前面的步骤请看搭建Hadoop集群 (一) 安装Hadoop 解压安装登录master, 下载解压hadoop 2.6.2压缩包到/home/hm/文件夹. (也可以从主机拖拽或者psftp压缩 ...
hadoop集群之HDFS和YARN启动和停止命令
假如我们只有3台linux虚拟机,主机名分别为hadoop01.hadoop02和hadoop03,在这3台机器上,hadoop集群的部署情况如下: hadoop01:1个namenode,1个dat ...
格式化hdfs后，hadoop集群启动hdfs，namenode启动成功，datanode未启动
集群格式化hdfs后,在主节点运行启动hdfs后,发现namenode启动了,而datanode没有启动,在其他节点上jps后没有datanode进程!原因: 当我们使用hdfs namenode - ...
hadoop集群完全分布式搭建
Hadoop环境搭建:完全分布式集群规划: ip hostname 192.168.204.154 master namenode resour ...
Hadoop集群(一) Zookeeper搭建
作为Hadoop初学者,自然要从安装入手.而hadoop的优势就是分布式,所以,也一定要安装分布式的系统. 整体安装步骤,包括Zookeeper+HDFS+Hbase,为了文章简洁,我会分三篇blog ...
hadoop集群为分布式搭建
1.准备Linux环境设置虚拟机网络 1.0点击VMware快捷方式,右键打开文件所在位置 -> 双击vmnetcfg.exe -> VMnet1 host-only ->修改 ...

随机推荐

day20re模块正则表达式
re模块,正则1 re是什么正则就是用一些具有特殊含义的符号组合到一起(称为正则表达式)来描述字符或者字符串的方法. 就是用来描述一类事物的规则.就是一些带有特殊含义的符号或者符号的组合. 作 ...
Centos下添加用户并赋权
创建新用户创建一个用户名为:linuxidc [root@localhost ~]# adduser linuxidc 为这个用户初始化密码,linux会判断密码复杂度,不过可以强行忽略: [roo ...
Javascript学习笔记5 - 滑动Slides
开始之前:http://docs.jquery.com/ 是jQuery文档的网站, https://jsfiddle.net/是js的在线验证工具在html中,有这几个标签: javascript ...
mysql 替代Oracle instr
在迁移项目时遇到的,原项目的数据库使用的Oracle,现在要迁移到MySQL中,而项目中用到了Oracle的instr函数,而MySQL只能查找子串是否在父串中,没法按照出现的次数进行查找. 先来介绍 ...
用javascript控制svg的图片颜色
SVG:(1.svg) <?xml version="1.0" standalone="no"?> <!DOCTYPE svg PUBLIC ...
if、else、elif
if后面+判断条件 elif后面+判断条件 else 否则进行某些操作举个列子 1 a = raw_input(">>>") 2 if a == "h ...
混合式应用真机调试（Android Studio + Chrome）
如何在chrome上调试混合式APP https://developers.google.com/web/tools/chrome-devtools/remote-debugging/ Get Sta ...
Flex NumberFormatter 数值的格式化
<?xml version="1.0" encoding="utf-8"?> <s:Application xmlns:fx="ht ...
scrapy执行流程
def _next_request_from_scheduler(self, spider): fetch一个request,调用http11 handler,client的agent,返回的是def ...
API / DOM
浏览器特性当控制台报错时,IE9会停止执行JS.当打开控制台时会执行后续JS ------------------------------------------------------------ ...

Hadoop集群(二) HDFS搭建

Hadoop集群(二) HDFS搭建的更多相关文章

随机推荐

热门专题