大数据技术之Hadoop3.1.2版本伪分布式部署

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

一.主机环境准备

1>.操作系统环境

[root@node101.yinzhengjie.org.cn ~]# cat /etc/redhat-release

CentOS Linux release 7.6. (Core)

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# uname -r

3.10.-.el7.x86_64

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# uname -m

x86_64

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# df -h

Filesystem      Size  Used Avail Use% Mounted on

/dev/sda3        48G  .1G   46G   % /

devtmpfs        .9G       .9G   % /dev

tmpfs           .9G       .9G   % /dev/shm

tmpfs           .9G   12M  .9G   % /run

tmpfs           .9G       .9G   % /sys/fs/cgroup

/dev/sda1       297M  120M  177M  % /boot

tmpfs           378M       378M   % /run/user/

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# free -h

              total        used        free      shared  buff/cache   available

Mem:           .7G         99M        .0G         11M        565M        .3G

Swap:          .0G          0B        .0G

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]#

2>.关闭防火墙

[root@node101.yinzhengjie.org.cn ~]# systemctl stop firewalld

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon

   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)

   Active: inactive (dead)

     Docs: man:firewalld()

Apr  :: node101.yinzhengjie.org.cn systemd[]: Starting firewalld - dynamic firewall daemon...

Apr  :: node101.yinzhengjie.org.cn systemd[]: Started firewalld - dynamic firewall daemon.

Apr  :: node101.yinzhengjie.org.cn systemd[]: Stopping firewalld - dynamic firewall daemon...

Apr  :: node101.yinzhengjie.org.cn systemd[]: Stopped firewalld - dynamic firewall daemon.

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# systemctl stop firewalld

[root@node101.yinzhengjie.org.cn ~]# systemctl disable firewalld

Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# systemctl disable firewalld

[root@node101.yinzhengjie.org.cn ~]# sed -i s'#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# grep SELINUX= /etc/selinux/config  | grep -v ^#

SELINUX=disabled

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# sed -i s'#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config

[root@node101.yinzhengjie.org.cn ~]# getenforce

Disabled

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# getenforce

3>.配置本机的免密登录（如果不做这步的话，在启动hdfs集群时会报权限被拒绝的错误）

[root@node101.yinzhengjie.org.cn ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

Generating public/private rsa key pair.

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:zViCNdoMvFre2Sury7Sxgy1ciwZx95u0KjBlOSCA5Ek root@node101.yinzhengjie.org.cn

The key's randomart image is:

+---[RSA ]----+

|=E   .. o        |

|= o   .B .       |

| + . .o.+ .      |

|  . * +  *       |

|   = * oSoo      |

|  + . o = .      |

|   = =oo + .     |

|    Bo++= .      |

|   . oB=.o       |

+----[SHA256]-----+

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

[root@node101.yinzhengjie.org.cn ~]# ssh-copy-id  root@node101.yinzhengjie.org.cn

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO:  key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@node101.yinzhengjie.org.cn's password: 

Number of key(s) added: 

Now try logging into the machine, with:   "ssh 'root@node101.yinzhengjie.org.cn'"

and check to make sure that only the key(s) you wanted were added.

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# ssh-copy-id root@node101.yinzhengjie.org.cn

4>.安装jdk

　　大家可以自行去Oracle官网下载相应的jdk，我这里给出下载地址：https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html（推荐下载tar.gz的安装包，解压后配置环境变量即可使用）。

[root@node101.yinzhengjie.org.cn ~]# tail - /etc/profile

#ADD JAVA_PATH

JAVA_HOME=/yinzhengjie/softwares/jdk

PATH=$PATH:$JAVA_HOME/bin

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# which java

/yinzhengjie/softwares/jdk/bin/java

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# java -version

java version "1.8.0_201"

Java(TM) SE Runtime Environment (build 1.8.0_201-b09)

Java HotSpot(TM) -Bit Server VM (build 25.201-b09, mixed mode)

[root@node101.yinzhengjie.org.cn ~]#

5>.下载Hadoop3.1.2版本

[root@node101.yinzhengjie.org.cn ~]# wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz

---- ::--  http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz

Resolving mirror.bit.edu.cn (mirror.bit.edu.cn)... 114.247.56.117, 114.247.56.117

Connecting to mirror.bit.edu.cn (mirror.bit.edu.cn)|114.247.56.117|:... connected.

HTTP request sent, awaiting response...  OK

Length:  (317M) [application/octet-stream]

Saving to: ‘hadoop-3.1..tar.gz’

%[============================================================================================================================================>] ,, .84MB/s   in 49s    

-- :: (6.52 MB/s) - ‘hadoop-3.1..tar.gz’ saved [/]

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz

二.编辑hadoop 3.1.2相关配置文件

1>.解压配置文件到指定目录并配置环境变量

[root@node101.yinzhengjie.org.cn ~]# tar -zxf hadoop-3.1..tar.gz -C /yinzhengjie/softwares/

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/hadoop-3.1./

total

drwxr-xr-x        Jan  : bin

drwxr-xr-x         Jan  : etc

drwxr-xr-x        Jan  : include

drwxr-xr-x         Jan  : lib

drwxr-xr-x        Jan  : libexec

-rw-rw-r--     Jan  : LICENSE.txt

-rw-rw-r--      Jan  : NOTICE.txt

-rw-rw-r--       Jan  : README.txt

drwxr-xr-x       Jan  : sbin

drwxr-xr-x         Jan  : share

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# tar -zxf hadoop-3.1.2.tar.gz -C /yinzhengjie/softwares/

[root@node101.yinzhengjie.org.cn ~]# mkdir -pv /data/hadoop/{hdfs,yarn,mapreduce}

mkdir: created directory ‘/data’

mkdir: created directory ‘/data/hadoop’

mkdir: created directory ‘/data/hadoop/hdfs’

mkdir: created directory ‘/data/hadoop/yarn’

mkdir: created directory ‘/data/hadoop/mapreduce’

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# mkdir -pv /data/hadoop/{hdfs,yarn,mapreduce}

[root@node101.yinzhengjie.org.cn ~]# tail - /etc/profile

#Add HADOOP_HOME PATH

HADOOP_HOME=/yinzhengjie/softwares/hadoop-3.1.

PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# tail -3 /etc/profile

2>.编辑hadoop相关配置文件

[root@node101.yinzhengjie.org.cn ~]# vi /yinzhengjie/softwares/hadoop-3.1./etc/hadoop/hadoop-env.sh

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# tail - /yinzhengjie/softwares/hadoop-3.1./etc/hadoop/hadoop-env.sh

#Add Hadoop Path by yinzhengjie

export JAVA_HOME=/yinzhengjie/softwares/jdk

export HDFS_NAMENODE_USER=root

export HDFS_DATANODE_USER=root

export HDFS_SECONDARYNAMENODE_USER=root

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# vi /yinzhengjie/softwares/hadoop-3.1.2/etc/hadoop/hadoop-env.sh

[root@node101.yinzhengjie.org.cn ~]# vi /yinzhengjie/softwares/hadoop-3.1./etc/hadoop/core-site.xml

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-3.1./etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

        <property>

                <name>fs.defaultFS</name>

                <value>hdfs://node101.yinzhengjie.org.cn:9820</value>

        </property>

    <property>

                <name>hadoop.tmp.dir</name>

                <value>/data/hadoop/hdfs</value>

        </property>

</configuration>

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# vi /yinzhengjie/softwares/hadoop-3.1.2/etc/hadoop/core-site.xml

[root@node101.yinzhengjie.org.cn ~]# vi /yinzhengjie/softwares/hadoop-3.1./etc/hadoop/hdfs-site.xml

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-3.1./etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

        <property>

                <name>dfs.replication</name>

                <value></value>

        </property>

        <property>

                <name>dfs.namenode.secondary.http-address</name>

                <value>node101.yinzhengjie.org.cn:</value>

        </property>

</configuration>

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# vi /yinzhengjie/softwares/hadoop-3.1.2/etc/hadoop/hdfs-site.xml

[root@node101.yinzhengjie.org.cn ~]# vi /yinzhengjie/softwares/hadoop-3.1./etc/hadoop/workers

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-3.1./etc/hadoop/workers

node101.yinzhengjie.org.cn

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# vi /yinzhengjie/softwares/hadoop-3.1.2/etc/hadoop/workers　　　　　　　　　　#该配置文件和hadoop2.x版本中的slaves功能一样！只不过在hadoop3.x给它做了个更名！

3>.格式化HDFS文件系统（目的是生成初始的元数据信息，生成的信息会存放在我们hdfs-core-site.xml配置文件中指定的工作目录下！温馨提示，在搭建集群的时执行一次就ok啦！千万别每次启动就执行！）

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# ll /data/hadoop/hdfs/

total

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# hdfs namenode -format

WARNING: /yinzhengjie/softwares/hadoop-3.1./logs does not exist. Creating.

-- ::, INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = node101.yinzhengjie.org.cn/172.30.1.101

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 3.1.2

STARTUP_MSG:   classpath = /yinzhengjie/softwares/hadoop-3.1.2/etc/hadoop:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/accessors-smart-1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/asm-5.0.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-beanutils-1.9.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-codec-1.11.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-compress-1.18.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-io-2.5.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-lang3-3.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/commons-net-3.6.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/curator-client-2.13.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/curator-framework-2.13.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/curator-recipes-2.13.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/gson-2.2.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/hadoop-annotations-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/hadoop-auth-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/httpclient-4.5.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/httpcore-4.4.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jackson-annotations-2.7.8.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jackson-core-2.7.8.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jackson-databind-2.7.8.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jersey-core-1.19.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jersey-json-1.19.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jersey-server-1.19.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jersey-servlet-1.19.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jettison-1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jetty-http-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jetty-io-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jetty-security-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jetty-server-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jetty-servlet-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jetty-util-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jetty-webapp-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jetty-xml-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jsch-0.1.54.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/json-smart-2.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerb-client-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerb-common-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerb-core-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerb-server-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerb-util-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerby-config-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerby-util-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/netty-3.10.5.Final.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/re2j-1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/stax2-api-3.1.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/token-provider-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/zookeeper-3.4.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/lib/metrics-core-3.2.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2-tests.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/hadoop-nfs-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/common/hadoop-kms-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-util-ajax-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/netty-all-4.0.52.Final.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/okio-1.6.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/hadoop-auth-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/httpclient-4.5.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/httpcore-4.4.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/nimbus-jose-jwt-4.41.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/json-smart-2.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/asm-5.0.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/netty-3.10.5.Final.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/curator-framework-2.13.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/curator-client-2.13.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-io-2.5.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-server-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-http-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-util-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-io-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-webapp-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-xml-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-servlet-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-security-9.3.24.v20180605.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/hadoop-annotations-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-net-3.6.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jettison-1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-beanutils-1.9.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-lang3-3.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/avro-1.7.7.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/paranamer-2.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-compress-1.18.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/re2j-1.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/gson-2.2.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jsch-0.1.54.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/curator-recipes-2.13.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-3.1.2-tests.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-nfs-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-client-3.1.2-tests.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-client-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.2-tests.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.2-tests.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/dnsjava-2.1.7.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/fst-2.50.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/guice-4.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/jackson-jaxrs-base-2.7.8.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.7.8.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.7.8.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/java-util-1.9.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/javax.inject-1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/jersey-client-1.19.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/json-io-2.5.1.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/objenesis-1.0.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-api-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-client-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-common-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-registry-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-common-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-router-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-services-api-3.1.2.jar:/yinzhengjie/softwares/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-services-core-3.1.2.jar

STARTUP_MSG:   build = https://github.com/apache/hadoop.git -r 1019dde65bcf12e05ef48ac71e84550d589e5d9a; compiled by 'sunilg' on 2019-01-29T01:39Z

STARTUP_MSG:   java = 1.8.0_201

************************************************************/

-- ::, INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]

-- ::, INFO namenode.NameNode: createNameNode [-format]

Formatting using clusterid: CID-f46b1d5c-b489-44e2-b8e0-7e02f938965a

-- ::, INFO namenode.FSEditLog: Edit logging is async:true

-- ::, INFO namenode.FSNamesystem: KeyProvider: null

-- ::, INFO namenode.FSNamesystem: fsLock is fair: true

-- ::, INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false

-- ::, INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)

-- ::, INFO namenode.FSNamesystem: supergroup          = supergroup

-- ::, INFO namenode.FSNamesystem: isPermissionEnabled = true

-- ::, INFO namenode.FSNamesystem: HA Enabled: false

-- ::, INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to . Disabling file IO profiling

-- ::, INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=, counted=, effected=

-- ::, INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true

-- ::, INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to :::00.000

-- ::, INFO blockmanagement.BlockManager: The block deletion will start around  Apr  ::

-- ::, INFO util.GSet: Computing capacity for map BlocksMap

-- ::, INFO util.GSet: VM type       = -bit

-- ::, INFO util.GSet: 2.0% max memory 839.5 MB = 16.8 MB

-- ::, INFO util.GSet: capacity      = ^ =  entries

-- ::, INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false

-- ::, INFO Configuration.deprecation: No unit for dfs.namenode.safemode.extension() assuming MILLISECONDS

-- ::, INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033

-- ::, INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes =

-- ::, INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension =

-- ::, INFO blockmanagement.BlockManager: defaultReplication         =

-- ::, INFO blockmanagement.BlockManager: maxReplication             =

-- ::, INFO blockmanagement.BlockManager: minReplication             =

-- ::, INFO blockmanagement.BlockManager: maxReplicationStreams      =

-- ::, INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms

-- ::, INFO blockmanagement.BlockManager: encryptDataTransfer        = false

-- ::, INFO blockmanagement.BlockManager: maxNumBlocksToLog          =

-- ::, INFO namenode.FSDirectory: GLOBAL serial map: bits= maxEntries=

-- ::, INFO util.GSet: Computing capacity for map INodeMap

-- ::, INFO util.GSet: VM type       = -bit

-- ::, INFO util.GSet: 1.0% max memory 839.5 MB = 8.4 MB

-- ::, INFO util.GSet: capacity      = ^ =  entries

-- ::, INFO namenode.FSDirectory: ACLs enabled? false

-- ::, INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true

-- ::, INFO namenode.FSDirectory: XAttrs enabled? true

-- ::, INFO namenode.NameNode: Caching file names occurring more than  times

-- ::, INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit:

-- ::, INFO snapshot.SnapshotManager: SkipList is disabled

-- ::, INFO util.GSet: Computing capacity for map cachedBlocks

-- ::, INFO util.GSet: VM type       = -bit

-- ::, INFO util.GSet: 0.25% max memory 839.5 MB = 2.1 MB

-- ::, INFO util.GSet: capacity      = ^ =  entries

-- ::, INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets =

-- ::, INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users =

-- ::, INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = ,,

-- ::, INFO namenode.FSNamesystem: Retry cache on namenode is enabled

-- ::, INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is  millis

-- ::, INFO util.GSet: Computing capacity for map NameNodeRetryCache

-- ::, INFO util.GSet: VM type       = -bit

-- ::, INFO util.GSet: 0.029999999329447746% max memory 839.5 MB = 257.9 KB

-- ::, INFO util.GSet: capacity      = ^ =  entries

-- ::, INFO namenode.FSImage: Allocated new BlockPoolId: BP--172.30.1.101-

-- ::, INFO common.Storage: Storage directory /data/hadoop/hdfs/dfs/name has been successfully formatted.

-- ::, INFO namenode.FSImageFormatProtobuf: Saving image file /data/hadoop/hdfs/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression

-- ::, INFO namenode.FSImageFormatProtobuf: Image file /data/hadoop/hdfs/dfs/name/current/fsimage.ckpt_0000000000000000000 of size  bytes saved in  seconds .

-- ::, INFO namenode.NNStorageRetentionManager: Going to retain  images with txid >=

-- ::, INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at node101.yinzhengjie.org.cn/172.30.1.101

************************************************************/

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# ll /data/hadoop/hdfs/

total

drwxr-xr-x  root root  Apr  : dfs

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# ll /data/hadoop/hdfs/dfs/

total

drwxr-xr-x  root root  Apr  : name

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# ll /data/hadoop/hdfs/dfs/name/current/

total

-rw-r--r--  root root  Apr  : fsimage_0000000000000000000

-rw-r--r--  root root   Apr  : fsimage_0000000000000000000.md5

-rw-r--r--  root root    Apr  : seen_txid

-rw-r--r--  root root  Apr  : VERSION

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# hdfs namenode -format

4>.启动hadoop集群

[root@node101.yinzhengjie.org.cn ~]# start-dfs.sh

Starting namenodes on [node101.yinzhengjie.org.cn]

Last login: Thu Apr  :: PDT  on pts/

Starting datanodes

Last login: Thu Apr  :: PDT  on pts/

Starting secondary namenodes [node101.yinzhengjie.org.cn]

Last login: Thu Apr  :: PDT  on pts/

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# start-dfs.sh

[root@node101.yinzhengjie.org.cn ~]# jps

 Jps

 SecondaryNameNode

 NameNode

 DataNode

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# jps

[root@node101.yinzhengjie.org.cn ~]# ss -ntl

State      Recv-Q Send-Q                                              Local Address:Port                                                             Peer Address:Port

LISTEN                                                                       *:                                                                        *:*

LISTEN                                                                        *:                                                                        *:*

LISTEN                                                                        *:                                                                        *:*

LISTEN                                                             172.30.1.101:                                                                        *:*

LISTEN                                                                        *:                                                                        *:*

LISTEN                                                                127.0.0.1:                                                                       *:*

LISTEN                                                                        *:                                                                          *:*

LISTEN                                                             172.30.1.101:                                                                        *:*

LISTEN                                                                       :::                                                                         :::*

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# ss -ntl

5>.访问NameNode的Web UI信息

　　注意访问地址的端口：http://node101.yinzhengjie.org.cn:9870/dfshealth.html#tab-datanode。

三.上传文件到测试伪分布式服务器

[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -mkdir /yinzhengjie

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# ll

total

-rw-r--r--  root root  Feb   : hadoop-3.1..tar.gz

-rw-r--r--  root root       Mar  : jce_policy-.zip

-rw-r--r--  root root  Mar  : jdk-8u201-linux-x64.tar.gz

drwxrwxr-x  root root         Mar  : UnlimitedJCEPolicyJDK8

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -put hadoop-3.1..tar.gz  /yinzhengjie

[root@node101.yinzhengjie.org.cn ~]#

　　更多hdfs相关命令请参考：HDFS集群中大数据运维常用的命令总结。

大数据技术之Hadoop3.1.2版本伪分布式部署的更多相关文章

  1. 大数据技术之Hadoop3.1.2版本完全分布式部署

    大数据技术之Hadoop3.1.2版本完全分布式部署 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.主机环境准备 1>.操作系统环境 [root@node101.yinz ...

  2. 大数据技术之Hadoop3.1.2版本HA模式

    大数据技术之Hadoop3.1.2版本HA模式 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.Hadoop的HA特点 1>.主备NameNode 2>.解决单点故障 ...

  3. 开源大数据技术专场（上午）:Spark、HBase、JStorm应用与实践

    16日上午9点,2016云栖大会“开源大数据技术专场” (全天)在阿里云技术专家封神的主持下开启.通过封神了解到,在上午的专场中,阿里云高级技术专家无谓.阿里云技术专家封神.阿里巴巴中间件技术部高级技 ...

  4. 【学习笔记】大数据技术原理与应用（MOOC视频、厦门大学林子雨）

    1 大数据概述 大数据特性:4v volume velocity variety value 即大量化.快速化.多样化.价值密度低 数据量大:大数据摩尔定律 快速化:从数据的生成到消耗,时间窗口小,可 ...

  5. 大数据技术 - 为什么是SQL

    在大数据处理以及分析中 SQL 的普及率非常高,几乎是每一个大数据工程师必须掌握的语言,甚至非数据处理岗位的人也在学习使用 SQL.今天这篇文章就聊聊 SQL 在数据分析中作用以及掌握 SQL 的必要 ...

  6. 大数据技术之kettle

    大数据技术之kettle 第1章            kettle概述 1.1    什么是kettle kettle是一款开源的ETL工具,纯java编写,可以在Windows.Linux.Uni ...

  7. MaxCompute 最新特性介绍 | 2019大数据技术公开课第三季

    摘要:距离上一次MaxCompute新功能的线上发布已经过去了大约一个季度的时间,而在这一段时间里,MaxCompute不断地在增加新的功能和特性,比如参数化视图.UDF支持动态参数.支持分区裁剪.生 ...

  8. 大数据技术之HBase

    第1章 HBase简介 1.1 什么是HBase HBase的原型是Google的BigTable论文,受到了该论文思想的启发,目前作为Hadoop的子项目来开发维护,用于支持结构化的数据存储. 官方 ...

  9. 大数据技术之Hadoop入门

      第1章 大数据概论 1.1 大数据概念 大数据概念如图2-1 所示. 图2-1 大数据概念 1.2 大数据特点(4V) 大数据特点如图2-2,2-3,2-4,2-5所示 图2-2 大数据特点之大量 ...

随机推荐

  1. 认领该应用 apk空白包签名 方法

    起因: apicloud开发的项目,上架应用宝市场,被要求做这个 解决方法: 下载签名工具,并解压缩.解压缩.解压后是两个文件夹,选择keystore签名工具. 下载地址: linux 签名工具(命令 ...

  2. [LeetCode] 582. Kill Process 终止进程

    Given n processes, each process has a unique PID (process id) and its PPID (parent process id). Each ...

  3. Linux下用dd命令测试硬盘的读写速度

    一.测试写速度:   time dd if=/dev/zero of=/tmp/test bs=8k count=1000000 测试结果:565 MB/s 二.测试读速度:   time dd if ...

  4. （生鲜项目）01. Vue环境搭建

    第一步: nodejs安装 https://nodejs.org/en/download/ 说明安装成功 第二步: cnpm 由于npm需要很多的依赖包,这些包下载都很慢,所以就有了cnpm : ht ...

  5. 阿里云使用Docker部署工单系统（redmine）

    环境:阿里云服务器 Redmine安装部署 Redmine是用Ruby开发的基于web的项目管理软件,是用ROR框架开发的一套跨平台项目管理系统,据说是源于Basecamp的ror版而来,支持多种数据 ...

  6. .Net 配置 swagger 使用nginx反向代理后请求带端口号导致无法正常访问---解决方法

    1 webconfig中 appsetting 中增加配置 <appSettings> <add key="aspnet:UseHostHeaderForRequestUr ...

  7. Cannot find bounds of current function

    MinGW编译平台的应用程序使用libcef.dll,当调用cef的capi接口时程序崩溃.调试单步到cef capi函数时,调试器报错“Cannot find bounds of current f ...

  8. Java语言中null与" "的区别

    null是空对象,没有地址,可以赋值给任何对象:""是空字符串,有地址但是里面的内容是空的,只能赋值给字符串对象. 如:String s;//分配了一个内存空间,没存入任何对象   ...

  9. LeetCode 290. 单词规律(Word Pattern) 41

    290. 单词规律 290. Word Pattern 题目描述 给定一种规律 pattern 和一个字符串 str,判断 str 是否遵循相同的规律. 这里的 遵循 指完全匹配,例如,pattern ...

  10. eoj monthly 2019.11

    原题 T1 纸条 题目大意: 给出一个长度为n的字符串,其中m位未知,对于每一位未知的字母,有k个备选字母,最终答案为备选字母按字典序排序后的第x个. 题解: 签到题-- 按照题目意思直接写就可以了. ...