hadoop HA+Federation（高可用联邦）搭建配置（一）

标签（空格分隔）：未分类

介绍

hadoop 集群一共有4种部署模式，详见《hadoop 生态圈介绍》。

HA联邦模式解决了单纯HA模式的性能瓶颈（主要指Namenode、ResourceManager），将整个HA集群划分为两个以上的集群，不同的集群之间通过Federation进行连接，使得HA集群拥有了横向扩展的能力。理论上，在该模式下，能够通过增加计算节点以处理无限增长的数据。

搭建准备环境

Centos

Hadoop 2.7.3

8台机器

ip	主机名	角色	所属集群
10.111.0.1	VECS00001	zk nn1 jn zkfc rm1	flash1
10.111.0.2	VECS00002	zk nn2 jn zkfc rm2	flash1
10.111.0.3	VECS00003	zk jn
10.111.0.4	VECS00004	zk nn1 jn zkfc	flash2
10.111.0.5	VECS00005	zk nn2 jn zkfc	flash2
10.111.0.6	VECS00006	dn nm	all
10.111.0.7	VECS00007	dn nm	all
10.111.0.8	VECS00008	dn nm	all

Tips：此次搭建并不涉及Resourcemanager的多集群

基础集群安装假设已经完毕：例如zookeeper，集群内所有机器hosts 映射全部已经完成。JDK已经安装完毕，

集群中所有机器环境变量 /etc/profile 全部已同步

JAVA_HOME=/app/jdk/

CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar

# HADOOP CONFIG

export HADOOP_HOME=/app/hadoop

export HADOOP_PREFIX=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib/native"

export YARN_CONF_DIR=$HADOOP_CONF_DIR

export SQOOP_HOME=/app/sqoop

export HIVE_HOME=/app/hive

export PRESTO_HOME=/app/presto

export SCALA_HOME=/usr/local/scala-2.12.4

export SPARK_HOME=/app/spark

export ZOOKEEPER_HOME=/app/zookeeper

export HBASE_HOME=/app/hbase

PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin:$ANT_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin:$PRESTO_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME/bin:$HBASE_HOME/bin:$ZOOKEEPER_HOME/bin:$PATH

export JAVA_HOME CLASSPATH PATH

集群中软件安装标准全部软连接化：

drwxr-xr-x 24 root   root         4096 Feb 26 19:34 anaconda3

drwxr-xr-x 10 deploy deploy       4096 Nov  2  2017 apache-hive-2.1.1-bin

drwxr-xr-x 12 root   root         4096 Feb 14 13:44 apache-hive-2.1.1-bin-test

drwxrwxr-x  5 deploy deploy       4096 Nov 27 17:15 apache-tez-0.9.0

drwxr-xr-x  5 deploy deploy       4096 Feb 14 11:19 apache-tez-0.9.0-bin

drwxrwxr-x  4 deploy deploy      12288 May 14 21:25 datacloud

lrwxrwxrwx  1 deploy deploy         10 Sep 18  2018 datax -> datax-3.0/

drwxrwxr-x 11 deploy deploy       4096 Sep 18  2018 datax-3.0

drwxrwxr-x  9 deploy deploy       4096 Feb 12 00:06 flink-1.7.2

-rw-r--r--  1 root   root    299718134 Feb 12 01:37 flink-1.7.2-bin-hadoop27-scala_2.11.tgz

lrwxrwxrwx  1 root   root           32 Oct 10  2018 hadoop -> hadoop-2.7.3-snappy-32core12disk

drwxrwxrwx 11 deploy deploy       4096 Oct 11  2018 hadoop-2.7.3

drwxr-xr-x 11 hdfs   hadoop       4096 Oct 10  2018 hadoop-2.7.3-snappy-32core12disk

lrwxrwxrwx  1 deploy deploy         27 Feb 14 13:53 hive -> apache-hive-2.1.1-bin-test/

lrwxrwxrwx  1 deploy deploy         12 Jun 25  2018 jdk -> jdk1.8.0_92/

drwxr-xr-x  8 deploy deploy       4096 Apr  1  2016 jdk1.8.0_92

lrwxrwxrwx  1 root   root           21 Nov 27 11:33 presto -> /app/presto-cli-0.207

drwxr-xr-x  7 root   root         4096 Nov 27 11:32 presto-cli-0.207

lrwxrwxrwx  1 deploy deploy         13 Jun 25  2018 scala -> scala-2.11.12

drwxrwxr-x  6 deploy deploy       4096 Nov 10  2017 scala-2.11.12

lrwxrwxrwx  1 deploy deploy         12 Jun 25  2018 spark -> spark-2.1.1/

drwxr-xr-x 15 deploy deploy       4096 Jun 25  2018 spark-2.1.1

-rw-r--r--  1 root   root   1938711032 Apr 10 13:34 spark-2.1.1.tar.gz

drwxrwxr-x 16 deploy deploy       4096 Apr  4 16:57 spark-2.3.1

lrwxrwxrwx  1 deploy deploy         12 Jun 25  2018 sqoop -> sqoop-1.4.6/

drwxrwxr-x  9 deploy deploy       4096 Jun 25  2018 sqoop-1.4.6

lrwxrwxrwx  1 deploy deploy         21 Feb 14 11:23 tez -> apache-tez-0.9.0-bin/

lrwxrwxrwx  1 deploy deploy         17 Jun 25  2018 zookeeper -> zookeeper-3.4.10/

drwxr-xr-x 10 deploy deploy       4096 Jun 25  2018 zookeeper-3.4.10

hadoop 各角色环境变量，JVM 参数配置文件

hadoop-env.sh

改配置文件为 hadoop namenode datanode 及两角色的log目录等设置环境

#Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements.  See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership.  The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License.  You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are

# optional.  When running a distributed configuration it is best to

# set JAVA_HOME in this file, so that it is correctly defined on

# remote nodes.

# The java implementation to use.

# echo java home:${JAVA_HOME}

# export JAVA_HOME=${JAVA_HOME}

export JAVA_HOME="/app/jdk"

export HADOOP_LOG_DIR=${HADOOP_HOME}/log

# The jsvc implementation to use. Jsvc is required to run secure datanodes

# that bind to privileged ports to provide authentication of data transfer

# protocol.  Jsvc is not required if SASL is configured for authentication of

# data transfer protocol using non-privileged ports.

#export JSVC_HOME=${JSVC_HOME}

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

# Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.

for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do

  if [ "$HADOOP_CLASSPATH" ]; then

    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f

  else

    export HADOOP_CLASSPATH=$f

  fi

done

# The maximum amount of heap to use, in MB. Default is 1000.

#export HADOOP_HEAPSIZE="2048"

#export HADOOP_NAMENODE_INIT_HEAPSIZE="2048"

# Extra Java runtime options.  Empty by default.

export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"

#export HADOOP_OPTS="$HADOOP_OPTS -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=8"

# Command specific options appended to HADOOP_OPTS when specified

# export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xmx64G -Xms64G -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/app/hadoop/log/gc/hadoop.log"

export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS -XX:+UseG1GC -Xmx64g -Xms64g  -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -XX:SurvivorRatio=8 -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:ParallelGCThreads=16 -XX:ConcGCThreads=8 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=60 -XX:G1MixedGCLiveThresholdPercent=85 -XX:G1NewSizePercent=1 -XX:G1MaxNewSizePercent=30 -verbose:gc -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1 -Xloggc:/app/hadoop/log/gc/hadoop.log"

export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xmx6144m -Xms6144m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/app/hadoop/log/gc/hadoop_dn.log"

export HADOOP_SECONDARYNAMENODE_OPTS=$HADOOP_NAMENODE_OPTS

#export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"

export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"

export HADOOP_PORTMAP_OPTS="-Xmx1024m $HADOOP_PORTMAP_OPTS"

# The following applies to multiple commands (fs, dfs, fsck, distcp etc)

export HADOOP_CLIENT_OPTS="-Xmx4096m $HADOOP_CLIENT_OPTS"

#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"

# On secure datanodes, user to run the datanode as after dropping privileges.

# This **MUST** be uncommented to enable secure HDFS if using privileged ports

# to provide authentication of data transfer protocol.  This **MUST NOT** be

# defined if SASL is configured for authentication of data transfer protocol

# using non-privileged ports.

export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}

# Where log files are stored.  $HADOOP_HOME/logs by default.

# export HADOOP_LOG_DIR="/workspace/logs/hadoop-logs"

# Where log files are stored in the secure data environment.

export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}

###

# HDFS Mover specific parameters

###

# Specify the JVM options to be used when starting the HDFS Mover.

# These options will be appended to the options specified as HADOOP_OPTS

# and therefore may override any similar flags set in HADOOP_OPTS

#

# export HADOOP_MOVER_OPTS=""

###

# Advanced Users Only!

###

# The directory where pid files are stored. /tmp by default.

# NOTE: this should be set to a directory that can only be written to by

#       the user that will run the hadoop daemons.  Otherwise there is the

#       potential for a symlink attack.

#export HADOOP_PID_DIR=${HADOOP_PID_DIR}

export HADOOP_PID_DIR=${HADOOP_HOME}/tmp

export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}

# A string representing this instance of hadoop. $USER by default.

export HADOOP_IDENT_STRING=$USER

yarn-env.sh

此配置文件决定 Yarn 集群的各角色的jvm 参数，log 日志等等

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# User for YARN daemons

export HADOOP_YARN_USER=${HADOOP_YARN_USER:-yarn}

# resolve links - $0 may be a softlink

export YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"

export YARN_PID_DIR="$HADOOP_HOME/tmp"

# some Java parameters

# export JAVA_HOME=/home/y/libexec/jdk1.6.0/

if [ "$JAVA_HOME" != "" ]; then

  #echo "run java in $JAVA_HOME"

  JAVA_HOME=$JAVA_HOME

fi

if [ "$JAVA_HOME" = "" ]; then

  echo "Error: JAVA_HOME is not set."

  exit 1

fi

JAVA=$JAVA_HOME/bin/java

JAVA_HEAP_MAX=-Xmx1000m 

# For setting YARN specific HEAP sizes please use this

# Parameter and set appropriately

# YARN_HEAPSIZE=1000

# check envvars which might override default args

if [ "$YARN_HEAPSIZE" != "" ]; then

  JAVA_HEAP_MAX="-Xmx""$YARN_HEAPSIZE""m"

fi

export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=-Xmx2048m

# Resource Manager specific parameters

# export YARN_RESOURCEMANAGER_OPTS="-XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xmx20480m -Xms20480m -XX:NewRatio=2 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/app/hadoop/log/gc/yarn-resourcemanager.log -Dcom.sun.management.jmxremote ${YARN_RESOURCEMANAGER_OPTS}"

export YARN_RESOURCEMANAGER_OPTS="-XX:+UseG1GC -Xmx48g -Xms48g  -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:ParallelGCThreads=24 -XX:ConcGCThreads=8 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1MixedGCLiveThresholdPercent=85 -XX:G1NewSizePercent=60 -XX:G1MaxNewSizePercent=75 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/app/hadoop/log/gc/yarn-resourcemanager-`hostname`.log   -Dcom.sun.management.jmxremote ${YARN_RESOURCEMANAGER_OPTS}"

export YARN_NODEMANAGER_OPTS="-XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xmx2048m -Xms2048m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/app/hadoop/log/gc/yarn-nodemanager.log -Dcom.sun.management.jmxremote ${YARN_NODEMANAGER_OPTS}"

# Specify the max Heapsize for the ResourceManager using a numerical value

# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set

# the value to 1000.

# This value will be overridden by an Xmx setting specified in either YARN_OPTS

# and/or YARN_RESOURCEMANAGER_OPTS.

# If not specified, the default value will be picked from either YARN_HEAPMAX

# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.

#export YARN_RESOURCEMANAGER_HEAPSIZE=1000

# Specify the max Heapsize for the timeline server using a numerical value

# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set

# the value to 1000.

# This value will be overridden by an Xmx setting specified in either YARN_OPTS

# and/or YARN_TIMELINESERVER_OPTS.

# If not specified, the default value will be picked from either YARN_HEAPMAX

# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.

#export YARN_TIMELINESERVER_HEAPSIZE=1000

# Specify the JVM options to be used when starting the ResourceManager.

# These options will be appended to the options specified as YARN_OPTS

# and therefore may override any similar flags set in YARN_OPTS

#export YARN_RESOURCEMANAGER_OPTS=

# Node Manager specific parameters

# Specify the max Heapsize for the NodeManager using a numerical value

# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set

# the value to 1000.

# This value will be overridden by an Xmx setting specified in either YARN_OPTS

# and/or YARN_NODEMANAGER_OPTS.

# If not specified, the default value will be picked from either YARN_HEAPMAX

# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.

#export YARN_NODEMANAGER_HEAPSIZE=1000

# Specify the JVM options to be used when starting the NodeManager.

# These options will be appended to the options specified as YARN_OPTS

# and therefore may override any similar flags set in YARN_OPTS

#export YARN_NODEMANAGER_OPTS=

# so that filenames w/ spaces are handled correctly in loops below

IFS=

# default log directory & file

if [ "$YARN_LOG_DIR" = "" ]; then

  YARN_LOG_DIR="$HADOOP_YARN_HOME/log"

fi

if [ "$YARN_LOGFILE" = "" ]; then

  YARN_LOGFILE='yarn.log'

fi

# default policy file for service-level authorization

if [ "$YARN_POLICYFILE" = "" ]; then

  YARN_POLICYFILE="hadoop-policy.xml"

fi

# restore ordinary behaviour

unset IFS

YARN_OPTS="$YARN_OPTS -Dhadoop.log.dir=$YARN_LOG_DIR"

YARN_OPTS="$YARN_OPTS -Dyarn.log.dir=$YARN_LOG_DIR"

YARN_OPTS="$YARN_OPTS -Dhadoop.log.file=$YARN_LOGFILE"

YARN_OPTS="$YARN_OPTS -Dyarn.log.file=$YARN_LOGFILE"

YARN_OPTS="$YARN_OPTS -Dyarn.home.dir=$YARN_COMMON_HOME"

YARN_OPTS="$YARN_OPTS -Dyarn.id.str=$YARN_IDENT_STRING"

YARN_OPTS="$YARN_OPTS -Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"

YARN_OPTS="$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"

if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then

  YARN_OPTS="$YARN_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"

fi

YARN_OPTS="$YARN_OPTS -Dyarn.policy.file=$YARN_POLICYFILE"

#tez config

export TEZ_CONF_DIR=/app/tez/conf

export TEZ_HOME=/app/tez

export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${TEZ_HOME}/*:${TEZ_CONF_DIR}:${TEZ_HOME}/lib/*

marped-site.xml

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# export JAVA_HOME=/home/y/libexec/jdk1.6.0/

export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=4000

export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA

#export HADOOP_JOB_HISTORYSERVER_OPTS=

#export HADOOP_MAPRED_LOG_DIR="" # Where log files are stored.  $HADOOP_MAPRED_HOME/logs by default.

export HADOOP_MAPRED_LOG_DIR="$HADOOP_MAPRED_HOME/log"

#export HADOOP_JHS_LOGGER=INFO,RFA # Hadoop JobSummary logger.

#export HADOOP_MAPRED_PID_DIR= # The pid files are stored. /tmp by default.

#export HADOOP_MAPRED_IDENT_STRING= #A string representing this instance of hadoop. $USER by default

#export HADOOP_MAPRED_NICENESS= #The scheduling priority for daemons. Defaults to 0.

#tez config

export TEZ_CONF_DIR=/app/tez/conf

export TEZ_HOME=/app/tez

export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${TEZ_HOME}/*:${TEZ_CONF_DIR}:${TEZ_HOME}/lib/*

hadoop 各角色本身工作参数配置文件

hadoop 一般常用的用于设置集群内各个角色本身的文件总共有4各：

core-site.xml(namespace 设置)

hdfs-site.xml(hdfs文件系统存储设置，包括nn,dn角色的通用，特殊设置)

yarn-site.xml(Yarn 资源管理集群设置)

mapred-site.xml(yarn 调度资源运行设置)

待续：...