Hadoop集群模式安装出现的若干问题

一、域名解析问题

域名解析暂时失败问题
　　vim /etc/sysconfig/network
　　查看主机名
　　vim etc/hosts
配置IP地址与主机名 192.168.60.132 centos　　#改正就OK啦

二、启动问题

Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [bogon]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.

处理1

$ vim sbin/start-dfs.sh
$ vim sbin/stop-dfs.sh
两处增加以下内容
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
处理2
$ vim sbin/start-yarn.sh
$ vim sbin/stop-yarn.sh
两处增加以下内容
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

三、端口问题

    localhost: ssh: connect to host localhost port 22: Cannot assign requested address
    cd /etc/ssh
    vim sshd_config
    添加 Port 22

四、Failed to get D-Bus connection: Operation not permitted

解决方法：docker run --privileged -ti -e "container=docker" -v /sys/fs/cgroup:/sys/fs/cgroup hadoop-master /usr/sbin/init

五、服务报错

sshd re-exec requires execution with an absolute path
在开启SSHD服务时报错.
sshd re-exec requires execution with an absolute path
用绝对路径启动,也报错如下:
Could not load host key: /etc/ssh/ssh_host_key
Could not load host key: /etc/ssh/ssh_host_rsa_key
Could not load host key: /etc/ssh/ssh_host_dsa_key
Disabling protocol version 1. Could not load host key
Disabling protocol version 2. Could not load host key
sshd: no hostkeys available — exiting

解决过程:

#ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
#ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
#/usr/sbin/sshd

执行后报错：

Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key

解决过程：

#ssh-keygen -t dsa -f /etc/ssh/ssh_host_ecdsa_key
#ssh-keygen -t rsa -f /etc/ssh/ssh_host_ed25519_key
#/usr/sbin/sshd

六、报错如下

    WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
    Starting namenodes on [master]
    master: /usr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 982: ssh: command not found
    Starting datanodes
    Last login: Mon Jan 28 08:32:32 UTC 2019 on pts/0
    localhost: /usr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 982: ssh: command not found
    Starting secondary namenodes [b982e2adc393]
    Last login: Mon Jan 28 08:32:33 UTC 2019 on pts/0
    b982e2adc393: /usr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 982: ssh: command not found
    Starting resourcemanager
    Last login: Mon Jan 28 08:32:35 UTC 2019 on pts/0
    Starting nodemanagers
    Last login: Mon Jan 28 08:32:42 UTC 2019 on pts/0
    localhost: /usr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 982: ssh: command not foun

解决：
    在
     $ vim sbin/start-dfs.sh
     $ vim sbin/stop-dfs.sh
    将HADOOP_SECURE_DN_USER=hdfs替换为HADOOP_DATANODE_SECURE_DN_USER=hdfs
    centos默认安装有ssh服务，没有客户端。
    查看ssh安装
    # rpm -qa | grep openssh
    openssh-5.3p1-123.el6_9.x86_64
    openssh-server-5.3p1-123.el6_9.x86_64
    没有安装openssh-clients
    yum安装ssh客户端
    yum -y install openssh-clients

七、docker: Error response from daemon: cgroups: cannot find cgroup mount destination: unknown.

没有找到具体的解决方法，重启后可以访问。

八、permission denied (publickey,gssapi-keyex,gssapi-with-mic,password)

vim /etc/ssh/sshd_config

# $OpenBSD: sshd_config,v 1.80 2008/07/02 02:24:18 djm Exp $

        # This is the sshd server system-wide configuration file.  See
        # sshd_config(5) for more information.

        # This sshd was compiled with PATH=/usr/local/bin:/bin:/usr/bin

        # The strategy used for options in the default sshd_config shipped with
        # OpenSSH is to specify options with their default value where
        # possible, but leave them commented.  Uncommented options change a
        # default value.

        #Port 22
        #AddressFamily any
        #ListenAddress 0.0.0.0
        #ListenAddress ::

        # Disable legacy (protocol version 1) support in the server for new
        # installations. In future the default will change to require explicit
        # activation of protocol 1
        Protocol 2

        # HostKey for protocol version 1

-----------
就是改一下信息
RSAAuthentication yes
PubkeyAuthentication yes
PasswordAuthentication no

改完后一定要重启ssh服务 service sshd restart
systemctl restart sshd.service

九、WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

系统预装的glibc库是2.12版本，而hadoop期望是2.14版本，所以打印警告信息。
现在有两个办法，重新编译glibc.2.14版本，安装后专门给hadoop使用，这个有点危险。
第二个办法直接在log4j日志中去除告警信息。在//usr/local/hadoop-2.5.2/etc/hadoop/log4j.properties文件中添加
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

十、MapReduce操作出现错误: 找不到或无法加载主类org.apache.hadoop.mapreduce.v2.app.MRAppMaster 问题解决方法

在命令行下输入 Hadoop classpath
编辑yarn-site.xml
添加一下内容

<configuration>
<property>
<name>yarn.application.classpath</name>
<value>输入刚才返回的Hadoop classpath路径</value>
</property>
</configuration>