【原创】大叔问题定位分享(28)openssh升级到7.4之后ssh跳转异常
服务器集群之间忽然ssh跳转不通
# ssh 192.168.0.1
The authenticity of host '192.168.0.1 (192.168.0.1)' can't be established.
RSA1 key fingerprint is 07:e4:54:79:62:60:22:c2:72:23:21:00:54:a0:90:79.
Are you sure you want to continue connecting (yes/no)?
输入yes之后要求输入密码,但是之前设置的是免密登录,查看 ~/.ssh 目录下文件均正常
# ls -l ~/.ssh
total 16
-rw------- 1 root root 2040 Jan 10 11:32 authorized_keys
-rwx------ 1 root root 1679 Jan 10 11:27 id_rsa
-rwx------ 1 root root 408 Jan 10 11:27 id_rsa.pub
-rwx------ 1 root root 2753 Jan 10 11:27 known_hosts
再检查ssh版本
# ssh -V
OpenSSH_7.4p1, SSH protocols 1.5/2.0, OpenSSL 0x100020bf# yum list installed|grep openssh
openssh.x86_64 7.4p1-16.el7 @base
openssh-clients.x86_64 7.4p1-16.el7 @base
openssh-server.x86_64 7.4p1-16.el7 @base# ls -l /usr/sbin/sshd
-rwxr-xr-x 1 root root 1288984 Feb 13 06:02 /usr/sbin/sshd# ps aux|grep sshd
root 8698 0.0 0.0 25236 1236 ? Ss 06:02 0:00 sshd
看起来是早上6点时sshd刚升级到7.4(文件更新同时进程重启),检查sshd状态发现有报错
# service sshd status
Redirecting to /bin/systemctl status sshd.service
鈼sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2019-02-13 13:41:32 CST; 18s ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 11999 ExecStart=/usr/sbin/sshd -D $OPTIONS (code=exited, status=255)
Main PID: 11999 (code=exited, status=255)
Tasks: 0
Memory: 0B
CGroup: /system.slice/sshd.serviceFeb 13 13:41:32 $server systemd[1]: sshd.service: main process exited, code=exited, status=255/n/a
Feb 13 13:41:32 $server sshd[11999]: This private key will be ignored.
Feb 13 13:41:32 $server sshd[11999]: bad permissions: ignore key: /etc/ssh/ssh_host_rsa_key
Feb 13 13:41:32 $server sshd[11999]: Could not load host key: /etc/ssh/ssh_host_rsa_key
Feb 13 13:41:32 $server sshd[11999]: Could not load host key: /etc/ssh/ssh_host_dsa_key
Feb 13 13:41:32 $server sshd[11999]: Disabling protocol version 2. Could not load host key
Feb 13 13:41:32 $server systemd[1]: Failed to start OpenSSH server daemon.
Feb 13 13:41:32 $server systemd[1]: Unit sshd.service entered failed state.
Feb 13 13:41:32 $server systemd[1]: sshd.service failed.
尝试手工启动sshd来看下具体的错误
# sshd -D -p 8822
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /etc/ssh/ssh_host_rsa_key
Could not load host key: /etc/ssh/ssh_host_rsa_key
Could not load host key: /etc/ssh/ssh_host_dsa_key
Disabling protocol version 2. Could not load host key
看起来是因为key文件权限太大导致ssh protocol 2被禁用
# ls -l /etc/ssh
total 612
-rw-r--r-- 1 root root 581843 Apr 11 2018 moduli
-rw-r--r-- 1 root root 1144 Feb 13 06:02 ssh_config
-rw------- 1 root root 2450 Feb 13 06:02 sshd_config
-rw-r-----. 1 root ssh_keys 227 Jan 22 2018 ssh_host_ecdsa_key
-rw-r--r--. 1 root root 162 Jan 22 2018 ssh_host_ecdsa_key.pub
-rw-r-----. 1 root ssh_keys 387 Jan 22 2018 ssh_host_ed25519_key
-rw-r--r--. 1 root root 82 Jan 22 2018 ssh_host_ed25519_key.pub
-rw------- 1 root root 991 Feb 13 06:02 ssh_host_key
-rw-r--r-- 1 root root 656 Feb 13 06:02 ssh_host_key.pub
-rw-r-----. 1 root ssh_keys 1675 Jan 22 2018 ssh_host_rsa_key
-rw-r--r--. 1 root root 382 Jan 22 2018 ssh_host_rsa_key.pub
将/etc/ssh下文件权限全部改为600
# chmod 600 /etc/ssh/*
然后使用测试的sshd进程跳转8822端口一切正常,但是sshd service还是不断启动失败,怀疑是因为当前的sshd进程不是通过service启动,所以sshd service不断重启但是无法绑定端口,将sshd进程kill掉,再看sshd service终于启动正常
# service sshd status
Redirecting to /bin/systemctl status sshd.service
鈼sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: activating (start) since Wed 2019-02-13 13:35:32 CST; 16s ago
Docs: man:sshd(8)
man:sshd_config(5)
Main PID: 4355 (sshd)
Tasks: 1
Memory: 416.0K
CGroup: /system.slice/sshd.service
鈹斺攢4355 /usr/sbin/sshd -DFeb 13 13:35:32 $server systemd[1]: Starting OpenSSH server daemon...
Feb 13 13:35:32 $server sshd[4355]: Could not load host key: /etc/ssh/ssh_host_dsa_key
Feb 13 13:35:32 $server sshd[4355]: Server listening on 0.0.0.0 port 22.
Feb 13 13:35:32 $server sshd[4355]: error: Bind to port 22 on :: failed: Address already in use.
但是还有一个error: Bind to port 22 on :: failed: Address already in use. 这个是因为在配置文件
$ vi /etc/ssh/sshd_config
#Port 22
#Protocol 2,1
#ListenAddress 0.0.0.0
#ListenAddress ::
默认会绑定ipv4和ipv6的22端口,将其中两行取消注释
Port 22
#Protocol 2,1
ListenAddress 0.0.0.0
#ListenAddress ::
启动正常
Feb 13 15:30:04 $server systemd[1]: Starting OpenSSH server daemon...
Feb 13 15:30:04 $server sshd[4731]: Could not load host key: /etc/ssh/ssh_host_dsa_key
Feb 13 15:30:04 $server sshd[4731]: Server listening on 0.0.0.0 port 22.
但是过一段时间进程就会消失,查看sshd.service
# cat /usr/lib/systemd/system/sshd.service
[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.service
Wants=sshd-keygen.service[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=42s[Install]
WantedBy=multi-user.target
通过journalctl查看日志发现
Feb 13 16:08:09 $server systemd[1]: sshd.service start operation timed out. Terminating.
Feb 13 16:08:09 $server sshd[26701]: Received signal 15; terminating.
Feb 13 16:08:09 $server systemd[1]: sshd.service: main process exited, code=exited, status=255/n/a
Feb 13 16:08:09 $server systemd[1]: Unit sshd.service entered failed state.
Feb 13 16:08:09 $server systemd[1]: sshd.service failed.
Feb 13 16:08:51 $server systemd[1]: sshd.service holdoff time over, scheduling restart.
看起来是因为不断启动超时被中止进程导致;
# vi /usr/lib/systemd/system/sshd.service
ExecStart=/usr/sbin/sshd $OPTIONS
将-D去掉,然后将之前的sshd kill掉,然后重启sshd
# systemctl daemon-reload
# systemctl start sshd
至此sshd服务恢复,ssh跳转恢复,所以openssh升级7.4时一定要修改/etc/ssh/下的文件权限,不然会禁用ssh protocol 2,同时也会导致之前的key失效;同时还要修改sshd.service;等一下,ansible还有报错
192.168.0.1 | FAILED | rc=-1 >>
failed to open a SFTP connection (Channel closed.)
查看sshd_config
# vi /etc/ssh/sshd_config
Subsystem sftp /usr/libexec/sftp-server
发现sftp-server路径不存在/usr/libexec/sftp-server,实际的路径是/usr/libexec/openssh/sftp-server,修改之后重启sshd,上面问题修复,再等一下,scp也有报错
# scp 192.168.0.1:/file1 .
command-line: line 0: Bad configuration option: PermitLocalCommand
这个是由于scp、sftp和ssh版本不匹配导致的,通过yum查看这几个命令都在openssh-clients包中,但是这几个命令的最后修改时间不一样,重装openssh-clients
# yum reinstall openssh-clients
然后这几个命令的最后修改时间都一致了,问题也消失了;
参考:
https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#managed-node-requirements
https://unix.stackexchange.com/questions/390224/openssh-server-start-failed-with-result-timeout/400644#400644
http://www.webopius.com/content/350/solution-to-error-command-line-line-0-bad-configuration-option-permitlocalcommand
【原创】大叔问题定位分享(28)openssh升级到7.4之后ssh跳转异常的更多相关文章
- 【原创】大叔问题定位分享(3)Kafka集群broker进程逐个报错退出
kafka0.8.1 一 问题现象 生产环境kafka服务器134.135.136分别在10月11号.10月13号挂掉: 134日志 [2014-10-13 16:45:41,902] FATAL [ ...
- 【原创】大叔问题定位分享(21)spark执行insert overwrite非常慢,比hive还要慢
最近把一些sql执行从hive改到spark,发现执行更慢,sql主要是一些insert overwrite操作,从执行计划看到,用到InsertIntoHiveTable spark-sql> ...
- 【原创】大叔问题定位分享(13)HBase Region频繁下线
问题现象:hive执行sql报错 select count(*) from test_hive_table; 报错 Error: java.io.IOException: org.apache.had ...
- 【原创】大叔问题定位分享(7)Spark任务中Job进度卡住不动
Spark2.1.1 最近运行spark任务时会发现任务经常运行很久,具体job如下: Job Id ▾ Description Submitted Duration Stages: Succeed ...
- 【原创】大叔问题定位分享(6)Dubbo monitor服务iowait高,负载高
一 问题 Dubbo monitor所在服务器状态异常,iowait一直很高,load也一直很高,监控如下: iowait如图: load如图: 二 分析 通过iotop命令可以查看当前系统中磁盘io ...
- 【原创】大叔问题定位分享(4)Kafka集群broker节点从zookeeper上消失
kafka_2.8.0-0.8.1 一 现象 生产环境一组kafka集群经常发生问题,现象是kafka在zookeeper上的broker节点消失,此时kafka进程和端口都在,然后每个broker都 ...
- 【原创】大叔问题定位分享(30)mesos agent启动失败:Failed to perform recovery: Incompatible agent info detected
mesos agent启动失败,报错如下: Feb 15 22:03:18 server1.bj mesos-slave[1190]: E0215 22:03:18.622994 1192 slave ...
- 【原创】大叔问题定位分享(29)datanode启动报错:50020端口被占用
集群中有一台datanode一直启动报错如下: java.net.BindException: Problem binding to [$server1:50020] java.net.BindExc ...
- 【原创】大叔问题定位分享(27)spark中rdd.cache
spark 2.1.1 spark应用中有一些task非常慢,持续10个小时,有一个task日志如下: 2019-01-24 21:38:56,024 [dispatcher-event-loop-2 ...
随机推荐
- .Net Core应用框架Util介绍(一)
距离上次发文,已经过去了三年半,这几年技术更新节奏异常迅猛,.Net进入了跨平台时代,前端也被革命性的颠覆. 回顾 2015年,正当我还沉迷于JQuery + EasyUi的封装时,突然意识到技术已经 ...
- C# 远程获取图片二进制
直接上代码, 紧做记录. public byte[] GetByteByImgUrl() { System.Net.WebRequest webreq = System.Net.WebRequest. ...
- HTML文字与列表
文字内容 添加文字 <body> 我是Mingle </body> HTML文字的需要添加在<body> </body>里面. 标题字 标题字就是以几种 ...
- 【转】shell脚本中如何传入参数
(1)直接用$1,$2取传入的参数vim /root/test.sh#!/bin/bashif [ $1 == "start" ] then echo "do ...
- Java JPA @Transient 在Hibernate中应用
jpa @Transient - 走过程序员的路 - CSDN博客https://blog.csdn.net/lafengwnagzi/article/details/55511066 Hiberna ...
- jmeter学习记录--03--jmeter负载与监听
jmeter场景主要通过线程组设置完成,有些复杂场景需要与逻辑控制器配合. 一.测试计划设计与执行 场景设计 jmete线程组实际是一个线程池,根据用户设置进行线程池的初始优化,在运行时做各种异常的处 ...
- springboot使用多数据源以及配置
1. 首先在application中配置数据源地址 my.datasource.koi.type=com.alibaba.druid.pool.DruidDataSource my.datasourc ...
- wepy项目创建
全局安装wepy npm install wepy-cli -g 创建项目 wepy init standard mywepy 安装依赖 npm install 实时编译 wepy build --w ...
- dubbo框架提供Main方法运行容器的几种方式(转)
本文使用的是dubbo提供的主类com.alibaba.dubbo.container.Main启动容器. 主要区别是提供不同插件的的启动方式. 目录 一.项目内容 1.1.目录结构图 1.2 ...
- flex知识点归纳
1.flex-shrink <div id="content"> <div class="box" style="backgroun ...