zmq导致master收不到minion的key

问题：

官网方式yum安装完saltstack之后，master与minion都成功启动后，但是却发现怎么都收不到minion的key。（zmq版本不一致所致，困扰了我好几天，希望能帮到大家。）

分析：

master上的状态：

# 查看master 上的配置文件,只修改了interface：
[root@master doubles]# cat /etc/salt/master

interface: 192.168.188.128

[root@master doubles]# systemctl status salt-master
● salt-master.service - The Salt Master Server
   Loaded: loaded (/usr/lib/systemd/system/salt-master.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-01-08 09:55:59 HKT; 7min ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltstack.com/en/latest/contents.html
 Main PID: 9331 (salt-master)
   CGroup: /system.slice/salt-master.service
           ├─9331 /usr/bin/python /usr/bin/salt-master
           ├─9340 /usr/bin/python /usr/bin/salt-master
           ├─9345 /usr/bin/python /usr/bin/salt-master
           ├─9348 /usr/bin/python /usr/bin/salt-master
           ├─9349 /usr/bin/python /usr/bin/salt-master
           ├─9350 /usr/bin/python /usr/bin/salt-master
           ├─9351 /usr/bin/python /usr/bin/salt-master
           ├─9352 /usr/bin/python /usr/bin/salt-master
           ├─9353 /usr/bin/python /usr/bin/salt-master
           ├─9354 /usr/bin/python /usr/bin/salt-master
           ├─9361 /usr/bin/python /usr/bin/salt-master
           ├─9362 /usr/bin/python /usr/bin/salt-master
           └─9364 /usr/bin/python /usr/bin/salt-master

Jan 08 09:55:54 master.doubles.cn systemd[1]: Starting The Salt Master Server...
Jan 08 09:55:59 master.doubles.cn systemd[1]: Started The Salt Master Server.

[root@master doubles]# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
Rejected Keys:
# 始终看不到minion的key。
[root@master doubles]# salt-key -a 192.168.188.129
The key glob '192.168.188.129' does not match any unaccepted keys.
[root@master doubles]# salt '*' test.ping
Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.

看下minion上的salt状态：

# 查看minion上的配置文件,发现只修改了master与id如下：
[root@slave doubles]# vim /etc/salt/minion

master: 192.168.188.128

id: 192.168.188.129

# salt-minion开启调试模式
[root@slave doubles]# salt-minion -l debug
...
[DEBUG   ] SaltReqTimeoutError, retrying. (1/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (2/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (3/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (4/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (5/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (6/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (7/7)
[DEBUG   ] Re-init ZMQ socket: Message timed out
[DEBUG   ] Trying to connect to: tcp://192.168.188.128:4506
[ERROR   ] Error while bringing up minion for multi-master. Is master at 192.168.188.128 responding?
[DEBUG   ] Connecting to master. Attempt 1 of 1
[DEBUG   ] Master URI: tcp://192.168.188.128:4506
[DEBUG   ] Re-using AsyncAuth for (u'/etc/salt/pki/minion', u'192.168.188.129', u'tcp://192.168.188.128:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (7456)
[DEBUG   ] Setting zmq_reconnect_ivl to '7456ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Re-using AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'192.168.188.129', u'tcp://192.168.188.128:4506', 'clear')
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] SaltReqTimeoutError, retrying. (1/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (2/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (3/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (4/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (5/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (6/7)
[DEBUG   ] SaltReqTimeoutError, retrying. (7/7)
[DEBUG   ] Re-init ZMQ socket: Message timed out
[DEBUG   ] Trying to connect to: tcp://192.168.188.128:4506
[ERROR   ] Error while bringing up minion for multi-master. Is master at 192.168.188.128 responding?
...

在这里，本机上的master也是无法收到本机上的minion的key。找了好久，有的说是配置里的master与id没有配对，有的说是增大线程数，有的说是防火墙，有的说是删掉pki下的id，试了都没有用。最后，认真思考一下，其实是minion无法与master 通信，偶然对比了一下minion与master的版本：

master上：

[root@master doubles]# salt-master --versions-report
Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.3.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 3.2.2

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

minion上：

[root@slave doubles]# salt-minion --versions-report
Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

发现master与minion上的ZMQ版本不一样，这应该就是问题所在了。

解决：

尝试更新master上的zmq版本。

master上：

[root@master doubles]# yum install *zmq* –y
# 再对比版本
[root@master doubles]# salt-master --versions-report
Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

此时发现zmq版本已经和minion一致了。重启master再试一下：

master:

[root@master doubles]# systemctl restart salt-master
[root@master doubles]# systemctl status salt-master
● salt-master.service - The Salt Master Server
   Loaded: loaded (/usr/lib/systemd/system/salt-master.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-01-08 11:01:43 HKT; 17s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltstack.com/en/latest/contents.html
 Main PID: 15436 (salt-master)
   CGroup: /system.slice/salt-master.service
           ├─15436 /usr/bin/python /usr/bin/salt-master
           ├─15445 /usr/bin/python /usr/bin/salt-master
           ├─15450 /usr/bin/python /usr/bin/salt-master
           ├─15453 /usr/bin/python /usr/bin/salt-master
           ├─15454 /usr/bin/python /usr/bin/salt-master
           ├─15455 /usr/bin/python /usr/bin/salt-master
           ├─15456 /usr/bin/python /usr/bin/salt-master
           ├─15457 /usr/bin/python /usr/bin/salt-master
           ├─15458 /usr/bin/python /usr/bin/salt-master
           ├─15459 /usr/bin/python /usr/bin/salt-master
           ├─15460 /usr/bin/python /usr/bin/salt-master
           ├─15461 /usr/bin/python /usr/bin/salt-master
           └─15462 /usr/bin/python /usr/bin/salt-master

Jan 08 11:01:42 master.doubles.cn systemd[1]: Stopped The Salt Master Server.
Jan 08 11:01:42 master.doubles.cn systemd[1]: Starting The Salt Master Server...
Jan 08 11:01:43 master.doubles.cn systemd[1]: Started The Salt Master Server.

minion上，重启：

[root@slave doubles]# salt-minion -l debug
...
[DEBUG   ] Trying to connect to: tcp://192.168.188.128:4506
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'192.168.188.129', u'tcp://192.168.188.128:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'192.168.188.129', u'tcp://192.168.188.128:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.188.128:4506
[DEBUG   ] Trying to connect to: tcp://192.168.188.128:4506
[INFO    ] Minion is ready to receive requests!
[DEBUG   ] Minion of '192.168.188.128' is handling event tag '/salt/minion/minion_schedule_delete_complete'
[DEBUG   ] Minion of '192.168.188.128' is handling event tag '/salt/minion/minion_schedule_add_complete'
[DEBUG   ] Minion of '192.168.188.128' is handling event tag '/salt/minion/minion_schedule_delete_complete'
[DEBUG   ] Minion of '192.168.188.128' is handling event tag '/salt/minion/minion_schedule_delete_complete'
...
# 跟之前不一样了

看下master上能否收到key了。

master上：

[root@master doubles]# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
192.168.188.129
Rejected Keys:
[root@master doubles]# salt-key -a 192.168.188.129
The following keys are going to be accepted:
Unaccepted Keys:
192.168.188.129
Proceed? [n/Y] y
Key for minion 192.168.188.129 accepted.
[root@master doubles]# salt-key -L
Accepted Keys:
192.168.188.129
Denied Keys:
Unaccepted Keys:
Rejected Keys:
[root@master doubles]# salt "*" test.ping
192.168.188.129:
    True

发现已经OK了。