网络错误定位案例 ICMP host *** unreachable - admin prohibited

1. 环境

一台物理服务器 9.115.251.86，上面创建两个虚机，每个虚机两个网卡：

• vm1：eth0 - 9.*.*.232 eth1：10.0.0.14
• vm2: eth0 - 9.8.*.219 eth1:10.0.0.10，上面运行DHCP Agent，管理 dnsmasq，提供 DHCP 服务

两块 eth1 连到物理机上的一个 bridge 上：

bridge name     bridge id               STP enabled     interfaces
mgtbr0          .fa8013216b56       no              tap0
                                                        vnet1
                                                        vnet12
                                                        vnet13
                                                        vnet14
                                                        vnet16
                                                        vnet18
                                                        vnet2
                                                        vnet5

2. 错误

vm1 上创建一个虚机 vmchild1，启动时发出 BOOTP 请求去向 DHCP Agent 获取 IP 地址。DHCP Agent 管理的 dnsmasq 可以正常收到 BOOTP 请求，而且发回了请求，但是 vm1 无法收到。

vm2 上 tcpdump：

::14.135874 IP (tos 0x0, ttl , id , offset , flags [none], proto UDP (), length )
    10.0.0.10. > 10.0.0.14.: OTV, flags [I] (0x08), overlay , instance
IP (tos 0xc0, ttl , id , offset , flags [none], proto UDP (), length )
    ---.static.sonic.net.bootps > ---.static.sonic.net.bootpc: BOOTP/DHCP, Reply, length , xid 0x11f7631f, secs , Flags [none]
          Your-IP ---.static.sonic.net
          Server-IP ---.static.sonic.net
          Client-Ethernet-Address fa::3e:ba::4c (oui Unknown)
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option , length : Offer
            Server-ID Option , length : ---.static.sonic.net
            Lease-Time Option , length :
            RN Option , length :
            RB Option , length :
            Subnet-Mask Option , length : 255.255.255.0
            BR Option , length : ---.static.sonic.net
            Domain-Name Option , length : "openstacklocal"
            Domain-Name-Server Option , length : ---.static.sonic.net
            Default-Gateway Option , length : ---.static.sonic.net
            Classless-Static-Route Option , length : (60.0.0.0/:0.0.0.0),(default:---.static.sonic.net)
::14.136118 IP (tos 0xc0, ttl , id , offset , flags [none], proto ICMP (), length )
    10.0.0.100 > 10.0.0.10: ICMP host 10.0.0.14 unreachable - admin prohibited, length 409
        IP (tos 0x0, ttl , id , offset , flags [none], proto UDP (), length )
    10.0.0.10. > 10.0.0.14.: OTV, flags [I] (0x08), overlay , instance 

而且发现另外的问题：

root@controller:~/s1# ping 10.0.0.13
PING 10.0.0.13 (10.0.0.13) () bytes of data.
 bytes from 10.0.0.13: icmp_seq= ttl= time=0.630 ms
From 10.0.0.13: icmp_seq= Redirect Host(New nexthop: 10.0.0.13)
 bytes from 10.0.0.13: icmp_seq= ttl= time=0.628 ms
From 10.0.0.13: icmp_seq= Redirect Host(New nexthop: 10.0.0.13)

3.定位

初步推断是某处防火墙阻止了 BOOTP 的包，于是将 vm1，vm2 上的防火墙全部关掉，无效。最后，查看物理机 iptables：

[root@rh65 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere            udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:bootps
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpts:vnc-server:cvsup
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             192.168.122.0/24    state RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

发现 FORWARD 阻止了 ICMP 相关的包。

4. 解决

在物理机上运行  service iptables stop 将 iptables 关闭，问题解决。