十一章 Kubernetes的服务发现插件--coredns

1、前言

• 简单来说，服务发现就是服务（应用）之间相互定位的过程；
• 服务发现并非云计算时代独有的，传统的单体架构时代也会用到，以下应用场景更加需要服务发现：
  
  服务（应用）的动态性强；
  
  服务（应用）更新发布频繁；
  
  服务（支持自动）。
• 在k8s集群中pod的IP是不断变化的，所以如何解决这种东问题呢：
  
  抽象出了service资源，通过标签选择器，关联一组pod；
  
  抽象出了集群网络，通过相对固定的“集群（service/Cluster）IP”是服务介入固定IP；
• 那么如何自动关联service资源的“名称”和“集群网络IP”，从而达到服务被集群自动发现的目的呢：
  
  考虑传统DNS模型：hdss7-21.host.com -->10.4.7.21；
  
  能否在k8s中建立这样的模型：nginx-ds-->192.168.0.5。

2、部署k8s的内网资源配置清单http服务

2.1 更新域名解析

在10.4.7.11域名服务器上配置新域名解析，注意要前滚一个序列号

[root@hdss7-11 ~]# vim /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600  ; 10 minutes
@       IN SOA  dns.od.com. dnsadmin.od.com. (
        2020010503 ; serial
        10800      ; refresh (3 hours)
        900        ; retry (15 minutes)
        604800     ; expire (1 week)
        86400      ; minimum (1 day)
        )
        NS   dns.od.com.
$TTL 60 ; 1 minute
dns                A    10.4.7.11
harbor             A    10.4.7.200
k8s-yaml           A    10.4.7.200
[root@hdss7-11 ~]# systemctl restart named
[root@hdss7-11 ~]# dig -t A k8s-yaml.od.com @10.4.7.11 +short
10.4.7.200

2.2 添加nginx代理

在运维主机10.4.7.200上部署一个nginx的虚拟主机，用以提供k8s统一的资源配置清单访问入口。

配置nginx

[root@hdss7-200 ~]# vim /etc/nginx/conf.d/k8s-yaml.od.com.conf
server {
    listen       80;
    server_name  k8s-yaml.od.com k8s-yaml.grep.pro;

    location / {
        autoindex on;
        default_type text/plain;
        root /data/k8s-yaml;
    }
}
[root@hdss7-200 ~]# mkdir -p /data/k8s-yaml
[root@hdss7-200 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@hdss7-200 ~]# nginx -s reload
[root@hdss7-200 ~]# cd /data/k8s-yaml/
[root@hdss7-200 k8s-yaml]# ls
[root@hdss7-200 k8s-yaml]# mkdir coredns

访问浏览器http://k8s-yaml.od.com/

3、部署coredns

2.1、准备镜像

[root@hdss7-200 k8s-yaml]# cd /data/k8s-yaml/coredns/
[root@hdss7-200 coredns]# docker pull coredns/coredns:1.6.1
[root@hdss7-200 coredns]# docker tag c0f6e815079e harbor.od.com/public/coredns:v1.6.1
[root@hdss7-200 coredns]# docker push harbor.od.com/public/coredns:v1.6.1
或者coredns]# docker push !$

2.2、准备资源配置清单

cm.yaml中 forward . 10.4.7.11为 集群的dns地址也是coredns的上游dns地址

svc.yaml中 clusterIP: 192.168.0.2地址为集群中kubelet定义好的集群dns地址

在10.4.7.200上部署

[root@hdss7-200 coredns]# cd /data/k8s-yaml/coredns/

配置rbac.yaml

[root@hdss7-200 coredns]# vim rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
  labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: Reconcile
  name: system:coredns
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: EnsureExists
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system

配置cm.yaml

[root@hdss7-200 coredns]# vim cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        log
        health
        ready
        kubernetes cluster.local 192.168.0.0/16
        forward . 10.4.7.11
        cache 30
        loop
        reload
        loadbalance
       }

配置dp.yaml

[root@hdss7-200 coredns]# vim dp.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: coredns
    kubernetes.io/name: "CoreDNS"
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: coredns
  template:
    metadata:
      labels:
        k8s-app: coredns
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      containers:
      - name: coredns
        image: harbor.od.com/public/coredns:v1.6.1
        args:
        - -conf
        - /etc/coredns/Corefile
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile

配置svc.yaml

[root@hdss7-200 coredns]# vim svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: coredns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: coredns
  clusterIP: 192.168.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
  - name: metrics
    port: 9153
    protocol: TCP

2.3、依次执行创建资源

在其中一个运算节点操作，此处以10.4.7.21为例

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/coredns/rbac.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/coredns/cm.yaml
configmap/coredns created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/coredns/dp.yaml
deployment.apps/coredns created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/coredns/svc.yaml
service/coredns created

2.4、在kube-system中查询资源

[root@hdss7-21 ~]# kubectl get all -n kube-system
NAME                           READY   STATUS    RESTARTS   AGE
pod/coredns-64f49f5655-xs9dq   1/1     Running   0          114s

NAME              TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
service/coredns   ClusterIP   192.168.0.2   <none>        53/UDP,53/TCP,9153/TCP   108s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns   1/1     1            1           114s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/coredns-64f49f5655   1         1         1       114s

2.5、检查

[root@hdss7-21 ~]# dig -t A www.baidu.com @192.168.0.2 +short
www.a.shifen.com.
39.156.66.14
39.156.66.18
[root@hdss7-21 ~]# dig -t A hdss7-200.host.com @192.168.0.2 +short
10.4.7.200
[root@hdss7-21 ~]# kubectl get svc -o wide
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE   SELECTOR
kubernetes   ClusterIP   192.168.0.1   <none>        443/TCP   9d    <none>

创建一个service

[root@hdss7-21 ~]# kubectl create deployment nginx-dp --image=harbor.od.com/public/nginx:v1.7.9
deployment.apps/nginx-dp created
[root@hdss7-21 ~]# kubectl expose deployment nginx-dp --port=80 --target-port=80
service/nginx-dp exposed
[root@hdss7-21 ~]# kubectl get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
nginx-dp-86678bb55c-tklvc   1/1     Running   0          19s   172.7.21.4   hdss7-21.host.com   <none>           <none>
nginx-ds-2rj9d              1/1     Running   0          84m   172.7.21.2   hdss7-21.host.com   <none>           <none>
nginx-ds-cnz5d              1/1     Running   0          95m   172.7.22.3   hdss7-22.host.com   <none>           <none>

[root@hdss7-21 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   192.168.0.1     <none>        443/TCP   9d
nginx-dp     ClusterIP   192.168.191.8   <none>        80/TCP    61s

解析域名

[root@hdss7-21 ~]#  dig -t A nginx-dp.default.svc.cluster.local @192.168.0.2 +short
192.168.191.8

在容器集群内部访问域名

[root@hdss7-21 ~]# kubectl exec -it nginx-dp-86678bb55c-tklvc bash
root@nginx-dp-86678bb55c-tklvc:/# ping nginx-dp.default.svc.cluster.local
PING nginx-dp.default.svc.cluster.local (192.168.191.8): 48 data bytes
56 bytes from 192.168.191.8: icmp_seq=0 ttl=64 time=0.281 ms
56 bytes from 192.168.191.8: icmp_seq=1 ttl=64 time=0.081 ms

root@nginx-dp-86678bb55c-tklvc:/# curl nginx-dp.default.svc.cluster.local
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

如下/etc/resolv.conf，可以直接访问短域名

root@nginx-dp-86678bb55c-tklvc:/# cat /etc/resolv.conf
nameserver 192.168.0.2
search default.svc.cluster.local svc.cluster.local cluster.local host.com
options ndots:5
root@nginx-dp-86678bb55c-tklvc:/# ping nginx-dp
PING nginx-dp.default.svc.cluster.local (192.168.176.18): 48 data bytes
56 bytes from 192.168.176.18: icmp_seq=0 ttl=64 time=0.135 ms
56 bytes from 192.168.176.18: icmp_seq=1 ttl=64 time=0.110 ms

但是容器外非集群却无法访问此域名

root@nginx-dp-86678bb55c-tklvc:/# exit
exit
[root@hdss7-21 ~]# ping nginx-dp.default.svc.cluster.local
ping: nginx-dp.default.svc.cluster.local: 未知的名称或服务
[root@hdss7-21 ~]# curl nginx-dp.default.svc.cluster.local
curl: (6) Could not resolve host: nginx-dp.default.svc.cluster.local; 未知的错误

目前coredns已经完成了服务的域名和cluster的IP做了一个自动关联，实现了服务的自动发现，下一节会学习如何在集群外暴露服务域名

2.6 容器可以访问外网

注意：并且之前我们用的192.168.0.2的dns在容器中时无法访问外网的，部署好coredns后也可以对外网进行访问了

root@nginx-dp-86678bb55c-tklvc:/# ping baidu.com
PING baidu.com (39.156.69.79): 48 data bytes
56 bytes from 39.156.69.79: icmp_seq=0 ttl=52 time=10.574 ms
56 bytes from 39.156.69.79: icmp_seq=1 ttl=52 time=9.379 ms
^C--- baidu.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 9.379/9.976/10.574/0.598 ms