07-kubernetes Ingress 原理 和 Ingress-nginx 案例

目录

• Service 类型
• namespace 名称空间
• Ingress Controller
• Ingress
  • Ingress-nginx 进行测试
    • 创建对应的后端Pod和Service
    • 创建 Ingress
    • 查看 Ingress-controller 对应的Pod配置信息
    • 访问测试
  • 模拟测试 Ingress 后端 Tomcat 访问
    • 创建 tomcat 的 Pod 和Service
    • 创建 tomcat-ingress
    • 测试访问 tomcat
• 模拟测试 Https
  • 自签SSL证书
  • 创建secret
  • 创建tls的https清单文件
  • 测试访问

Service 类型

1. ClusterIP 只在集群内部访问，无法跨越集群边界
2. NodePort 提供集群外部流量进入到集群内部
3. LoadBalancer 集群部署在公有云或私有云之上，提供给公有云或私有云的负载均衡中间使用
4. ExternelName 把集群外部的服务映射给集群内部使用， 是一个FQDN的名称，一个CNAME名称，这个CNAM指向公网的FQDN

No ClusterIP 成为 Headless Sercie

ServiceName -> PodIP

Service 是一个四层调度器

也就是说，当请求为HTTPS请求的时候，则无法卸载证书。

namespace 名称空间

；

创建 namespace 如下：

[root@master manifests]# kubectl create namespace dev
namespace/dev created
[root@master manifests]# kubectl get ns
NAME              STATUS   AGE
default           Active   20d
dev               Active   4s       # 新创建的namespace
kube-node-lease   Active   20d
kube-public       Active   20d
kube-system       Active   20d

删除 namespace 如下：

[root@master manifests]# kubectl delete ns/dev
namespace "dev" deleted
[root@master manifests]# kubectl get ns
NAME              STATUS   AGE
default           Active   20d
kube-node-lease   Active   20d
kube-public       Active   20d
kube-system       Active   20d

Ingress Controller

举例

当kubernetes集群有上千甚至跟多个节点的时候，此时需要特有的web七层代理
如在集群其中的四个节点上打上污点，这四个节点上只运行web七层代理所对应的Pod
由此Pod来代理集群内部的Service，Service再把流量转发给集群内部对应的Pod。

这就叫做 Ingress Controller

Ingress Controller 是基于 DaemonSet 控制器来实现。

DaemonSet 是实现保证集群内部每个节点上都运行一个指定的Pod。

Kubernetes 中三种常用七层代理:

1. Nginx # 常规代理
2. Traefik # 微服务前段负载均衡
3. Envoy # 微服务代理负载均衡，使用的多

Ingress

Ingress 和 Ingress Controller 是两回事。

Ingress 原理

Ingress 启动一个独立的Pod来运行七层代理，可以是 Nginx 、 Traefik 或者是 Envoy，此时 ingress Pod 会直接代理 后端运行服务的 Pod，为了能监听后端Pod的变化，需要一个 无头Service Headless Sercie 通过标签选择器来选择后端指定的Pod，并收集到后端Pod 对应的IP，而此Service不会被使用，它主要是用于被Ingress Pod 来监听后端Pod的变化，而一旦后端Pod产生变化，无头Service 会知道，此时被Ingress Pod 发现后，会自动根据变化来更改配置文件，并重载。

如果此时使用的是 Nginx 类型的 Ingress Pod ，则每次变化后修改配置文件后都会自动重新 reload。

但如果使用 Traefik 或者是 Envoy 天生就是为微服务开发的，更友好的支持。

Ingress-nginx 进行测试

可以在 GitHub 上找到Ingress-nginx

本次使用的是相对简单部署 ingress-nginx 的配置清单。

ingress-nginx 单独的详情页面

因为是直接在物理机上直接安装的kubernetes集群，所以需要两个配置清单。

[root@master ingress]#  kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created

还需要一个Ingress-name名称空间中创建的NodePort Service，以便引入集群外流量

需要根据实际情况修改一下：

[root@master ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml
[root@master ingress]# cat service-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 30080       # 手动指定node对外的http端口
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
      nodePort: 30443       # 手动指定node对外的https端口
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

[root@master ingress]#kubectl apply -f service-nodeport.yaml
service/ingress-nginx created

查看创建的ingress-nginx Pod

[root@master ingress]# kubectl get pods -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS   AGE   IP            NODE                NOMINATED NODE   READINESS GATES
nginx-ingress-controller-7995bd9c47-b4fzh   1/1     Running   0          34s   10.244.1.27   node03.kubernetes   <none>           <none>
[root@master ingress]# kubectl get svc -n ingress-nginx     # 这里是第二次执行在Ingress-name名称空间中创建的NodePort Service，以便引入集群外流量
NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.111.201.108   <none>        80:30080/TCP,443:30443/TCP   44s

创建对应的后端Pod和Service

查看创建的ingress-myapp

[root@master ingress]# cat myapp-deplay.yaml
apiVersion: v1
kind: Service
metadata:
  name: myapp       # Service 名称
  namespace: default
spec:
  selector:
    app: myapp
    release: canary
  ports:
  - name: http
    targetPort: 80      # 指定容器端口
    port: 80            # Service 自己开房的端口

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deploy
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      release: canary
  template:
    metadata:
      labels:
        app: myapp
        release: canary
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v2
        ports:
        - name: http
          containerPort: 80
[root@master ingress]# kubectl apply -f myapp-deplay.yaml
service/myapp created
deployment.apps/myapp-deploy unchanged

查看

[root@master ingress]# kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
myapp-deploy-55b78d8548-b4c2c   1/1     Running   0          42h
myapp-deploy-55b78d8548-gpdw8   1/1     Running   0          42h
myapp-deploy-55b78d8548-hr6vx   1/1     Running   0          42h
[root@master ingress]# kubectl get svc
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1     <none>        443/TCP   22d
myapp        ClusterIP   10.111.44.7   <none>        80/TCP    6s

创建 Ingress

[root@master ingress]# cat ingress-myapp.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"        # 自动生成 nginx 相关的匹配和修改规则
spec:
  rules:
  - host: myapp.sijiayong.com       # 使用 host 主机模式来访问
    http:
      paths:
      - path:
        backend:
          serviceName: myapp        # 这里指定刚刚创建的后端Pod对应的Service名称
          servicePort: 80
[root@master ingress]# kubectl apply -f ingress-myapp.yaml
ingress.extensions/ingress-myapp created
[root@master ingress]# kubectl get ing
NAME            HOSTS                 ADDRESS   PORTS   AGE
ingress-myapp   myapp.sijiayong.com             80      6s

查看 Ingress-controller 对应的Pod配置信息

上面一系列都配置完成之后，此时在看查看 Ingress-controller 的Pod中nginx的配置信息包含如下:

[root@master ingress]# kubectl exec -n ingress-nginx -it nginx-ingress-controller-7995bd9c47-b4fzh -- /bin/sh
	server {
		server_name myapp.sijiayong.com ;
		listen 80;

		set $proxy_upstream_name "-";
		set $pass_access_scheme $scheme;
		set $pass_server_port $server_port;
		set $best_http_host $http_host;
		set $pass_port $pass_server_port;
		location / {

			set $namespace      "default";
			set $ingress_name   "ingress-myapp";
			set $service_name   "myapp";
			set $service_port   "80";
			set $location_path  "/";

... ...
... ...

访问测试

此时在集群外部访问测试：

注意，临时测试，上面使用的host模式来访问，所以在测试的集群外部机器上手动添加hosts记录

[root@xinguan-test-node1 ~]# grep myapp /etc/hosts
10.0.20.20 myapp.sijiayong.com
[root@xinguan-test-node1 ~]# curl myapp.sijiayong.com:30080
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>
[root@xinguan-test-node1 ~]# curl myapp.sijiayong.com:30080/hostname.html
myapp-deploy-55b78d8548-hr6vx

模拟测试 Ingress 后端 Tomcat 访问

还是基于上面的配置，一切都不动，增加tomcat后端Pod和Service 以及 Ingress

创建 tomcat 的 Pod 和Service

[root@master ingress]# cat tomcat-deploy.yaml
apiVersion: v1
kind: Service
metadata:
  name: tomcat
  namespace: default
spec:
  selector:
    app: tomcat
    release: canary
  ports:
  - name: http
    targetPort: 8080
    port: 8080
  - name: ajp
    targetPort: 8009
    port: 8009

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-depoly
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tomcat
      release: canary
  template:
    metadata:
      labels:
        app: tomcat
        release: canary
    spec:
      containers:
      - name: tomcat
        image: tomcat:8.5.32-jre8-alpine
        ports:
        - name: http
          containerPort: 8080
        - name: ajp
          containerPort: 8009

创建和查看

[root@master ingress]# kubectl apply -f tomcat-deploy.yaml
service/tomcat created
deployment.apps/tomcat-depoly created
[root@master ingress]# kubectl get pods -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP            NODE                NOMINATED NODE   READINESS GATES
myapp-deploy-55b78d8548-b4c2c    1/1     Running   0          43h   10.244.3.29   node01.kubernetes   <none>           <none>
myapp-deploy-55b78d8548-gpdw8    1/1     Running   0          43h   10.244.1.26   node03.kubernetes   <none>           <none>
myapp-deploy-55b78d8548-hr6vx    1/1     Running   0          43h   10.244.2.22   node02.kubernetes   <none>           <none>
tomcat-depoly-579d97b849-66wrh   1/1     Running   0          7s    10.244.3.30   node01.kubernetes   <none>           <none>
tomcat-depoly-579d97b849-rh5r6   1/1     Running   0          7s    10.244.1.28   node03.kubernetes   <none>           <none>
tomcat-depoly-579d97b849-sd49w   1/1     Running   0          7s    10.244.2.23   node02.kubernetes   <none>           <none>
[root@master ingress]# kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP             22d
myapp        ClusterIP   10.111.44.7    <none>        80/TCP              76m
tomcat       ClusterIP   10.106.39.78   <none>        8080/TCP,8009/TCP   3m18s

创建 tomcat-ingress

[root@master ingress]# cat ingress-tomcat.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-tomcat
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: tomcat.sijiayong.com
    http:
      paths:
      - path:
        backend:
          serviceName: tomcat
          servicePort: 8080

创建和查看

[root@master ingress]# kubectl apply -f ingress-tomcat.yaml
ingress.extensions/ingress-tomcat created
[root@master ingress]# kubectl get ing
NAME             HOSTS                  ADDRESS   PORTS   AGE
ingress-myapp    myapp.sijiayong.com              80      53m
ingress-tomcat   tomcat.sijiayong.com             80      4s

测试访问 tomcat

注意，同样写Hosts 之后进行测试

[root@xinguan-test-node1 ~]# grep tomcat /etc/hosts
10.0.20.20 myapp.sijiayong.com tomcat.sijiayong.com
[root@xinguan-test-node1 ~]# curl tomcat.sijiayong.com:30080

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <title>Apache Tomcat/8.5.32</title>
        <link href="favicon.ico" rel="icon" type="image/x-icon" />
        <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" />
        <link href="tomcat.css" rel="stylesheet" type="text/css" />
    </head>

    <body>
        <div id="wrapper">
            <div id="navigation" class="curved container">
                <span id="nav-home"><a href="http://tomcat.apache.org/">Home</a></span>
                <span id="nav-hosts"><a href="/docs/">Documentation</a></span>
                <span id="nav-config"><a href="/docs/config/">Configuration</a></span>
                <span id="nav-examples"><a href="/examples/">Examples</a></span>
                <span id="nav-wiki"><a href="http://wiki.apache.org/tomcat/FrontPage">Wiki</a></span>
                <span id="nav-lists"><a href="http://tomcat.apache.org/lists.html">Mailing Lists</a></span>
                <span id="nav-help"><a href="http://tomcat.apache.org/findhelp.html">Find Help</a></span>
                <br class="separator" />
            </div>
            <div id="asf-box">
                <h1>Apache Tomcat/8.5.32</h1>
            </div>
... ...
... ...

模拟测试 Https

模拟测试 通过Ingress 的七层代理，访问Https，通过Ingress卸载证书后访问后端

所有先需要手动自签SSL证书

自签SSL证书

[root@master https]# openssl genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
...............................+++
.....................................................................................................................................+++
e is 65537 (0x10001)
[root@master https]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=BeiJing/L=BeiJing/O=DefOps/CN=tomcat.sijiayong.com
[root@master https]# ll
total 8
-rw-r--r-- 1 root root 1302 Aug  1 11:33 tls.crt
-rw-r--r-- 1 root root 1675 Aug  1 11:31 tls.key

自签证书完成

创建secret

注意，此证书不能直接帖进去，需要先创建成secret，才能应用到Ingress中

[root@master https]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key
secret/tomcat-ingress-secret created
[root@master https]# kubectl get secret
NAME                    TYPE                                  DATA   AGE
default-token-bc86p     kubernetes.io/service-account-token   3      22d
tomcat-ingress-secret   kubernetes.io/tls                     2      10s
[root@master https]# kubectl describe secret tomcat-ingress-secret
Name:         tomcat-ingress-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.key:  1675 bytes
tls.crt:  1302 bytes

创建tls的https清单文件

[root@master https]# cat ingress-tomcat-tls.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-tomcat-tls
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - tomcat.sijiayong.com
    secretName: tomcat-ingress-secret       # 这里指定刚刚创建的secret名称
  rules:
  - host: tomcat.sijiayong.com
    http:
      paths:
      - path:
        backend:
          serviceName: tomcat
          servicePort: 8080

创建和查看

[root@master https]# kubectl apply -f ingress-tomcat-tls.yaml
ingress.extensions/ingress-tomcat-tls created
[root@master https]# kubectl get ing
NAME                 HOSTS                  ADDRESS   PORTS     AGE
ingress-myapp        myapp.sijiayong.com              80        174m
ingress-tomcat       tomcat.sijiayong.com             80        121m
ingress-tomcat-tls   tomcat.sijiayong.com             80, 443   4s          # 这里看到被创建成功
[root@master https]# kubectl describe ingres-tomcat-tls
error: the server doesn't have a resource type "ingres-tomcat-tls"
[root@master https]# kubectl describe ingress ingress-tomcat-tls
Name:             ingress-tomcat-tls
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
TLS:
  tomcat-ingress-secret terminates tomcat.sijiayong.com
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  tomcat.sijiayong.com
                           tomcat:8080 (10.244.1.28:8080,10.244.2.23:8080,10.244.3.30:8080)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-tomcat-tls","namespace":"default"},"spec":{"rules":[{"host":"tomcat.sijiayong.com","http":{"paths":[{"backend":{"serviceName":"tomcat","servicePort":8080},"path":null}]}}],"tls":[{"hosts":["tomcat.sijiayong.com"],"secretName":"tomcat-ingress-secret"}]}}

  kubernetes.io/ingress.class:  nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  26s   nginx-ingress-controller  Ingress default/ingress-tomcat-tls

测试访问

此时就已经完成，可以测试访问

因前面测试hosts已经写好