Ingress-nginx 部署使用

一、Ingress 简介

在Kubernetes中，服务和Pod的IP地址仅可以在集群网络内部使用，对于集群外的应用是不可见的。为了使外部的应用能够访问集群内的服务，在Kubernetes 目前 提供了以下几种方案：

NodePort

LoadBalancer

Ingress

Ingress 组成

ingress controller

　　将新加入的Ingress转化成Nginx的配置文件并使之生效

ingress服务

　　将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可

Ingress 工作原理

1.ingress controller通过和kubernetes api交互，动态的去感知集群中ingress规则变化，

2.然后读取它，按照自定义的规则，规则就是写明了哪个域名对应哪个service，生成一段nginx配置，

3.再写到nginx-ingress-control的pod里，这个Ingress controller的pod里运行着一个Nginx服务，控制器会把生成的nginx配置写入/etc/nginx.conf文件中，

4.然后reload一下使配置生效。以此达到域名分配置和动态更新的问题。

Ingress 可以解决什么问题

1.动态配置服务

　　如果按照传统方式, 当新增加一个服务时, 我们可能需要在流量入口加一个反向代理指向我们新的k8s服务. 而如果用了Ingress, 只需要配置好这个服务, 当服务启动时, 会自动注册到Ingress的中, 不需要而外的操作.

2.减少不必要的端口暴露

　　配置过k8s的都清楚, 第一步是要关闭防火墙的, 主要原因是k8s的很多服务会以NodePort方式映射出去, 这样就相当于给宿主机打了很多孔, 既不安全也不优雅. 而Ingress可以避免这个问题, 除了Ingress自身服务可能需要映射出去, 其他服务都不要用NodePort方式

二、部署配置Ingress

1 .部署文件介绍、准备

获取配置文件位置: https://github.com/kubernetes/ingress-nginx/tree/nginx-0.20.0/deploy

下载部署文件

提供了两种方式 ：

　　默认下载最新的yaml：

　　　　wget  https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml

　　指定版本号下载对应的yaml

部署文件介绍

1.namespace.yaml

创建一个独立的命名空间 ingress-nginx

2.configmap.yaml

ConfigMap是存储通用的配置变量的，类似于配置文件，使用户可以将分布式系统中用于不同模块的环境变量统一到一个对象中管理；而它与配置文件的区别在于它是存在集群的“环境”中的，并且支持K8S集群中所有通用的操作调用方式。

从数据角度来看，ConfigMap的类型只是键值组，用于存储被Pod或者其他资源对象（如RC）访问的信息。这与secret的设计理念有异曲同工之妙，主要区别在于ConfigMap通常不用于存储敏感信息，而只存储简单的文本信息。

ConfigMap可以保存环境变量的属性，也可以保存配置文件。

创建pod时，对configmap进行绑定，pod内的应用可以直接引用ConfigMap的配置。相当于configmap为应用/运行环境封装配置。

pod使用ConfigMap，通常用于：设置环境变量的值、设置命令行参数、创建配置文件。

3.default-backend.yaml

如果外界访问的域名不存在的话，则默认转发到default-http-backend这个Service，其会直接返回404：

4.rbac.yaml

负责Ingress的RBAC授权的控制，其创建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding

5.with-rbac.yaml

是Ingress的核心，用于创建ingress-controller。前面提到过，ingress-controller的作用是将新加入的Ingress进行转化为Nginx的配置

2.部署ingress

准备镜像，从这里mandatory.yaml查看需要哪些镜像

镜像名称	版本	下载地址
k8s.gcr.io/defaultbackend-amd64	1.5	registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64
quay.io/kubernetes-ingress-controller/nginx-ingress-controller	0.20.0	registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller

在每一个节点(Node)上下载镜像:

[root@k8s-node1 ~]# docker pull registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5  #注意版本号

Trying to pull repository registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64 ...

1.5: Pulling from registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64

26de8f6c1f4b: Pull complete

Digest: sha256:2cdff48ab9b20ca5f9b0ee48bf3c139c51d6fb1a15245966583bc371c121c238

Status: Downloaded newer image for registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5

[root@k8s-node1 ~]#  docker pull registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0 #注意版本号

Trying to pull repository registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller ...

0.20.0: Pulling from registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller

8e41b996a802: Pull complete

f8e7d603ef88: Pull complete

610da4f3123f: Pull complete

62702a85e6c9: Pull complete

7cedf5b2083f: Pull complete

755f7fa719b6: Pull complete

6adbdd0c8aaf: Pull complete

2389d8b0d2a2: Pull complete

1ea794448393: Pull complete

bb0c388ee432: Pull complete

9626641c5a97: Pull complete

bd0bebb5ba38: Pull complete

Digest: sha256:3f06079f7727b2fb7ad5c97d8152eb622ae504674395dfa71fda7ce315aaaf30

Status: Downloaded newer image for registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0

[root@k8s-node1 ~]# docker images  #检查镜像是否下载成功

REPOSITORY                                                                    TAG                 IMAGE ID            CREATED             SIZE

registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64       1.5                 d8f37b8cdaf4        2 weeks ago         5.13 MB

registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller   0.20.0              3cc332ecde4f        3 weeks ago         513 MB

k8s.gcr.io/kube-proxy-amd64                                                   v1.10.0             bfc21aadc7d3        7 months ago        97 MB

k8s.gcr.io/kube-controller-manager-amd64                                      v1.10.0             ad86dbed1555        7 months ago        148 MB

k8s.gcr.io/kube-scheduler-amd64                                               v1.10.0             704ba848e69a        7 months ago        50.4 MB

k8s.gcr.io/kube-apiserver-amd64                                               v1.10.0             af20925d51a3        7 months ago        225 MB

k8s.gcr.io/etcd-amd64                                                         3.1.12              52920ad46f5b        8 months ago        193 MB

k8s.gcr.io/kubernetes-dashboard-amd64                                         v1.8.3              0c60bcf89900        8 months ago        102 MB

quay.io/coreos/flannel                                                        v0.10.0-amd64       f0fad859c909        9 months ago        44.6 MB

k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64                                        1.14.8              c2ce1ffb51ed        10 months ago       41 MB

k8s.gcr.io/k8s-dns-sidecar-amd64                                              1.14.8              6f7f2dc7fab5        10 months ago       42.2 MB

k8s.gcr.io/k8s-dns-kube-dns-amd64                                             1.14.8              80cc5ea4b547        10 months ago       50.5 MB

k8s.gcr.io/pause-amd64                                                        3.1                 da86e6ba6ca1        10 months ago       742 kB

docker.io/kubeguide/tomcat-app                                                v1                  a29e200a18e9        2 years ago         358 MB

下载yaml文件并更新mandatory.yaml中的镜像地址（master上）

[root@k8s-master ~]# mkdir ingress-nginx

[root@k8s-master ingress-nginx]# wget  https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.20.0/deploy/mandatory.yaml

[root@k8s-master ingress-nginx]# wget  https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml      #对外提供服务，如果不需要可以不下载

[root@k8s-master ingress-nginx]# sed -i 's#k8s.gcr.io/defaultbackend-amd64#registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64#g' mandatory.yaml  #替换defaultbackend-amd64镜像地址

 sed -i 's#quay.io/kubernetes-ingress-controller/nginx-ingress-controller#registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller#g' mandatory.yaml  #替换nginx-ingress-controller镜像地址

[root@k8s-master ingress-nginx]# grep image mandatory.yaml   #检查替换结果

          # Any image is permissible as long as:

          image: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5

          image: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0

修改service-nodeport.yaml文件，添加NodePort端口，默认为随机端口

[root@k8s-master ingress-nginx]# cat service-nodeport.yaml

apiVersion: v1

kind: Service

metadata:

  name: ingress-nginx

  namespace: ingress-nginx

  labels:

    app.kubernetes.io/name: ingress-nginx

    app.kubernetes.io/part-of: ingress-nginx

spec:

  type: NodePort

  ports:

    - name: http

      port: 80

      targetPort: 80

      protocol: TCP

      nodePort: 32080  #http

    - name: https

      port: 443

      targetPort: 443

      protocol: TCP

      nodePort: 32443  #https

  selector:

    app.kubernetes.io/name: ingress-nginx

    app.kubernetes.io/part-of: ingress-nginx

部署nginx-ingress-controller

[root@k8s-master ingress-nginx]# kubectl apply -f mandatory.yaml

namespace "ingress-nginx" created

deployment.extensions "default-http-backend" created

service "default-http-backend" created

configmap "nginx-configuration" created

configmap "tcp-services" created

configmap "udp-services" created

serviceaccount "nginx-ingress-serviceaccount" created

clusterrole.rbac.authorization.k8s.io "nginx-ingress-clusterrole" created

role.rbac.authorization.k8s.io "nginx-ingress-role" created

rolebinding.rbac.authorization.k8s.io "nginx-ingress-role-nisa-binding" created

clusterrolebinding.rbac.authorization.k8s.io "nginx-ingress-clusterrole-nisa-binding" created

deployment.extensions "nginx-ingress-controller" created

[root@k8s-master ingress-nginx]# kubectl apply -f service-nodeport.yaml

service "ingress-nginx" created

3.查看ingress-nginx组件状态

[root@k8s-master ingress-nginx]#  kubectl get pods -n ingress-nginx    #pod状态

NAME                                        READY     STATUS    RESTARTS   AGE

default-http-backend-66c4fbf5b4-x2n8w       1/1       Running   0          58s

nginx-ingress-controller-64bcff8657-5gdrd   1/1       Running   0          58s

[root@k8s-master ingress-nginx]#  kubectl get svc -n ingress-nginx  #service状态及暴露端口

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE

default-http-backend   ClusterIP   10.96.87.65     <none>        80/TCP                       1m

ingress-nginx          NodePort    10.100.48.237   <none>        80:32080/TCP,443:32443/TCP   1m

4.访问ingress-nginx服务，查看是否配置成功

可以看到，提示404，这个因为当前ingress-nginx服务现在还没有后端服务，这是正常的

三、创建ingress-nginx后端服务

1.创建一个Service及后端Deployment(以nginx为例)

[root@k8s-master01 ingress]# cat deploy-demon.yaml

apiVersion: v1

kind: Service

metadata:

  name: myapp

  namespace: default

spec:

  selector:

    app: myapp

    release: canary

  ports:

  - name: http

    port: 80

    targetPort: 80

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: myapp-deploy

spec:

  replicas: 5

  selector:

    matchLabels:

      app: myapp

      release: canary

  template:

    metadata:

      labels:

        app: myapp

        release: canary

    spec:

      containers:

      - name: myapp

        image: ikubernetes/myapp:v2

        ports:

        - name: httpd

          containerPort: 80

创建相关服务及检查状态是否就绪

[root@k8s-master ingress-nginx]# kubectl apply -f deploy-demon.yaml

service "myapp" created

deployment.apps "myapp-deploy" created

[root@k8s-master ingress-nginx]# kubectl get pods

NAME                            READY     STATUS    RESTARTS   AGE

myapp-deploy-5cfd895984-ffzm5   1/1       Running   0          1m

myapp-deploy-5cfd895984-ftg9t   1/1       Running   0          1m

myapp-deploy-5cfd895984-jg887   1/1       Running   0          1m

myapp-deploy-5cfd895984-mk4jq   1/1       Running   0          1m

myapp-deploy-5cfd895984-nqz6s   1/1       Running   0          1m

myweb-hrfqm                     1/1       Running   0          8d

myweb-pb5tb                     1/1       Running   0          8d

myweb-xrk22                     1/1       Running   0          8d

[root@k8s-master ingress-nginx]# kubectl get svc

NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE

kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP          9d

myapp        ClusterIP   10.102.30.215    <none>        80/TCP           1m

myweb        NodePort    10.106.138.244   <none>        8080:31888/TCP   8d

将myapp添加至ingress-nginx中

[root@k8s-master ingress-nginx]# cat ingress-myapp.yaml

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  name: ingress-myapp

  namespace: default

  annotations:

    kubernetes.io/ingress.class: "nginx"

spec:

  rules:

  - host: myapp.magedu.com #生产中该域名应当可以被公网解析

    http:

      paths:

      - path:

        backend:

          serviceName: myapp

          servicePort: 80

[root@k8s-master ingress-nginx]# kubectl apply -f ingress-myapp.yaml
ingress.extensions "ingress-myapp" created

配置域名解析，当前测试环境我们使用hosts文件进行解析

172.33.16.241  myapp.magedu.com

使用域名进行访问

2.再创建一个Service及后端Deployment(以tomcat为例)

[root@k8s-master ingress-nginx]# cat tomcat-deploy.yaml

apiVersion: v1

kind: Service

metadata:

  name: tomcat

  namespace: default

spec:

  selector:

    app: tomcat

    release: canary

  ports:

  - name: http

    port: 8080

    targetPort: 8080

  - name: ajp

    port: 8009

    targetPort: 8009

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: tomcat-deploy

spec:

  replicas: 3

  selector:

    matchLabels:

      app: tomcat

      release: canary

  template:

    metadata:

      labels:

        app: tomcat

        release: canary

    spec:

      containers:

      - name: tomcat

        image: tomcat:7-alpine

        ports:

        - name: httpd

          containerPort: 8080

        - name: ajp

          containerPort: 8009

[root@k8s-master ingress-nginx]# kubectl apply -f tomcat-deploy.yaml

service "tomcat" created

deployment.apps "tomcat-deploy" created

[root@k8s-master ingress-nginx]# kubectl get pod #等待pod状态就绪

将tomcat添加至ingress-nginx中

[root@k8s-master ingress-nginx]# cat ingress-tomcat.yaml

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  name: ingress-tomcat

  namespace: default

  annotations:

    kubernets.io/ingress.class: "nginx"

spec:

  rules:

  - host: tomcat.magedu.com

    http:

      paths:

      - path:

        backend:

          serviceName: tomcat

          servicePort: 8080

[root@k8s-master ingress-nginx]# kubectl apply -f ingress-tomcat.yaml

ingress.extensions "ingress-tomcat" created

添加域名解析及访问服务

3.下面我们对tomcat服务添加httpds服务

创建私有证书及secret

[root@k8s-master ingress-nginx]# openssl genrsa -out tls.key 2048

Generating RSA private key, 2048 bit long modulus

.......+++

..............................+++

e is 65537 (0x10001)

[root@k8s-master ingress-nginx]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.magedu.com #注意域名要和服务的域名一致

[root@k8s-master ingress-nginx]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key #创建secret

secret "tomcat-ingress-secret" created

[root@k8s-master ingress-nginx]# kubectl get secret

NAME                    TYPE                                  DATA      AGE

default-token-bf52l     kubernetes.io/service-account-token   3         9d

tomcat-ingress-secret   kubernetes.io/tls                     2         7s

[root@k8s-master ingress-nginx]# kubectl describe secret tomcat-ingress-secret

Name:         tomcat-ingress-secret

Namespace:    default

Labels:       <none>

Annotations:  <none>

Type:  kubernetes.io/tls

Data

====

tls.crt:  1294 bytes  #base64加密

tls.key:  1679 bytes

将证书应用至tomcat服务中

[root@k8s-master01 ingress]# cat ingress-tomcat-tls.yaml

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  name: ingress-tomcat-tls

  namespace: default

  annotations:

    kubernets.io/ingress.class: "nginx"

spec:

  tls:

  - hosts:

    - tomcat.magedu.com        #与secret证书的域名需要保持一致

    secretName: tomcat-ingress-secret   #secret证书的名称

  rules:

  - host: tomcat.magedu.com

    http:

      paths:

      - path:

        backend:

          serviceName: tomcat

          servicePort: 8080

[root@k8s-master01 ingress]#  kubectl apply -f ingress-tomcat-tls.yaml

访问服务

#参考文档：

https://www.jianshu.com/p/e30b06906b77

https://kubernetes.io/docs/concepts/services-networking/ingress/

https://kubernetes.github.io/ingress-nginx/deploy/

https://github.com/kubernetes/ingress-nginx/blob/nginx-0.20.0/docs/deploy/index.md