Pod和容器的LimitRange原理和实践总结

一、背景介绍

通常情况下，Pod中的容器可以无限制的使用节点上的CPU和内存资源，在共享资源和资源有限的情况下，若不加以限制，某个集群或命名空间的资源可能会消耗殆尽，导致其他节点上优先级低的Pod发生驱逐、重新调度、重启等问题，因此针对资源限额需要有针对命名空间级别的使用限制，以及针对Pod和容器级别的使用限制，LimitRange就是在后者的背景下诞生的。

二、应用场景

1、适用对集群的某个命名空间中的容器设置默认的统一的request和limit值，配合Resource Quotas更好的实现命名空间级别的资源限额。

2、适用对单个Pod的CPU和内存资源限额控制，如实现Pod中所有容器CPUlimit之和不得超过某值。

3、适用对单个容器的CPU和内存资源限额控制，如实现容器的CPUlimit值不得超过某值。

三、实践

一、LimitRange设置默认request和limit

适用场景：适用对集群的某个命名空间中的容器设置默认的统一的request和limit值

【新建limitrange】

LimitRange针对某个namespace的Pod或容器的CPU和内存的默认值、request和limit值实现管控，因此限制的的范围是在namespace下，在创建LimitRange之前，先创建1个用于测试的namespace

 kubectl create namespace limitrange-test-ns

[root@k8s-master limitrange]# kubectl get ns |grep limitrange-test-ns

limitrange-test-ns   Active   78m

namespace创建成功后，在此命名空间下创建limitrange

[root@k8s-master limitrange]# ls

limitrange.yaml

[root@k8s-master limitrange]# cat limitrange.yaml

apiVersion: v1

kind: LimitRange

metadata:

  name: mem-limit-range-test

  namespace: limitrange-test-ns   #指定命名空间

spec:

  limits:

  - default:

      memory: 256Mi

    defaultRequest:

      memory: 128Mi

    type: Container

kubectl apply -f limitrange.yaml

[root@k8s-master limitrange]# kubectl describe limitrange mem-limit-range-test -n limitrange-test-ns

Name:       mem-limit-range-test

Namespace:  limitrange-test-ns

Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio

----        --------  ---  ---  ---------------  -------------  -----------------------

Container   memory    -    -    128Mi            256Mi          -

[root@k8s-master limitrange]# kubectl describe ns limitrange-test-ns

Name:         limitrange-test-ns

Labels:       <none>

Annotations:  <none>

Status:       Active

No resource quota.

Resource Limits

 Type       Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio

 ----       --------  ---  ---  ---------------  -------------  -----------------------

 Container  memory    -    -    128Mi            256Mi          -

可以看到已为limitrange-test-ns的命名空间下所有的容器设置了默认的request值和limit值，接下来看下在容器中是如何生效的

【容器limitrange生效验证】

1、容器未设置limit，且未设置request

[root@k8s-master limitrange]# cat container-limitrange.yaml

apiVersion: v1

kind: Pod

metadata:

  name: pod-container-men-limit-test

spec:

  containers:

  - name: container-limlitrange-test

    image: nginx:latest

    imagePullPolicy: IfNotPresent

[root@k8s-master limitrange]# kubectl get pod -o wide -n limitrange-test-ns

NAME                           READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES

pod-container-men-limit-test   1/1     Running   0          28s   10.122.235.216   k8s-master   <none>           <none>

[root@k8s-master limitrange]# curl 10.122.235.216:80

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>

    body {

        width: 35em;

        margin: 0 auto;

        font-family: Tahoma, Verdana, Arial, sans-serif;

    }

</style>

</head>

<body>

<h1>Welcome to nginx!</h1>

<p>If you see this page, the nginx web server is successfully installed and

working. Further configuration is required.</p>

<p>For online documentation and support please refer to

<a href="http://nginx.org/">nginx.org</a>.<br/>

Commercial support is available at

<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

</body>

</html>

可以看到此Pod已经running，且业务正常，在创建此Pod容器部分并未设置默认值，但是Pod所在的命名空间有配置过默认的limitrange策略，验证下此策略是否生效到Pod的容器部分

spec:

  containers:

  - image: nginx:latest

    imagePullPolicy: IfNotPresent

    name: container-limlitrange-test

    resources:

      limits:

        memory: 256Mi

      requests:

        memory: 128Mi

总结：容器未设置limit，且未设置request，则容器运行起来后的request和limit值默认与limitrange type为container设置的request和limit的保持一致。

2、容器中设置limit，但是未设置request

[root@k8s-master limitrange]# cat container-limitrange.yaml

apiVersion: v1

kind: Pod

metadata:

  name: pod-container-men-limit-test

  namespace: limitrange-test-ns

spec:

  containers:

  - name: container-limlitrange-test

    image: nginx:latest

    imagePullPolicy: IfNotPresent

    resources:

      limits:

        memory: "500Mi"

只设置内存的limit值且大于limitrange设置的limit值

[root@k8s-master limitrange]# kubectl apply -f container-limitrange.yaml

pod/pod-container-men-limit-test created

[root@k8s-master limitrange]# kubectl get pod -n limitrange-test-ns

NAME                           READY   STATUS    RESTARTS   AGE

pod-container-men-limit-test   1/1     Running   0          9s

[root@k8s-master limitrange]# kubectl get pod pod-container-men-limit-test -o json -n limitrange-test-ns

"spec": {

        "containers": [

            {

                "image": "nginx:latest",

                "imagePullPolicy": "IfNotPresent",

                "name": "container-limlitrange-test",

                "resources": {

                    "limits": {

                        "memory": "500Mi"

                    },

                    "requests": {

                        "memory": "500Mi"

                    }

                },

                "terminationMessagePath": "/dev/termination-log",

                "terminationMessagePolicy": "File"

总结：容器中设置实际limit，但是未设置request，容器运行成功后request=实际limit

3、容器中未设置limit，但是设置request

[root@k8s-master limitrange]# cat container-limitrange.yaml

apiVersion: v1

kind: Pod

metadata:

  name: pod-container-men-limit-test

  namespace: limitrange-test-ns

spec:

  containers:

  - name: container-limlitrange-test

    image: nginx:latest

    imagePullPolicy: IfNotPresent

    resources:

      requests:

        memory: "150Mi"

[root@k8s-master limitrange]# kubectl apply -f container-limitrange.yaml

pod/pod-container-men-limit-test created

[root@k8s-master limitrange]# kubectl get pod pod-container-men-limit-test -o json -n limitrange-test-ns

"spec": {

        "containers": [

            {

                "image": "nginx:latest",

                "imagePullPolicy": "IfNotPresent",

                "name": "container-limlitrange-test",

                "resources": {

                    "limits": {

                        "memory": "256Mi"

                    },

                    "requests": {

                        "memory": "150Mi"

                    }

                },

                "terminationMessagePath": "/dev/termination-log",

                "terminationMessagePolicy": "File",

总结：容器中未设置limit，但是设置实际request，则容器运行起来后request=实际request，limit=limitrange.limit

二、Pod和容器的max、min、maxLimitRequestRatio

适用场景：控制容器或Pod的CPU或内存资源使用，如容器的CPU的request不得小于多少，limit不得大于多少，不满足条件则创建失败，先介绍下参数：

容器部分：
1、defaultRequest：容器默认的request值，若不指定request值，则默认为此值

2、max：容器的实际设置的limit值应小于等于此值

3、min：容器实际设置的request值应大于等于此值

4、maxLimitRequestRatio：Max Limit/Request Ratio，为容器CPU或内存的Limit/Request值应小于等于此值

Pod部分：

1、max：Pod中所有容器的实际设置的limit之和值应小于等于此值

2、min：Pod中所有容器实际设置的request值之和应大于等于此值

3、maxLimitRequestRatio：Max Limit/Request Ratio，为Pod中所有容器CPU或内存的Limit之和/Request之和值应小于等于此值

演示如下：

 kubectl create namespace limitrange-02-test-ns

[root@k8s-master limitrange]# cat limitrange.yaml

apiVersion: v1

kind: LimitRange

metadata:

  name: limitrange-test

  namespace: limitrange-02-test-ns

spec:

 limits:

 - max:

    cpu: 2000m

    memory: 1000Mi

   min:

    cpu: 200m

    memory: 6Mi

   maxLimitRequestRatio:

    cpu: 3

    memory: 2

   type: Pod

 - default:                      #容器的默认limit值

    cpu: 300m

    memory: 200Mi

   defaultRequest:               #容器的默认request值

    cpu: 200m

    memory: 100Mi

   max:                          #容器的limit值不得大于max

    cpu: 1000m

    memory: 500Mi

   min:                          #容器的request的值不得小于min

    cpu: 100m

    memory: 3Mi

   maxLimitRequestRatio:         #容器的limit/request不得大于，如容器的CPU的limit/request不得大于4

    cpu: 5

    memory: 4

   type: Container

 kubectl apply -f container-02-limitrange.yaml

[root@k8s-master limitrange]# kubectl describe ns limitrange-02-test-ns

Name:         limitrange-02-test-ns

Labels:       <none>

Annotations:  <none>

Status:       Active

No resource quota.

Resource Limits

 Type       Resource  Min   Max     Default Request  Default Limit  Max Limit/Request Ratio

 ----       --------  ---   ---     ---------------  -------------  -----------------------

 Pod        cpu       200m  2       -                -              3

 Pod        memory    6Mi   1000Mi  -                -              2

 Container  cpu       100m  1       200m             300m           5

 Container  memory    3Mi   500Mi   100Mi            200Mi          4

可以看到设置的Pod和容器设置的max和min已经生效了，接下来验证下设置的策略是否生效，创建1个Pod的容器的内存limit值大于500Mi

[root@k8s-master limitrange]# cat container-02-limitrange.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-limitrange-test
namespace: limitrange-02-test-ns
spec:
containers:
- name: container-limlitrange-test
image: nginx:latest
imagePullPolicy: IfNotPresent
resources:
requests:
memory: "100Mi"
limits:
memory: "600Mi"

[root@k8s-master limitrange]# kubectl apply -f container-02-limitrange.yaml
Error from server (Forbidden): error when creating "container-02-limitrange.yaml": pods "pod-limitrange-test" is forbidden: [memory max limit to request ratio per Pod is 2, but provided ratio is 6.000000, maximum memory usage per Container is 500Mi, but limit is 600Mi, memory max limit to request ratio per Container is 4, but provided ratio is 6.000000]

提示容器的limit/request值为6大于4，且limit为600Mi大于设置最大max值500Mi，因此创建失败，在Pod只有1个容器的情况下podlimit/request值为6大于2，同样做下request值小于min值的测试

[root@k8s-master limitrange]# cat container-02-limitrange.yaml

apiVersion: v1

kind: Pod

metadata:

  name: pod-limitrange-test

  namespace: limitrange-02-test-ns

spec:

  containers:

  - name: container-limlitrange-test

    image: nginx:latest

    imagePullPolicy: IfNotPresent

    resources:

      requests:

        memory: "1Mi"    #小于min值

      limits:

        memory: "200Mi

[root@k8s-master limitrange]# kubectl apply -f container-02-limitrange.yaml

Error from server (Forbidden): error when creating "container-02-limitrange.yaml": pods "pod-limitrange-test" is forbidden: [minimum memory usage per Pod is 6Mi, but request is 1048576, memory max limit to request ratio per Pod is 2, but provided ratio is 200.000000, minimum memory usage per Container is 3Mi, but request is 1Mi, memory max limit to request ratio per Container is 4, but provided ratio is 200.000000]

提示容器request的值为1Mi小于min值3Mi，且limit/request=200大于4，同样不满足Pod的min为6和ratio，因此创建失败。

四、总结

本文描述了LimitRange的诞生背景、适用场景、以及具体如何实现和控制Pod、容器的资源限额，产品设计上主要考虑如下几点：

1、产品设计上需要方便用户低门槛快速设置默认request、limit值，配合Resource Quotas提供强大而稳定的K8S原生资源限额能力。

2、max、min、Limit/Request Ratio等功能如果开放给对K8S不了解的用户，门槛较高，若无意配置完成且生效，可能会导致容器的创建异常，且不知如何解决，是否直接开放需要认真考虑。