[转帖]pod容器开启pid限制

https://zhdya.gitee.io/zhdya/archives/

 

cgroup中对pid进行了隔离，通过更改docker/kubelet配置，可以限制pid总数，从而达到限制线程总数的目的。线程数限制与系统中多处配置有关，取最小值，参考stackoverflow上线程数的设置

• docker，容器启动时设置 --pids-limit 参数，限制容器级别pid总数
• kubelet，开启SupportPodPidsLimit特性，设置–pod-max-pids参数，限制node每个pod的pid总数

以kubelet为例，开启SupportPodPidsLimit，--feature-gates=SupportPodPidsLimit=true

1、配置kubelet，每个pod允许最大pid数目为150

[root@node01 ~]# ps -ef |grep kubelet
root     18735     1 14 11:19 ?        00:53:28 ./kubelet --v=1 --address=0.0.0.0 --feature-gates=SupportPodPidsLimit=true --pod-max-pids=150 --allow-privileged=true --root-dir=/home/kubelet --node-status-update-frequency=5s --kubeconfig=/home/xbox/kubelet/conf/kubelet-kubeconfig --fail-swap-on=false --max-pods=254 --runtime-cgroups=/systemd/system.slice/frigga.service --kubelet-cgroups=/systemd/system.slice/frigga.service --make-iptables-util-chains=falseCopy

现状：

[root@k8s-10 system]# ps aux | grep kubelet | grep feature
root      4054  8.3  1.3 2551816 108444 ?      Ssl  10:36   1:21 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --allowed-unsafe-sysctls=net.* --logtostderr=false --log-dir=/data/logs/kubernetes/kubelet --v=2--feature-gates=RotateKubeletClientCertificate=true --cert-dir=/var/lib/kubelet/pki --rotate-certificatesCopy

kubeadm/kubectl/kube-apiserver turn on feature gate

配置文件：

[root@k8s-10 system]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

[root@k8s-10 system]# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --allowed-unsafe-sysctls='net.*' --logtostderr=false --log-dir=/data/logs/kubernetes/kubelet --v=2 --feature-gates=RotateKubeletClientCertificate=true  --cert-dir=/var/lib/kubelet/pki --rotate-certificates

##增加：
--feature-gates=SupportPodPidsLimit=true --pod-max-pids=150

##如下：
[root@k8s-10 system]# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --allowed-unsafe-sysctls='net.*' --logtostderr=false --log-dir=/data/logs/kubernetes/kubelet --v=2 --feature-gates=RotateKubeletClientCertificate=true --feature-gates=SupportPodPidsLimit=true --pod-max-pids=150 --cert-dir=/var/lib/kubelet/pki --rotate-certificates

##验证
[root@k8s-10 system]# ps aux | grep kubelet | grep feature
root     14540 40.0  1.2 1092992 101260 ?      Ssl  10:53   0:01 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --allowed-unsafe-sysctls=net.* --logtostderr=false --log-dir=/data/logs/kubernetes/kubelet --v=2--feature-gates=RotateKubeletClientCertificate=true --feature-gates=SupportPodPidsLimit=true --pod-max-pids=150 --cert-dir=/var/lib/kubelet/pki --rotate-certificatesCopy

在pod中起测试线程，root下起160个线程

for i in $(seq 1 160);do
sleep 1000 &
doneCopy

在cgroup中查看，pids达到最大限制

[root@node01 ~]# cat /sys/fs/cgroup/pids/kubepods/burstable/pod34ca46b7-f702-11ea-a6b8-005056a092a5/pids.current
149