K8s ipvs mode kube-proxy

IPVS vs. IPTABLES

IPVS模式在Kubernetes 1.8中被引入，在1.9中进入beta测试。 IPTABLES模式在1.1版本中被添加进来，在1.2开始就变成了默认的操作模式。IPVS 和 IPTABLES都是基于netfilter实现的。

IPVS与IPTABLES有以下几点不同的地方:

IPVS为大规模集群提供更好的可扩展性和性能。
IPVS比IPTABLES支持更复杂的负载均衡算法 (least load, least connections, locality, weighted, etc.)。
IPVS支持服务器健康检查和重连等等。

配置kube-proxy ipvs模式

k8s版本：1.10

默认情况下，Kube-proxy依然使用iptables来实现Service到POD之间的负载均衡。

[root@node01 ~]# kubectl get svc -o wide

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE       SELECTOR

httpd-svc    ClusterIP   10.110.73.171   <none>        /TCP   1d        run=httpd

kubernetes   ClusterIP   10.96.0.1       <none>        /TCP    1d        <none>

[root@node01 ~]# iptables-save
.
.
.

-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE

-A KUBE-SEP-6E7XQMQ4RAYOWTTM -s 10.244.0.3/ -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ

-A KUBE-SEP-6E7XQMQ4RAYOWTTM -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.3:

-A KUBE-SEP-7SDFQQ3CPAUK3SXM -s 10.244.1.2/32 -m comment --comment "default/httpd-svc:" -j KUBE-MARK-MASQ

-A KUBE-SEP-7SDFQQ3CPAUK3SXM -p tcp -m comment --comment "default/httpd-svc:" -m tcp -j DNAT --to-destination 10.244.1.2:80

-A KUBE-SEP-FXPWU3QR6KR3AMXC -s 172.16.65.180/ -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ

-A KUBE-SEP-FXPWU3QR6KR3AMXC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-FXPWU3QR6KR3AMXC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.65.180:

-A KUBE-SEP-IT2ZTR26TO4XFPTO -s 10.244.0.2/ -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ

-A KUBE-SEP-IT2ZTR26TO4XFPTO -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.244.0.2:

-A KUBE-SEP-TOSIKZVXYIAJSFHP -s 10.244.2.2/32 -m comment --comment "default/httpd-svc:" -j KUBE-MARK-MASQ

-A KUBE-SEP-TOSIKZVXYIAJSFHP -p tcp -m comment --comment "default/httpd-svc:" -m tcp -j DNAT --to-destination 10.244.2.2:80

-A KUBE-SEP-UO4LFRT7VSL3X4EB -s 10.244.0.4/32 -m comment --comment "default/httpd-svc:" -j KUBE-MARK-MASQ

-A KUBE-SEP-UO4LFRT7VSL3X4EB -p tcp -m comment --comment "default/httpd-svc:" -m tcp -j DNAT --to-destination 10.244.0.4:80

-A KUBE-SEP-YIL6JZP7A3QYXJU2 -s 10.244.0.2/ -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ

-A KUBE-SEP-YIL6JZP7A3QYXJU2 -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.2:

-A KUBE-SEP-ZXMNUKOKXUTL2MK2 -s 10.244.0.3/ -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ

-A KUBE-SEP-ZXMNUKOKXUTL2MK2 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.244.0.3:

-A KUBE-SERVICES ! -s 10.244.0.0/ -d 10.96.0.1/ -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport  -j KUBE-MARK-MASQ

-A KUBE-SERVICES -d 10.96.0.1/ -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport  -j KUBE-SVC-NPX46M4PTMTKRN6Y

-A KUBE-SERVICES ! -s 10.244.0.0/ -d 10.96.0.10/ -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport  -j KUBE-MARK-MASQ

-A KUBE-SERVICES -d 10.96.0.10/ -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport  -j KUBE-SVC-TCOU7JCQXEZGVUNU

-A KUBE-SERVICES ! -s 10.244.0.0/ -d 10.96.0.10/ -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport  -j KUBE-MARK-MASQ

-A KUBE-SERVICES -d 10.96.0.10/ -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport  -j KUBE-SVC-ERIFXISQEP7F7OF4

-A KUBE-SERVICES ! -s 10.244.0.0/ -d 10.110.73.171/ -p tcp -m comment --comment "default/httpd-svc: cluster IP" -m tcp --dport  -j KUBE-MARK-MASQ

-A KUBE-SERVICES -d 10.110.73.171/ -p tcp -m comment --comment "default/httpd-svc: cluster IP" -m tcp --dport  -j KUBE-SVC-RL3JAE4GN7VOGDGP

-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS

-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-IT2ZTR26TO4XFPTO

-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-ZXMNUKOKXUTL2MK2

-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds  --reap --name KUBE-SEP-FXPWU3QR6KR3AMXC --mask 255.255.255.255 --rsource -j KUBE-SEP-FXPWU3QR6KR3AMXC

-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-FXPWU3QR6KR3AMXC

-A KUBE-SVC-RL3JAE4GN7VOGDGP -m comment --comment "default/httpd-svc:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-UO4LFRT7VSL3X4EB

-A KUBE-SVC-RL3JAE4GN7VOGDGP -m comment --comment "default/httpd-svc:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-7SDFQQ3CPAUK3SXM

-A KUBE-SVC-RL3JAE4GN7VOGDGP -m comment --comment "default/httpd-svc:" -j KUBE-SEP-TOSIKZVXYIAJSFHP

-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-YIL6JZP7A3QYXJU2

-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-6E7XQMQ4RAYOWTTM

.
.
.

第一步，在内核中加载ip_vs模块：

cat > /etc/sysconfig/modules/ipvs.modules <<EOF

#!/bin/bash

ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack_ipv4"

for kernel_module in \${ipvs_modules}; do

    /sbin/modinfo -F filename \${kernel_module} > /dev/null >&

    if [ $? -eq  ]; then

        /sbin/modprobe \${kernel_module}

    fi

done

EOF

chmod  /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs

输出结果应该为：

[root@node01 ~]# lsmod | grep ip_vs

ip_vs_ftp

ip_vs_sed

ip_vs_nq

ip_vs_sh

ip_vs_dh

ip_vs_lblcr

ip_vs_lblc

ip_vs_wrr

ip_vs_rr

ip_vs_wlc

ip_vs_lc

nf_nat                    ip_vs_ftp,nf_nat_ipv4,nf_nat_masquerade_ipv4

ip_vs                    ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_lblcr,ip_vs_lblc

nf_conntrack             ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4

libcrc32c                 xfs,ip_vs,nf_nat,nf_conntrack

第二步，安装ipvs管理工具ipvsadm

yum install -y ipvsadm

第三步，修改集群配置文件

在使用kubeadm init --config config.yaml初始化集群前，修改集群配置文件

Kubernetes v1. v1.

kind: MasterConfiguration

apiVersion: kubeadm.k8s.io/v1alpha1

...

kubeProxy:

  config:

    featureGates: SupportIPVSProxyMode=true

    mode: ipvs

...

Kubernetes v1.

kind: MasterConfiguration

apiVersion: kubeadm.k8s.io/v1alpha1

...

kubeProxy:

  config:

    featureGates:

      SupportIPVSProxyMode: true

    mode: ipvs

...

第四步，结果验证

[root@node01 ~]# ipvsadm -L -n

IP Virtual Server version 1.2. (size=)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  10.96.0.1: rr persistent

  -> 172.16.65.180:           Masq

TCP  10.96.0.10: rr

  -> 10.244.0.2:                Masq

  -> 10.244.0.3:                Masq

UDP  10.96.0.10: rr

  -> 10.244.0.2:                Masq

  -> 10.244.0.3:                Masq

第五步，测试

创建http-svc service

apiVersion: apps/v1beta1

kind: Deployment

metadata:

  name: httpd

spec:

  replicas:

  template:

    metadata:

      labels:

        run: httpd

    spec:

      containers:

      - name: httpd

        image: httpd

        ports:

        - containerPort: 

---

apiVersion: v1

kind: Service

metadata:

  name: httpd-svc

spec:

  selector:

    run: httpd
  type: NodePort

  ports:

  - protocol: TCP

    port:

    targetPort: 80
    nodePort: 30001

查看效果

[root@node01 yaml-store]# kubectl get pod -o wide

NAME                     READY     STATUS    RESTARTS   AGE       IP           NODE

httpd-749bf8c6f4-htzgl   /       Running             11m       10.244.2.2   node03

httpd-749bf8c6f4-vqwff   /       Running             11m       10.244.1.2   node02

httpd-749bf8c6f4-wkv9w   /       Running             11m       10.244.0.4   node01

[root@node01 yaml-store]# kubectl get svc -o wide

NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE       SELECTOR

httpd-svc    ClusterIP   10.99.129.90   <none>        8080:30001/TCP   11m       run=httpd

kubernetes   ClusterIP   10.96.0.1      <none>        /TCP    58m       <none>

[root@node01 yaml-store]# ipvsadm -L -n

IP Virtual Server version 1.2. (size=)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  172.16.65.181: rr

  -> 10.244.0.4:                Masq

  -> 10.244.1.2:                Masq

  -> 10.244.2.2:                Masq

TCP  172.17.0.1: rr

  -> 10.244.0.4:                Masq

  -> 10.244.1.2:                Masq

  -> 10.244.2.2:                Masq

TCP  10.96.0.1: rr persistent

  -> 172.16.65.180:           Masq

TCP  10.96.0.10: rr

  -> 10.244.0.2:                Masq

  -> 10.244.0.3:                Masq

TCP  10.99.129.90: rr

  -> 10.244.0.4:                Masq

  -> 10.244.1.2:                Masq

  -> 10.244.2.2:                Masq

TCP  10.244.0.0: rr

  -> 10.244.0.4:                Masq

  -> 10.244.1.2:                Masq

  -> 10.244.2.2:                Masq

TCP  10.244.0.1: rr

  -> 10.244.0.4:                Masq

  -> 10.244.1.2:                Masq

  -> 10.244.2.2:                Masq

TCP  127.0.0.1: rr

  -> 10.244.0.4:                Masq

  -> 10.244.1.2:                Masq

  -> 10.244.2.2:                Masq

UDP  10.96.0.10: rr

  -> 10.244.0.2:                Masq

  -> 10.244.0.3:                Masq

参考文章：

https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs

https://www.kubernetes.org.cn/3025.html

K8s ipvs mode kube-proxy的更多相关文章

k8s Ipvs 内部网络自动分配和内部网络一致ip地址，导致ip冲突
[链接]Linux负载均衡--LVS(IPVS)https://www.jianshu.com/p/36880b085265 [链接]kube-proxyipvsmodecreateunnecessa ...
kubernetes实战(十六)：k8s高可用集群平滑升级 v1.11.x 到v1.12.x
1.基本概念升级之后所有的containers会重启,因为hash值会变. 不可跨版本升级. 2.升级Master节点当前版本 [root@k8s-master01 ~]# kubeadm ver ...
k8s集群升级
集群升级由于课程中的集群版本是 v1.10.0,这个版本相对有点旧了,最新版本都已经 v1.14.x 了,为了尽量保证课程内容的更新度,所以我们需要将集群版本更新.我们的集群是使用的 kubeadm ...
k8s的kube-proxy
kube-proxy 运行在每个节点上,监听 API Server 中服务对象的变化,再通过管理 IPtables 来实现网络的转发. Kube-Proxy 不同的版本可支持三种工作模式: Use ...
kubeadm部署k8s
Kubernetes技术已经成为了原生云技术的事实标准,它是目前基础软件领域最为热门的分布式调度和管理平台.于是,Kubernetes也几乎成了时下开发工程师和运维工程师必备的技能之一. 官方文档 ...
1.k8s概述、安装、名词解释、资源清单
一.k8s概述 1.简介 Kubernetes是一个开源的,用于管理云平台中多个主机上的容器化的应用,Kubernetes的目标是让部署容器化的应用简单并且高效(powerful),Kubernete ...
k8s入坑之路（4）kubenetes安装
三种安装方法: 1.kubeadm 2.kubespray 3.二进制安装 kubespray安装kubernetes集群优点: 1.kuberspray对比kubeadm更加简洁内部集成了kube ...
2.k8s的架构
之前了解了k8s到底是什么,接下来看看k8s的组成. 一.Kubernetes架构学习k8s,最终目的是为了部署应用,部署一个完整的k8s, 就要知道k8s的组成.k8s主要包含两大部分: 中间包含 ...
基于Kubernetes/K8S构建Jenkins持续集成平台(上)-2
基于Kubernetes/K8S构建Jenkins持续集成平台(上)-2 Kubernetes实现Master-Slave分布式构建方案传统Jenkins的Master-Slave方案的缺陷 Mas ...

随机推荐

(转载)Unity3D研究院之使用 C#合成解析XML与JSON（四十一）
XML与JSON在开发中非常重要, 其实核心就是处理字符串.一个是XML的字符串一个是JSON的字符串,尤其是在处理网络请求的时候,肯定是要用的.另外现在JSON非常的流行,我写了一个简单的例子融合了 ...
Unity3D学习笔记——NGUI之UIButton
前言:用于接受点击,悬停,触摸等事件.UIButton还有重要的用途,就是改变控件不同状态下的颜色. 一:使用方式: 1.在UI Root中右键创建一个Sprite 2.为其添加一个碰撞组件,就添加B ...
可以这样创建E-Notebook数据库
最新版的ChemDraw的名称是ChemOffice Professional 15.用户朋友们在使用它的E-Notebook功能的时候,会发现需要先创立一个数据库,作为上传文档的储存空间.并且还可以 ...
UIAlertView及UIActionSheet 在ios8极其以下版本的兼容问题解决方案
本文转载至 http://www.aichengxu.com/view/35326 UIAlertView及UIActionSheet在ios8中被放弃,其功能将完全由UIAlertControlle ...
Win7 maven安装及配置
1. 前期准备 ① jdk 1.8 ② maven 3.5.4 下载地址:http://maven.apache.org/download.cgi 2. 配置maven环境变量 ① maven解压到指 ...
M²的经典语录
1. If you failed, stop and think! You should work in the correct way. 2. If I can do all of it, why ...
【BZOJ3073】[Pa2011]Journeys 线段树+堆优化Dijkstra
[BZOJ3073][Pa2011]Journeys Description Seter建造了一个很大的星球,他准备建造N个国家和无数双向道路.N个国家很快建造好了,用1..N编号,但是他发现道路实在 ...
spark2.0.2基于hadoop2.4搭建分布式集群
一.Scala安装因为spark的版本原因,所以Scala我用的2.11.7. 下载目录http://www.scala-lang.org/download/ 拷贝到要安装的地址,我的地址是/usr ...
取消select默认样式
/*清除select默认样式*/select { border: solid 1px #CACDD0; appearance:none; -moz-appearance:none; -webkit-a ...
Django报：AttributeError: tuple object has no attribute get
def index(request): hero_list=models.HeroInfo.objects.all() return render_to_response('index.html',{ ...