转自:https://medium.com/wish-engineering/katalog-sync-reliable-integration-of-consul-and-kubernetes-ebe8aae0852a

Why use consul with Kubernetes (k8s)?

Consul is a well-known and widely used service discovery mechanism. Here at Wish, we have standardized on using consul as our service discovery system for quite some time. Although k8s has a built-in service discovery mechanism, we want to continue our usage of consul as the primary service discovery mechanism. This way services in k8s are discoverable outside of k8s and aren’t tied to a specific cluster. Now when a service needs to ramp in and out of k8s they can do so gradually.

The previous solution: sidecar consul-agent

When we launched k8s we decided to add a consul-agent sidecar to each pod. In our k8s environment, each pod has a routable IP in our VPC and as such this functions pretty well. However, after having used this for several months we have noticed a few pain points:

  1. Configuration: Each service/namespace in k8s needs to have the same consul configuration (client configuration, encryption key, etc.). We largely dealt with this through jsonnet templating, but even so, we ended up having the encryption key in each namespace and a fair amount of configuration duplicated across services.
  2. Complexity: Using this sidecar approach means that we now have a full-blown consul node for each pod in k8s. For the initial migration to k8s things were generally moved over 1:1, EC2 instance → pod, but as we continued to refine our sizing etc. we ended up having significantly more pods than we had EC2 instances before. In addition, this means we effectively had N nodes participating in the consul memberlist running on the same instance or hardware.
  3. Failure modes: With a consul-agent sidecar on each pod we can run into thundering herd issues in consul failure modes due to the large number of nodes in the cluster.
  4. Noisy alerts: Consul’s memberlist expects members to be more-or-less long-lived. Deregistration of a node in the memberlist takes (by default) 72h which means that a node will still be part of the memberlist even after leaving intentionally. In practice, this is a nuisance as the node still shows up in consul’s service discovery until it drops off (e.g. Prometheus’ consul discovery).
  5. Consul checks vs k8s checks: Probably the most painful issue we’ve run into is configuring consul checks. K8s itself has concepts of liveliness and readiness which are used within k8s to manage the pods themselves. In addition to this k8s readiness, we also needed to configure consul so it would add/remove the service from rotation based on the pod’s readiness. Operationally this is painful to keep in-sync as consul and k8s offer different mechanisms for health checks.

Looking for alternatives: consul-k8s

At the end of last yearhashicorp announced consul-k8s as a mechanism to sync services to/from k8s and consul. We were excited to switch to a more k8s-native mechanism for syncing state to consul, and quickly started prototyping with it. Going into it we listed our requirements as:

  • Configuration through k8s annotations
  • Readiness sync
  • High availability with no single point of failure (SPOF)

The good

Consul-k8s offers mechanisms to sync both from k8s → consul and consul → k8s. We don’t have a need for consul → k8s, so we’ll focus on the k8s → consul sync. Consul-k8s sync is focused on syncing services from k8s → consul. This means that you can configure syncing etc. at the service-level in k8s through annotations. For example (borrowed from here):

kind: Service
apiVersion: v1
metadata:
name: my-service
annotations:
"consul.hashicorp.com/service-name": my-consul-service

This configuration-through-annotation both dramatically simplifies templating and is significantly easier to understand.

The bad

Unsurprisingly (since we are writing this post) we ran into some issues while testing out consul-k8s. Initially, we ran into some issues with multi-cluster support but those were resolved relatively quickly. After getting a proof of concept working with multi-cluster support we started some failure mode testing. During this testing, we found 2 major issues:

In addition to those issues, we found a requirement we didn’t know we had! With the sidecar consul-agent approach if the consul-agent was unable to join the cluster for some reason the pod would fail, and k8s would halt the deployment. Consul-k8s, however, is a single-process for the cluster which asynchronously syncs state from k8s to consul.

  1. Kubelet starts container on Node
  2. Kubelet updates k8s API
  3. Consul-k8s notices change in k8s-api
  4. Consul-k8s pushes change to consul

This means the ability of consul-k8s to sync the k8s state to consul is completely independent of the k8s pod deployments. This implies that we could easily create scenarios where the entire service would complete a rolling update (with new pod IPs, etc.) without that state being synced to consul. This means that we could get into a state where service discovery has 0 correct entries in it so clients would be unable to connect to the service

katalog-sync: Reliable Integration of Consul and Kubernetes的更多相关文章

  1. Announcing HashiCorp Consul + Kubernetes

    转自:https://www.hashicorp.com/blog/consul-plus-kubernetes We're excited to announce multiple features ...

  2. kubernetes实战之部署一个接近生产环境的consul集群

    系列目录 前面我们介绍了如何在windows单机以及如何基于docker部署consul集群,看起来也不是很复杂,然而如果想要把consul部署到kubernetes集群中并充分利用kubernete ...

  3. Building a Service Mesh with HAProxy and Consul

    转自:https://www.haproxy.com/blog/building-a-service-mesh-with-haproxy-and-consul/ HashiCorp added a s ...

  4. Spring Cloud 微服务一:Consul注册中心

    Consul介绍 Consul is a service mesh solution providing a full featured control plane with service disc ...

  5. docker及k8s安装consul

    一.docker部署consul集群 参考文献:https://www.cnblogs.com/lonelyxmas/p/10880717.html https://blog.csdn.net/qq_ ...

  6. 基于Docker的Consul集群实现服务发现

    服务发现 其实简单说,服务发现就是解耦服务与IP地址之间的硬绑定关系,以典型的集群为例,对于集群来说,是有多个节点的,这些节点对应多个IP(或者同一个IP的不同端口号),集群中不同节点责任是不一样的. ...

  7. 08-SpringCloud Consul

    Consul简介 官网 Consul下载地址 What is Consul? Consul is a service mesh solution providing a full featured c ...

  8. oracle_hc.sql

    select event,count(1) from gv$session group by event order by 2;exec dbms_workload_repository.create ...

  9. 多语言(Java、.NET、Node.js)混合架构下开源调用链追踪APM项目初步选型

    1. 背景 我们的技术栈包括了Java..NET.Node.js等,并且采用了分布式的技术架构,系统性能管理.问题排查成本越来越高. 2. 基本诉求 针对我们的情况,这里列出了选型的主要条件,作为最终 ...

随机推荐

  1. 协程(Coroutine)与多线程,多进程

    执行多个任务可以使用多线程或多进程. 多进程中,同一个变量,各自有一份拷贝存在于每个进程中,互不影响 多线程中,所有变量都由所有线程共享.而线程间的切换是系统进行调度,无法控制,所以可能 一个进程中的 ...

  2. VSTO:使用C#开发Excel、Word【9】

    文件背后的代码VSTO支持文档背后的代码,要求开发人员使用VSTO项目中生成的具有预连接上下文和预连接事件的类.这些类有时被称为“代码后面”类,因为它们是与特定文档或工作表相关联的代码.在Word中, ...

  3. python中各种遇到的函数

    函数:split() Python中有split()和os.path.split()两个函数,具体作用如下:split():拆分字符串.通过指定分隔符对字符串进行切片,并返回分割后的字符串列表(lis ...

  4. linux系统安装jdk详细配置

    1.通过指令 whereis java 查看是否已经配置jdk 如果已经安装,通过指令 rm -rf <jdk路径> 删除 2.通过ssh工具将jdk-8u11-linux-x64.tar ...

  5. PE文件 02 导出表

    0x01  导出表结构  导出表是由数据目录表中的第一个成员DataDirectory[0]指出的: typedef struct _IMAGE_DATA_DIRECTORY { DWORD Virt ...

  6. centos7 安装jdk8 bash脚本 并配置环境变量

    #!/bin/bash #安装java脚本 if type -p java; then echo 'java已安装.' exit else echo '开始安装java...' wget --no-c ...

  7. 了解下webpack的几个命令

    [ webpack3.0.0刚刚出来  所以文章是跟着低版本 教程 操作熟悉  结果好多对不上喔] 六:了解下webpack的几个命令 webpack         // 最基本的启动webpack ...

  8. Join Algorithm

    Join(SQL) An SQL join clause combines columns from one or more tables in a relational database. It c ...

  9. Android SurfaceView内容获取

    Android SurfaceView内容获取 这几天在做一个Android的小项目,项目中需要使用到SurfaceView来显示相机捕获的内容,同时还有一个SurfaceView用于绘制一些辅助线, ...

  10. JAVA数组与List相互转换

    1.数组转成List 数组转成List可以用方法 :Arrays.asList,一起来了解一下 System.out.println(Arrays.asList(new String[] { &quo ...