HAProxy、Keepalived 在 Ocatvia 的应用实现与分析

目录

文章目录

• 目录
• Amphora
• 启动 keepalived
• 启动 haproxy
• 最后

Amphora

创建一个 loadbalancer 需要占用一到两台 Amphora Instance 作为 “负载均衡器“ 的运行载体，实际提供高可用负载均衡底层支撑是 HAProxy & Keepalived。

• HAProxy：L4-L7 负载均衡器
• Keepalived：Linux 体系的高可用解决方案

不过 Amphora 并非是一开始就运行着 haproxy 和 keepalived 服务进程的，而是在需要运行它们的时候才会被 amphora-agent 启动。

启动 keepalived

keepalived 服务进程在 Amphora 被 loadbalancer 纳管后启动，TASK:AmphoraVRRPStart 就是启动服务的逻辑实现，而且从 UML 图可见，只有当 loadbalancer_topology = ACTIVE_STANDBY 时才会加载 keepalived，提供高可用服务。

# file: /opt/rocky/octavia/octavia/controller/worker/tasks/amphora_driver_tasks.py

class AmphoraVRRPStart(BaseAmphoraTask):
    """Task to start keepalived of all amphorae of a LB."""

    def execute(self, loadbalancer):
        self.amphora_driver.start_vrrp_service(loadbalancer)
        LOG.debug("Started VRRP of loadbalancer %s amphorae",
                  loadbalancer.id)

进过一系列调用后最终由 AmphoraAPIClient 发出 PUT vrrp/start 请求到 amphora-agent 的 view_func:manage_service_vrrp 接收处理。

# file: /opt/rocky/octavia/octavia/amphorae/backends/agent/api_server/keepalived.py

    def manager_keepalived_service(self, action):
        action = action.lower()
        if action not in [consts.AMP_ACTION_START,
                          consts.AMP_ACTION_STOP,
                          consts.AMP_ACTION_RELOAD]:
            return webob.Response(json=dict(
                message='Invalid Request',
                details="Unknown action: {0}".format(action)), status=400)

        if action == consts.AMP_ACTION_START:
            keepalived_pid_path = util.keepalived_pid_path()
            try:
                # Is there a pid file for keepalived?
                with open(keepalived_pid_path, 'r') as pid_file:
                    pid = int(pid_file.readline())
                os.kill(pid, 0)

                # If we got here, it means the keepalived process is running.
                # We should reload it instead of trying to start it again.
                action = consts.AMP_ACTION_RELOAD
            except (IOError, OSError):
                pass

        cmd = ("/usr/sbin/service octavia-keepalived {action}".format(
            action=action))

        try:
            subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
        except subprocess.CalledProcessError as e:
            LOG.debug('Failed to %s octavia-keepalived service: %s %s',
                      action, e, e.output)
            return webob.Response(json=dict(
                message="Failed to {0} octavia-keepalived service".format(
                    action), details=e.output), status=500)

        return webob.Response(
            json=dict(message='OK',
                      details='keepalived {action}ed'.format(action=action)),
            status=202)

上述代码可知，Amphora 中的 amphora-agent 是通过执行 CLI /usr/sbin/service octavia-keepalived start 来启动 keepalived 的。

# file: /usr/lib/systemd/system/octavia-keepalived.service

[Unit]
Description=Keepalive Daemon (LVS and VRRP)
After=network-online.target .service
Wants=network-online.target
Requires=.service

[Service]
# Force context as we start keepalived under "ip netns exec"
SELinuxContext=system_u:system_r:keepalived_t:s0
Type=forking
KillMode=process

ExecStart=/sbin/ip netns exec amphora-haproxy /usr/sbin/keepalived  -D -d -f /var/lib/octavia/vrrp/octavia-keepalived.conf -p /var/lib/octavia/vrrp/octavia-keepalived.pid

ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/var/lib/octavia/vrrp/octavia-keepalived.pid

[Install]
WantedBy=multi-user.target

octavia-keepalived.service 定义了 keepalived 的启动脚本、配置文件以 PID 文件的路径。配置文件的内容如下：

# file: /var/lib/octavia/vrrp/octavia-keepalived.conf

vrrp_script check_script {
  script /var/lib/octavia/vrrp/check_script.sh
  interval 5
  fall 2
  rise 2
}

vrrp_instance 01197be798d5440da846cd70f52dc503 { # VRRP instance name is loadbalancer UUID
  state MASTER                                   # Master router
  interface eth1                                 # VRRP IP device
  virtual_router_id 1                            # VRID
  priority 100
  nopreempt
  garp_master_refresh 5
  garp_master_refresh_repeat 2
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass b76d77e
  }

  unicast_src_ip 172.16.1.3                      # VRRP IP
  unicast_peer {
    172.16.1.7                                   # Backup router VRRP IP
  }

  virtual_ipaddress {
    172.16.1.10                                  # VIP address
  }
  track_script {
    check_script
  }
}

从配置文件可知 keepalived 使用 NIC eth1 作为 VRRP IP 和 VIP 的 interface，但是直接在 Amphora 执行 ifconfig 是看不见 eth1 的。因为 Amphora 将 VIP 设置到了 namespace amphora-haproxy 中：

root@amphora-cd444019-ce8f-4f89-be6b-0edf76f41b77:~# ip netns
amphora-haproxy
root@amphora-cd444019-ce8f-4f89-be6b-0edf76f41b77:~# ip netns exec amphora-haproxy bash
root@amphora-cd444019-ce8f-4f89-be6b-0edf76f41b77:~# ifconfig
eth1      Link encap:Ethernet  HWaddr fa:16:3e:f4:69:4b
          inet addr:172.16.1.3  Bcast:172.16.1.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fef4:694b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

eth1:0    Link encap:Ethernet  HWaddr fa:16:3e:f4:69:4b
          inet addr:172.16.1.10  Bcast:172.16.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1

除了 start 之外，view_func:manage_service_vrrp 还支持 stop 和 reload 操作。而 keepalived 配置文件的更新则交由 view_func:upload_keepalived_config 来完成。

而在 octavia-worker 端，更新 keepalived 配置文件的逻辑实现在 Task:AmphoraVRRPUpdate，AmphoraVRRPUpdate 在 AmphoraVRRPStart 之前执行，配置文件通渲染 Jinja 模板的方式生成。

启动 haproxy

我们知道 HAProxy 负责监听 frontend 的请求，然后根据不同的条件和 ACL 规则将请求分发到 backend，这一特性正是 Octavia Listener 对象的定义。所以，当为 loadbalancer 创建 listener 时才会启动 haproxy 服务进程。

从 UML 可知，执行指令 openstack loadbalancer listener create --protocol HTTP --protocol-port 8080 lb-1 创建 Listener 时会执行到 Task:ListenersUpdate，由 ListenersUpdate 完成了 haproxy 配置文件的 Upload 和 haproxy 服务进程的 Reload。

配置文件 /var/lib/octavia/1385d3c4-615e-4a92-aea1-c4fa51a75557/haproxy.cfg，其中 1385d3c4-615e-4a92-aea1-c4fa51a75557 为 Listener UUID：

# Configuration for loadbalancer 01197be7-98d5-440d-a846-cd70f52dc503
global
    daemon
    user nobody
    log /dev/log local0
    log /dev/log local1 notice
    stats socket /var/lib/octavia/1385d3c4-615e-4a92-aea1-c4fa51a75557.sock mode 0666 level user
    maxconn 1000000

defaults
    log global
    retries 3
    option redispatch

peers 1385d3c4615e4a92aea1c4fa51a75557_peers
    peer l_Ustq0qE-h-_Q1dlXLXBAiWR8U 172.16.1.7:1025
    peer O08zAgUhIv9TEXhyYZf2iHdxOkA 172.16.1.3:1025

frontend 1385d3c4-615e-4a92-aea1-c4fa51a75557
    option httplog
    maxconn 1000000
    bind 172.16.1.10:8080
    mode http
    timeout client 50000

因为此时的 Listener 只指定了监听的协议和端口，所以 frontend section 也设置了相应的 bind 172.16.1.10:8080 和 mode http。

服务进程：systemctl status haproxy-1385d3c4-615e-4a92-aea1-c4fa51a75557.service 其中 1385d3c4-615e-4a92-aea1-c4fa51a75557 为 Listener UUID：

# file: /usr/lib/systemd/system/haproxy-1385d3c4-615e-4a92-aea1-c4fa51a75557.service

[Unit]
Description=HAProxy Load Balancer
After=network.target syslog.service amphora-netns.service
Before=octavia-keepalived.service
Wants=syslog.service
Requires=amphora-netns.service

[Service]
# Force context as we start haproxy under "ip netns exec"
SELinuxContext=system_u:system_r:haproxy_t:s0

Environment="CONFIG=/var/lib/octavia/1385d3c4-615e-4a92-aea1-c4fa51a75557/haproxy.cfg" "USERCONFIG=/var/lib/octavia/haproxy-default-user-group.conf" "PIDFILE=/var/lib/octavia/1385d3c4-615e-4a92-aea1-c4fa51a75557/1385d3c4-615e-4a92-aea1-c4fa51a75557.pid"

ExecStartPre=/usr/sbin/haproxy -f $CONFIG -f $USERCONFIG -c -q -L O08zAgUhIv9TEXhyYZf2iHdxOkA

ExecReload=/usr/sbin/haproxy -c -f $CONFIG -f $USERCONFIG -L O08zAgUhIv9TEXhyYZf2iHdxOkA
ExecReload=/bin/kill -USR2 $MAINPID

ExecStart=/sbin/ip netns exec amphora-haproxy /usr/sbin/haproxy-systemd-wrapper -f $CONFIG -f $USERCONFIG -p $PIDFILE -L O08zAgUhIv9TEXhyYZf2iHdxOkA

KillMode=mixed
Restart=always
LimitNOFILE=2097152

[Install]
WantedBy=multi-user.target

从服务进程配置可以看出实际启动的服务为 /usr/sbin/haproxy-systemd-wrapper，它是运行在 namespace amphora-haproxy 中的，该脚本做的事情可以从日志了解到：

Nov 15 10:12:01 amphora-cd444019-ce8f-4f89-be6b-0edf76f41b77 ip[13206]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /var/lib/octavia/1385d3c4-615e-4a92-aea1-c4fa51a75557/haproxy.cfg -f /var/lib/octavia/haproxy-default-user-group.conf -p /var/lib/octavia/1385d3c4-615e-4a92-aea1-c4fa51a75557/1385d3c4-615e-4a92-aea1-c4fa51a75557.pid -L O08zAgUhIv9TEXhyYZf2iHdxOkA -Ds

就是调用了 /usr/sbin/haproxy 指令而已。

最后

本篇介绍了 Octavia 是如何将 HAProxy、Keepalived 等常用的负载均衡解决方案封装到 Amphora Instance 的，同时也介绍了 Create Listener 所需要处理的事情。需要注意的是 HAProxy 的配置文件会随着 Listener、Pool、Member、L7policy、L7rule、health-monitor 等对象的变更而变更，这些我们以后再作讨论。还有一点补充的就是创建 Listener 会执行 Task:UpdateVIP，这是因为 Lisenter 含有的协议及端口信息都需要被更新到 VIP 的安全组规则中，否则 Listener 要如何监听得到传输层的数据包呢？