Octavia 的 HTTPS 与自建、签发 CA 证书

目录

文章目录

• 目录
• Octavia 为什么需要自建 CA 证书？
• GenerateServerPEMTask
• CertComputeCreate
• Amphora Agent
• AmphoraAPIClient
• 最后

Octavia 为什么需要自建 CA 证书？

Note: For production use the ca issuing the client certificate and the ca issuing the server certificate need to be different so a hacker can’t just use the server certificate from a compromised amphora to control all the others.

可以想象，如果 octavia-worker、amphora 通信使用的证书与 User、Dashboard 通信使用的证书相同，那么 User 进出 amphora 就如入无人之地。简而言之，Octavia 自建 CA 证书是为了防止恶意用户登录并利用 amphora 作为 “肉鸡” 来攻击 OpenStack 的内部网络。

Octavia 提供了脚本 octavia/bin/create_certificates.sh 和配置文件 octavia/etc/certificates/openssl.cnf，只需执行下述指令即可完成自建 CA。

$ source /opt/rocky/octavia/bin/create_certificates.sh /etc/octavia/certs/ /opt/rocky/octavia/etc/certificates/openssl.cnf

$ ll /etc/octavia/certs/
total 44
-rw-r--r-- 1 stack stack 1294 Oct 26 12:51 ca_01.pem
-rw-r--r-- 1 stack stack  989 Oct 26 12:51 client.csr
-rw-r--r-- 1 stack stack 1708 Oct 26 12:51 client.key
-rw-r--r-- 1 stack stack 4405 Oct 26 12:51 client-.pem
-rw-r--r-- 1 stack stack 6113 Oct 26 12:51 client.pem
-rw-r--r-- 1 stack stack   71 Oct 26 12:51 index.txt
-rw-r--r-- 1 stack stack   21 Oct 26 12:51 index.txt.attr
-rw-r--r-- 1 stack stack    0 Oct 26 12:51 index.txt.old
drwxr-xr-x 2 stack stack   20 Oct 26 12:51 newcerts
drwx------ 2 stack stack   23 Oct 26 12:51 private
-rw-r--r-- 1 stack stack    3 Oct 26 12:51 serial
-rw-r--r-- 1 stack stack    3 Oct 26 12:51 serial.old

• ca_01.pem：CA 证书文件
• client.csr：Server CSR 证书签名请求文件
• client.key：Server 私钥文件
• client-.pem：PEM 编码的 Server 证书文件
• client.pem：结合了 client-.pem 和 client.key 的文件

与 CA 认证相关的主要有以下配置：

[certificates]
ca_private_key_passphrase = foobar
ca_private_key = /etc/octavia/certs/private/cakey.pem
ca_certificate = /etc/octavia/certs/ca_01.pem

[haproxy_amphora]
server_ca = /etc/octavia/certs/ca_01.pem
client_cert = /etc/octavia/certs/client.pem

[controller_worker]
client_ca = /etc/octavia/certs/ca_01.pem

• [haproxy_amphora] section 在 octavia-worker AmphoraAPIClient 被应用，AmphoraAPIClient 拿着 client.pem（包含 Server 证书、Server 私钥）和 CA 公钥向 amphora-agent 发起 SSL 通信。

• [certificates] section 应用在 create amphora flow 的 GenerateServerPEMTask 中，指定 CA 中心为 amphora 签署服务端证书。

• [controller_worker] 的 client_ca 在 Task:CertComputeCreate 中被应用，指定了 CA 证书路径。

下面主要围绕这 3 个 section 来进行分析。

GenerateServerPEMTask

从 UML 图可以看出当 octavia-worker 需要使用 REST 方式与 amphora-agent 通信时 create_amphora_flow 才会加载 GenerateServerPEMTask 用于签发一张 PEM 编码的 Amphora 证书用于建立 HTTPS 通信。

class GenerateServerPEMTask(BaseCertTask):
    """Create the server certs for the agent comm

    Use the amphora_id for the CN
    """

    def execute(self, amphora_id):
        cert = self.cert_generator.generate_cert_key_pair(
            cn=amphora_id,
            validity=CERT_VALIDITY)

        return cert.certificate + cert.private_key

octavia.certificates 实现了 local_cert_generator(default) 和 anchor_cert_generator 两种证书生成器，通过配置 [certificates] cert_generator 选定，这里以 local_cert_generator 为例：

# file: /opt/rocky/octavia/octavia/certificates/generator/local.py

    @classmethod
    def generate_cert_key_pair(cls, cn, validity, bit_length=2048,
                               passphrase=None, **kwargs):
        pk = cls._generate_private_key(bit_length, passphrase)
        csr = cls._generate_csr(cn, pk, passphrase)
        cert = cls.sign_cert(csr, validity, **kwargs)
        cert_object = local_common.LocalCert(
            certificate=cert,
            private_key=pk,
            private_key_passphrase=passphrase
        )
        return cert_object

method generate_cert_key_pair 的语义为：

1. 生成 Amphora 私钥
2. 生成 Amphora 证书签名请求（CSR）
3. 向 CA 中心申请签署 Amphora 证书

区别于 create_certificates.sh 脚本的 openssl 指令，octavia-worker 使用了内建的 cryptography 库来实现。GenerateServerPEMTask 的是 Amphora 的 server.pem，包含了 Amphora 证书（公钥）和 Amphora 私钥。

(Pdb) cert
<octavia.certificates.common.local.LocalCert object at 0x7ff1b879ee50>
(Pdb) cert.certificate + cert.private_key
'-----BEGIN CERTIFICATE-----\nMIIDiTCCAnGgAwIBAgIRAO8gS0SKikCiuMCRUiKkLYUwDQYJKoZIhvcNAQELBQAw\nXDELMAkGA1UEBhMCVVMxDzANBgNVBAgMBkRlbmlhbDEUMBIGA1UEBwwLU3ByaW5n\nZmllbGQxDDAKBgNVBAoMA0RpczEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tMB4X\nDTE4MTExNDA2MjY0NVoXDTIwMTExMzA2MjY0NVowLzEtMCsGA1UEAwwkZjQ5M2M5\nMDQtZWFhMy00NTljLThjNzctMTU2OGYyOWMwYzI3MIIBIjANBgkqhkiG9w0BAQEF\nAAOCAQ8AMIIBCgKCAQEA2spT4jGxt1nrfRwFburC2ErkdPaO3KE15ndoc17Blw+h\ngtLKhMhqCGGmWx7h9XsLV98JtglYuJDwh2vI7iToqN22izIhyUfef9j6asgUDs4K\nAB8502uPPeVwxmB60KeZQzES2PxgxpngUeMktpMv90/YIyJyEPgeEKj4/41C8Pr0\nMwvQVfJD28O2A/S5GgJWWjSgZA3vvEVflyhe0nnCWdJNwSFSzoR68IAF8RCBOm2e\n0UAYS/p9nabvkCPxpO8fhNbqzg/NCwIb+Vi06mC3L0tzyApI+Oks1Jd7uMM4O/BV\nvzD68Me/KbirYr6JEUHYAlIvyQ51ljAC0nWWJviW1wIDAQABo3MwcTAMBgNVHRMB\nAf8EAjAAMC8GA1UdEQQoMCaCJGY0OTNjOTA0LWVhYTMtNDU5Yy04Yzc3LTE1Njhm\nMjljMGMyNzAOBgNVHQ8BAf8EBAMCA7gwIAYDVR0lAQH/BBYwFAYIKwYBBQUHAwEG\nCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAgP+yoiW/Otfl9Sx/VBh2Ehw0H\n+82rPT7Yvs8UfnOv/AqDiUzOZMbmJQNfwMgzwY9l7yY4h0DMQsTortYHCuOMTaWB\nIsPRCCZeFtSxgYnHjW8Ggw1bEHNUWlkazENzA/2fKNnAkj/ddcr26TOADTGi4wB1\nVqvU51+UckA+Qn19fDdmlYWhRDhh3ye1wgQNZmIGiBoFuwJpQBBYST9AS+xcr80g\ntCgThHgjqTMr0I3K1Dx/zMA+/eTIWb4e8T7jcd0iW7fTFtux5/NVjcTWjcAFvTAd\nZ9mNB5DTKrc3V61AybpIyZeI545Had0GM+SdHXNzFI2tCtgTvT/ZEAsqRiRv\n-----END CERTIFICATE-----\n-----BEGIN RSA PRIVATE KEY-----\nMIIEpQIBAAKCAQEA2spT4jGxt1nrfRwFburC2ErkdPaO3KE15ndoc17Blw+hgtLK\nhMhqCGGmWx7h9XsLV98JtglYuJDwh2vI7iToqN22izIhyUfef9j6asgUDs4KAB85\n02uPPeVwxmB60KeZQzES2PxgxpngUeMktpMv90/YIyJyEPgeEKj4/41C8Pr0MwvQ\nVfJD28O2A/S5GgJWWjSgZA3vvEVflyhe0nnCWdJNwSFSzoR68IAF8RCBOm2e0UAY\nS/p9nabvkCPxpO8fhNbqzg/NCwIb+Vi06mC3L0tzyApI+Oks1Jd7uMM4O/BVvzD6\n8Me/KbirYr6JEUHYAlIvyQ51ljAC0nWWJviW1wIDAQABAoIBAEAJpE+6V9fgm8p8\nnyJ92BXSpdeOKvZswQf5vzq1a1g5nP5bkCcZOd/GJRjaiyx8nS9U+tSrG6q50Yzx\ngVgiuW5jpoBLZhQx0u/8pB8I/MXwjIDIovY8rypgs4d8ybW0uGkwPeIAzJqUg1G0\neBRwNEPgvNRbyqMo3DPoISk7QXKilmk/h5em1KmodBU9YtgIqi8F+Rs60csOPo5Z\nX70YGq5dICN1xIqW8eqzvbVO4JF/oU7JqhEyfcG6S029Ilj29bAPCRLEMZIUxeP5\ncMjGVP59PCt1z42UEnaxLEmzIpD2vVq06xEC3/0BvLLGTY6FOfzXjeEt3i/fdcUq\nLiNIejECgYEA/QxXwmPu9c9v4uUqhMcG6AUJd1Hlrfq/gJIaNRGsp/UtK4qtPMgw\n6GTv9Sv1D4Vuad65recdrGXYgAPWj1TRbIqimBE0Yu0z5vuAfScu7yrrZsWZXHJO\nVIP2310ZPnJFXwRSCA1VpIM8MoKfoAfpAMfDr7PM0gEjDapPpFLsl4UCgYEA3Veu\n8fTqqgxCWFe0IRZRiRT5FciRupBvuOVmYMIqIGtSD/l21dOQ0rd5o5ZBZsInO80y\ndXWk6YEni6dqSjxYwlaTHfG3ZyZlSEeaSHEhGiLvItxaMh6DJGcbTKUgkDiHFf5K\nW/KpJ91C4ByykHZmGYesz/z4Z8vadjPrf8gpLasCgYEAoJHedjlXfp88jiuAyXRJ\ni5z2nsJXDgkYz4rmGlq2xnUrTn/W4cTeU/kI0vgrrseqgn+ULyeCisytjr3gvl7B\n7TAjcH8qUMPXtXBN3hypCZagfTxRznmx/qsmUiIPTLLSFjL1oqpjd9rWre55P+EF\nFzurjqh3BaM3DQrPMqR0AMkCgYEAkt5BqS7YHulvhGr9jQ7gH1OZS8kAWYjJeShO\nXFm51jUgCJWBMrTlXcx8m/1xfBvMKLQpjSL4wDAA63u03XlZc+o6SB5BkeI6RlGs\nn/DhBBS2FK2d86+nWRpJVPwktU2s5P0MniJP97GrVEX2fkDx0nLiSkgTE9yCIvik\nhO9t020CgYEAxbRcW1jAB9cFeWKo4YxXAQv3DWc3PNYgNTAX8/GvS98DQC8LprtC\njMUPwaDQZBt1gMiPBYyYE8nqlIt6lHTJ8jZxQNIDYkbC8NNzPU26nj5UPfXGo5Io\nLYi4xiLXfbAZhTP7M2Bf3sJXAjENtPUh/AHexcBSHF3ZRPxxokxx9aQ=\n-----END RSA PRIVATE KEY-----\n'

CertComputeCreate

CertComputeCreate 用于创建具有证书的 Amphora 虚拟机。

# file: /opt/rocky/octavia/octavia/controller/worker/tasks/compute_tasks.py

class CertComputeCreate(ComputeCreate):
    def execute(self, amphora_id, server_pem,
                build_type_priority=constants.LB_CREATE_NORMAL_PRIORITY,
                server_group_id=None, ports=None):
        """Create an amphora

        :returns: an amphora
        """

        # load client certificate
        with open(CONF.controller_worker.client_ca, 'r') as client_ca:
            ca = client_ca.read()
        config_drive_files = {
            '/etc/octavia/certs/server.pem': server_pem,
            '/etc/octavia/certs/client_ca.pem': ca}
        return super(CertComputeCreate, self).execute(
            amphora_id, config_drive_files=config_drive_files,
            build_type_priority=build_type_priority,
            server_group_id=server_group_id, ports=ports)

形参 server_pem 的实参是 GenerateServerPEMTask 的 return，也就是说 config_drive_files 包含了 CA 证书、Amphora 证书、Amphora 私钥三者。最终通过 novaclient 调用 Nova 的 userdata 和 config drive 机制来完成 config_drive_files 文件注入。

# file: /opt/rocky/octavia/octavia/compute/drivers/nova_driver.py

            amphora = self.manager.create(
                name=name, image=image_id, flavor=amphora_flavor,
                key_name=key_name, security_groups=sec_groups,
                nics=nics,
                files=config_drive_files,
                userdata=user_data,
                config_drive=True,
                scheduler_hints=server_group,
                availability_zone=CONF.nova.availability_zone
            )

登录到 Amphora Instance 可以查看到这些文件：

ubuntu@amphora-3a980a2f-4954-47cc-bf56-fc206fc50ae3:~$ ll /etc/octavia/certs/
total 16
drwxr-xr-x 2 root root 4096 Nov  5 02:36 ./
drwxr-xr-x 3 root root 4096 Nov  5 02:36 ../
-rw-rw---- 1 root root 1294 Nov  5 02:36 client_ca.pem
-rw-rw---- 1 root root 2964 Nov  5 02:36 server.pem

/etc/octavia/amphora-agent.conf 配置文件的内容如下：

[DEFAULT]
debug = True

[haproxy_amphora]
base_cert_dir = /var/lib/octavia/certs
base_path = /var/lib/octavia
bind_host = ::
bind_port = 9443
haproxy_cmd = /usr/sbin/haproxy
respawn_count = 2
respawn_interval = 2
use_upstart = True

[health_manager]
controller_ip_port_list = 192.168.0.4:5555
heartbeat_interval = 10
heartbeat_key = insecure

[amphora_agent]
agent_server_ca = /etc/octavia/certs/client_ca.pem
agent_server_cert = /etc/octavia/certs/server.pem
agent_request_read_timeout = 120
amphora_id = a47ce718-1b3e-40a9-9fd9-a6ac5e1deba1
amphora_udp_driver = keepalived_lvs

• [health_manager] section 描述了 amphora-agent 如何向 octavia-health-manager 上报监控状态。
• [amphora_agent] section 中的 agent_server_ca 和 agent_server_cert 在启动 amphora-agent 时候被加载启用 SSL 通信，后文再继续展开。

Amphora Agent

amphora-agent 服务进程启动脚本：

# file: /usr/local/bin/amphora-agent

#!/opt/amphora-agent-venv/bin/python3

# -*- coding: utf-8 -*-
import re
import sys

from octavia.cmd.agent import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

amphora-agent 服务进程的入口函数：

# file: /opt/rocky/octavia/octavia/cmd/agent.py

# start api server
def main():
    # comment out to improve logging
    service.prepare_service(sys.argv)

    gmr.TextGuruMeditation.setup_autorun(version)

    health_sender_proc = multiproc.Process(name='HM_sender',
                                           target=health_daemon.run_sender,
                                           args=(HM_SENDER_CMD_QUEUE,))
    health_sender_proc.daemon = True
    health_sender_proc.start()

    # Initiate server class
    server_instance = server.Server()

    bind_ip_port = utils.ip_port_str(CONF.haproxy_amphora.bind_host,
                                     CONF.haproxy_amphora.bind_port)
    options = {
        'bind': bind_ip_port,
        'workers': 1,
        'timeout': CONF.amphora_agent.agent_request_read_timeout,
        'certfile': CONF.amphora_agent.agent_server_cert,
        'ca_certs': CONF.amphora_agent.agent_server_ca,
        'cert_reqs': True,
        'preload_app': True,
        'accesslog': '/var/log/amphora-agent.log',
        'errorlog': '/var/log/amphora-agent.log',
        'loglevel': 'debug',
    }
    AmphoraAgent(server_instance.app, options).run()

AmphoraAgent Application Class 继承自 gunicorn.app.base.BaseApplication，通过 options 中的 CONF.amphora_agent.agent_server_cert 与 CONF.amphora_agent.agent_server_ca 加载 Amphora 的证书文件路径并启用 SSL 通信。

实际上 amphora-agent 使用的是 Flask 框架，在生成 Web Application 的过程中也完成了路由函数和视图函数的初始化。

# file: /opt/rocky/octavia/octavia/amphorae/backends/agent/api_server/server.py

class Server(object):
    def __init__(self):
        self.app = flask.Flask(__name__)
        self._osutils = osutils.BaseOS.get_os_util()
        self._keepalived = keepalived.Keepalived()
        self._listener = listener.Listener()
        self._udp_listener = (udp_listener_base.UdpListenerApiServerBase.
                              get_server_driver())
        self._plug = plug.Plug(self._osutils)
        self._amphora_info = amphora_info.AmphoraInfo(self._osutils)

        register_app_error_handler(self.app)

        self.app.add_url_rule(rule=PATH_PREFIX +
                              '/listeners/<amphora_id>/<listener_id>/haproxy',
                              view_func=self.upload_haproxy_config,
                              methods=['PUT'])
...

AmphoraAPIClient

再回过头来看看 AmphoraAPIClient 的实现，AmphoraAPIClient 在建立 REST 连接的 session 时会导入了 create_certificates.sh 创建的 Server 证书，由 CONF.haproxy_amphora.client_cert 和 CONF.haproxy_amphora.server_ca 指定。

class AmphoraAPIClient(object):
    def __init__(self):
        super(AmphoraAPIClient, self).__init__()
        self.secure = False

        self.get = functools.partial(self.request, 'get')
        self.post = functools.partial(self.request, 'post')
        self.put = functools.partial(self.request, 'put')
        self.delete = functools.partial(self.request, 'delete')
        self.head = functools.partial(self.request, 'head')

        self.start_listener = functools.partial(self._action,
                                                consts.AMP_ACTION_START)
        self.stop_listener = functools.partial(self._action,
                                               consts.AMP_ACTION_STOP)
        self.reload_listener = functools.partial(self._action,
                                                 consts.AMP_ACTION_RELOAD)

        self.start_vrrp = functools.partial(self._vrrp_action,
                                            consts.AMP_ACTION_START)
        self.stop_vrrp = functools.partial(self._vrrp_action,
                                           consts.AMP_ACTION_STOP)
        self.reload_vrrp = functools.partial(self._vrrp_action,
                                             consts.AMP_ACTION_RELOAD)

        self.session = requests.Session()
        self.session.cert = CONF.haproxy_amphora.client_cert
        self.ssl_adapter = CustomHostNameCheckingAdapter()
        self.session.mount('https://', self.ssl_adapter)
        ...

    def request(self, method, amp, path='/', timeout_dict=None, **kwargs):
        cfg_ha_amp = CONF.haproxy_amphora
        if timeout_dict is None:
            timeout_dict = {}
        req_conn_timeout = timeout_dict.get(
            consts.REQ_CONN_TIMEOUT, cfg_ha_amp.rest_request_conn_timeout)
        req_read_timeout = timeout_dict.get(
            consts.REQ_READ_TIMEOUT, cfg_ha_amp.rest_request_read_timeout)
        conn_max_retries = timeout_dict.get(
            consts.CONN_MAX_RETRIES, cfg_ha_amp.connection_max_retries)
        conn_retry_interval = timeout_dict.get(
            consts.CONN_RETRY_INTERVAL, cfg_ha_amp.connection_retry_interval)

        LOG.debug("request url %s", path)
        _request = getattr(self.session, method.lower())
        _url = self._base_url(amp.lb_network_ip) + path
        LOG.debug("request url %s", _url)
        reqargs = {
            'verify': CONF.haproxy_amphora.server_ca,
            'url': _url,
            'timeout': (req_conn_timeout, req_read_timeout), }
        reqargs.update(kwargs)
        headers = reqargs.setdefault('headers', {})
        ...

由于 Server 证书和 Amphora 证书都是由同一个 CA 中心签发的，存在信任链关系，所以 AmphoraAPIClient 也就可以与 Amphora 进行 HTTPS 通信了。

最后

一言以蔽之，Octavia 使用 CA 认证是为了保证 octavia-worker 和 Amphora 的通信安全。我们需要注意的是相关配置项目的意义与证书的作用，通过分析代码实现的方式能够对这个认证配置了解得更加深入。在 R 版本中，octavia-worker 和 Amphora 的通信仍不能称之为非常稳定，所以还是很有必要了解一下前因后果，以便候查。