Ceph 之 Background on http frontends
Background on http frontends
-civetweb frontend
--thread per connection, requires a lot of threads
---qos or priority queuing would block frontend threads
-beast frontend
--boost::beast for http parsing
--boost::asio for async networking/io
--async for accepting connections and reading headers
---good model for qos - can queue requests without blocking threads
--synchronous call to process_request()
---thread per request, still need lots of threads
goal: scale requests independently of threads
-why boost::asio
--doesn't impose a threading model. io_service object is a reactor, call run() from any thread
--mature library, basis for C++ std::net library in Networking TS [1]
--the Extensible Asynchronous Model [2] provides several options for async primitives (callbacks, futures, coroutines)
--boost::asio::spawn() stackful coroutines: "enables programs to implement asynchronous logic in a synchronous manner" [3 http://www.boost.org/doc/libs/1_65_1/doc/html/boost_asio/reference/spawn.html]
proposed librados interfaces for asio [4 https://github.com/ceph/ceph/pull/19054]
--header-only wrapper over librados c++ api
--conform to the Extensible Asynchronous Model, so support the same primitives - see unit tests for examples
--deeper Objecter integration work in progress by Adam Emerson [5]
--gives radosgw a unified interface for async operations over http and rados
async process_request() [6]
-add optional yield_context* argument to process_request()
-beast frontend passes one, civetweb passes nullptr
-any librados calls use new interface when given a yield_context
-requires the yield_context* to be passed everywhere in between
--but we can stash it in req_state to make it available to all ops
-getting started with the easy stuff
--rgw_get_system_obj()
--reading user objects for authentication
--reading bucket/bucket instance objects (common to most s3/swift ops)
-this process leaves a lot of gaps. for example, rgw_get_system_obj() is in tons of call paths without access to a yield_context
--(either outside process_request(), or just aren't hooked up yet)
--just passing 'nullptr' makes it impossible to differentiate the yield_context argument from its 4 other arguments that default to nullptr!
--that makes it impossible to reason about which call paths could run asynchronously
-measurable progress towards full asynchrony
--new vocabulary type 'optional_yield_context' with 'null_yield' for empty value
--null_yield designates a call site that is definitely synchronous
--makes it easy to audit the code and find the pieces that still need conversion
-fighting regression once we're close
--have librados calls log warnings when called synchronously from a beast frontend thread (using a thread_local flag)
--scan those logs in teuthology runs to flag failures
and then?
-vastly reduce the number of frontend threads for beast
-consolidate other background threads
remaining work:
- RGWGetObj waits on AioCompletions - use AioThrottle from PutObj instead
- replace librados IoCtx::operate() calls with rgw_rados_operate() and optional_yield_context
- thread optional_yield_context all the way from beast frontend to rgw_rados_operate() calls
- some cls client calls use IoCtx::operate() directly
- block_while_resharding() sleeps on a condition variable
- no async interface for pool object listings with IoCtx.nobjects_begin()
- libcurl http requests for auth (Keystone and OPA)
[1] "C++ Technical Specification - Extensions for Networking"
http://cplusplus.github.io/networking-ts/draft.pdf
[2] "Library Foundations for Asynchronous Operations"
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3896.pdf
[3] Reference: boost::asio::spawn
http://www.boost.org/doc/libs/1_65_1/doc/html/boost_asio/reference/spawn.html
[4] "librados: add async interfaces for use with boost::asio"
https://github.com/ceph/ceph/pull/19054
[5] "osdc/Objecter: Boost.Asio (I object!)"
https://github.com/ceph/ceph/pull/16715
[6] work in progress branch:
https://github.com/cbodley/ceph/commits/wip-rgw-async-process-171120
Ceph 之 Background on http frontends的更多相关文章
- 虚拟机使用ceph-deploy安装ceph
参考: ceph官网介绍 使用ceph-deploy安装Ceph 12.x 安装虚拟机 首先安装虚拟机环境,虚拟机安装这里不做介绍,本实验使用的镜像为CentOS-7-x86_64-Everythin ...
- Ceph常用命令
目录 [1.环境准备] [2.部署管理] [3.集群扩容] [4.用户管理] [5.密钥环管理] [6.块设备管理] [7.快照管理] [8.参考链接] 简要说明: 最近心血来潮,对分布式存储感兴趣, ...
- ceph mimic版本 部署安装
ceph 寻址过程 1. file --- object映射, 把file分割成N个相同的对象 2. object - PG 映射, 利用静态hash得到objectID的伪随机值,在 "位 ...
- Ceph常规操作及常见问题梳理
Ceph集群管理 每次用命令启动.重启.停止Ceph守护进程(或整个集群)时,必须指定至少一个选项和一个命令,还可能要指定守护进程类型或具体例程. **命令格式如 {commandline} [opt ...
- ceph kubernetes中使用
1.在管理节点上,进入刚创建的放置配置文件的目录,用 ceph-deploy 执行如下步骤 mkdir /opt/cluster-ceph cd /opt/cluster-ceph ceph-depl ...
- 1、ceph-deploy之部署ceph集群
环境说明 server:3台虚拟机,挂载卷/dev/vdb 10G 系统:centos7.2 ceph版本:luminous repo: 公网-http://download.ceph.com,htt ...
- CEPH RGW多 ZONE的配置
相关的名称解释 Region :可以理解为区域,是基于地理位置的逻辑划分:如:华南,华北之类,包含多个region的Ceph集群必须指定一个master region,一个region可以包含一个或者 ...
- Ceph 管理和使用
ceph 管理 上次介绍了Ceph集群架构并且搭建了ceph集群,本节介绍ceph用户认证流程和挂载.cephFS.ceph RBD以及ceph mds高可用 1. ceph 授权流程和用户权限管理 ...
- 安装ceph (快速) 步骤二:存储集群
用 ceph-deploy 从管理节点建立一个 Ceph 存储集群,该集群包含三个节点,以此探索 Ceph 的功能. 创建一个 Ceph 存储集群,它有一个 Monitor 和两个 OSD 守护进程. ...
随机推荐
- [转]js模块化——AMD及require.js
由CommonJS组织提出了许多新的JavaScript架构方案和标准,希望能为前端开发提供统一的指引.AMD规范就是其中比较著名一个,全称是Asynchronous Module Definitio ...
- numpy使用中的疑惑
numpy中有一些常用的用来产生随机数的函数,randn就是其中一个,randn函数位于numpy.random中,函数原型如下: numpy.random.randn(d0, d1, ..., dn ...
- 嘴巴题2 UVA10779 收集者的难题
UVA10779 收集者的难题 题目: 有\(T(T\leq 20)\)组数据.\(Bob\)在与他的\(n−1(2\leq n\leq 10)个\)朋友交换糖纸,一共有\(m(5\leq m\leq ...
- Luogu P1712 [NOI2016]区间(线段树)
P1712 [NOI2016]区间 题意 题目描述 在数轴上有 \(N\) 个闭区间 \([l_1,r_1],[l_2,r_2],...,[l_n,r_n]\) .现在要从中选出 \(M\) 个区间, ...
- Redis源码解析:18Hiredis同步API和回复解析API代码解析
Redis的sentinel模式使用了Hiredis代码,Hiredis是redis数据库一个轻量级的C语言客户端库.它实现的向Redis发送命令的API函数redisCommand,使用方法类似于p ...
- [Array] 566. Reshape the Matrix
In MATLAB, there is a very useful function called 'reshape', which can reshape a matrix into a new o ...
- Codeforces 467D
题目链接 D. Fedor and Essay time limit per test 2 seconds memory limit per test 256 megabytes input stan ...
- ACdream 1108(莫队)
题目链接 The kth number Time Limit: 12000/6000MS (Java/Others)Memory Limit: 128000/64000KB (Java/Others) ...
- spring cloud深入学习(三)-----服务消费
在上一篇博文中简单实现了eureka-server以及eureka-provider,后面会实现eureka-cosumer,现在针对eureka做进一步的详解. 微服务整体架构 文字再美也没有图片直 ...
- mysql优化-数据库设计基本原则
mysql优化-数据库设计基本原则 一.数据库设计三范式 第一范式:字段具有原子性 原子性是指数据库的所有字段都不可被再次划分,如下表就不满足原子性,起点与终点 字段就可被拆分为起点与终点两个字段. ...