RDMA Programming - Base on linux-rdma

首页分类标签留言关于订阅2017-11-08 | 分类 Network  | 标签 RDMA  RoCE  Linux-RDMA

linux-rdma为Linux内核Infiniband子系统drivers/infiniband对应的用户态库,提供了Infiniband Verbs APIRDMA Verbs API.

基本概念

  • Queue Pair(QP)

为了进行RDMA操作,需要在两端建立连接,这通过Queue Pair (QP)来完成,QP相当于socket。通信的两端都需要进行QP的初始化,Communication Manager (CM) 在双方真正建立连接前交换QP信息。

Once a QP is established, the verbs API can be used to perform RDMA reads, RDMA writes, and atomic operations. Serialized send/receive operations, which are similar to socket reads/writes, can be performed as well.

QP对应数据结构struct ibv_qpibv_create_qp用于创建QP.

/**
* ibv_create_qp - Create a queue pair.
*/
struct ibv_qp *ibv_create_qp(struct ibv_pd *pd,
struct ibv_qp_init_attr *qp_init_attr);
  • Completion Queue(CQ)

Completion Queue is an object which contains the completed work requests which were posted to the Work Queues (WQ). Every completion says that a specific WR was completed (both successfully completed WRs and unsuccessfully completed WRs). A Completion Queue is a mechanism to notify the application about information of ended Work Requests (status, opcode, size, source).

对应数据结构struct ibv_cqibv_create_cq用于创建CQ:

/**
* ibv_create_cq - Create a completion queue
* @context - Context CQ will be attached to
* @cqe - Minimum number of entries required for CQ
* @cq_context - Consumer-supplied context returned for completion events
* @channel - Completion channel where completion events will be queued.
* May be NULL if completion events will not be used.
* @comp_vector - Completion vector used to signal completion events.
* Must be >= 0 and < context->num_comp_vectors.
*/
struct ibv_cq *ibv_create_cq(struct ibv_context *context, int cqe,
void *cq_context,
struct ibv_comp_channel *channel,
int comp_vector);
  • Memory Registration (MR)

Memory Registration is a mechanism that allows an application to describe a set of virtually con- tiguous memory locations or a set of physically contiguous memory locations to the network adapter as a virtually contiguous buffer using Virtual Addresses.

对应数据结构struct ibv_mr:

struct ibv_mr {
struct ibv_context *context;
struct ibv_pd *pd;
void *addr;
size_t length;
uint32_t handle;
uint32_t lkey;
uint32_t rkey;
};

Every MR has a remote and a local key (rkey, lkey).

Local keys are used by the local HCA to access local memory, such as during a receive operation.

Remote keys are given to the remote HCA to allow a remote process access to system memory during RDMA operations.

ibv_reg_mr registers a memory region (MR), associates it with a protection domain (PD), and assigns it local and remote keys (lkey, rkey).

/**
* ibv_reg_mr - Register a memory region
*/
struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr,
size_t length, int access);
  • Protection Domain (PD)

Object whose components can interact with only each other. These components can be AH, QP, MR, and SRQ.

A protection domain is used to associate Queue Pairs with Memory Regions and Memory Windows , as a means for enabling and controlling network adapter access to Host System memory.

struct ibv_pd is used to implement protection domains:

struct ibv_pd {
struct ibv_context *context;
uint32_t handle;
};

ibv_alloc_pd creates a protection domain (PD). PDs limit which memory regions can be accessed by which queue pairs (QP) providing a degree of protection from unauthorized access.

/**
* ibv_alloc_pd - Allocate a protection domain
*/
struct ibv_pd *ibv_alloc_pd(struct ibv_context *context);
  • Send Request (SR)

An SR defines how much data will be sent, from where, how and, with RDMA, to where. struct ibv_send_wr is used to implement SRs.参考struct ibv_send_wr

示例(IB Verbs API example)

RDMA应用可以使用librdmacm或者libibverbs API编程。前者是对后者的进一步封装。

rc_pingpong是直接使用libibverbs API编程的示例。

一般来说,使用IB Verbs API的基本流程如下:

  • (1) Get the device list

First you must retrieve the list of available IB devices on the local host. Every device in this list contains both a name and a GUID. For example the device names can be: mthca0, mlx4_1.参考这里.

IB devices对应数据结构struct ibv_device:

struct ibv_device {
struct _ibv_device_ops _ops;
enum ibv_node_type node_type;
enum ibv_transport_type transport_type;
/* Name of underlying kernel IB device, eg "mthca0" */
char name[IBV_SYSFS_NAME_MAX];
/* Name of uverbs device, eg "uverbs0" */
char dev_name[IBV_SYSFS_NAME_MAX];
/* Path to infiniband_verbs class device in sysfs */
char dev_path[IBV_SYSFS_PATH_MAX];
/* Path to infiniband class device in sysfs */
char ibdev_path[IBV_SYSFS_PATH_MAX];
};

应用程序通过API ibv_get_device_list获取IB设备列表:

/**
* ibv_get_device_list - Get list of IB devices currently available
* @num_devices: optional. if non-NULL, set to the number of devices
* returned in the array.
*
* Return a NULL-terminated array of IB devices. The array can be
* released with ibv_free_device_list().
*/
struct ibv_device **ibv_get_device_list(int *num_devices);
  • (2) Open the requested device

Iterate over the device list, choose a device according to its GUID or name and open it.参考这里.

应用调用ibv_open_device打开IB设备:

/**
* ibv_open_device - Initialize device for use
*/
struct ibv_context *ibv_open_device(struct ibv_device *device);

返回一个ibv_context对象:

struct ibv_context {
struct ibv_device *device;
struct ibv_context_ops ops;
int cmd_fd;
int async_fd;
int num_comp_vectors;
pthread_mutex_t mutex;
void *abi_compat;
};
  • (3) Allocate a Protection Domain

分配一个PD,参考这里

A Protection Domain (PD) allows the user to restrict which components can interact with only each other.

These components can be AH, QP, MR, MW, and SRQ.

  • (4) Register a memory region

注册一个MR,参考这里.

Any memory buffer which is valid in the process’s virtual space can be registered.

During the registration process the user sets memory permissions and receives local and remote keys (lkey/rkey) which will later be used to refer to this memory buffer.

  • (5) Create a Completion Queue(CQ)

创建一个CQ,参考这里.

A CQ contains completed work requests (WR). Each WR will generate a completion queue entry (CQE) that is placed on the CQ.

The CQE will specify if the WR was completed successfully or not.

  • (6) Create a Queue Pair(QP)

创建QP,参考这里.

Creating a QP will also create an associated send queue and receive queue.

  • (7) Bring up a QP

启动QP,参考这里.

A created QP still cannot be used until it is transitioned through several states, eventually getting to Ready To Send (RTS).

This provides needed information used by the QP to be able send / receive data.

ibv_modify_qp修改QP的状态:

/**
* ibv_modify_qp - Modify a queue pair.
*/
int ibv_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr,
int attr_mask);

例如,对于client/server,需要将QP设置为RTS状态,参考rc_pingpong@pp_connect_ctx.

QP有如下一些状态:

RESET               Newly created, queues empty.
INIT Basic information set. Ready for posting to receive queue.
RTR Ready to Receive. Remote address info set for connected QPs, QP may now receive packets.
RTS Ready to Send. Timeout and retry parameters set, QP may now send packets.
  • (8) Post work requests and poll for completion

Use the created QP for communication operations.

参考pp_post_sendibv_poll_cq.

  • (9) Cleanup
Destroy objects in the reverse order you created them:
Delete QP
Delete CQ
Deregister MR
Deallocate PD
Close device

测试

  • server
# ibv_rc_pingpong -d rxe0 -g 0 -s 128 -r 1 -n 1
local address: LID 0x0000, QPN 0x000011, PSN 0x626753, GID fe80::5054:61ff:fe57:1211
remote address: LID 0x0000, QPN 0x000011, PSN 0x849753, GID fe80::5054:61ff:fe56:1211
256 bytes in 0.00 seconds = 11.38 Mbit/sec
1 iters in 0.00 seconds = 180.00 usec/iter
  • client
# ibv_rc_pingpong -d rxe0 -g 0 172.18.42.162 -s 128 -r 1 -n 1
local address: LID 0x0000, QPN 0x000011, PSN 0x849753, GID fe80::5054:61ff:fe56:1211
remote address: LID 0x0000, QPN 0x000011, PSN 0x626753, GID fe80::5054:61ff:fe57:1211
256 bytes in 0.00 seconds = 16.13 Mbit/sec
1 iters in 0.00 seconds = 127.00 usec/iter

抓包可以查看client与server端的通信流程:

其中,第一个RC Send only为client发送给server的包,参考这里. 然后server回了一个RC Ack,并给client发送了一个RC Send only,参考这里.

前面的一些TCP包为client与server交互的控制信息,参考这里.

Refs

RDMA Programming - Base on linux-rdma的更多相关文章

  1. Git Base For Linux

    GitHub实战系列汇总:http://www.cnblogs.com/dunitian/p/5038719.html Linux安装git,做个记录吧(使用github提供的隐私邮箱) # git官 ...

  2. NVMe over Fabrics又让RDMA技术火了一把

    RDMA是个什么鬼?相信大部分不关心高性能网络的童鞋都不太了解.但是NVMe over Fabrics的出现让搞存储的不得不抽出时间来看看这个东西,这篇文章就来介绍下我所了解的RDMA. RDMA(R ...

  3. [中英对照]Introduction to Remote Direct Memory Access (RDMA) | RDMA概述

    前言: 什么是RDMA? 简单来说,RDMA就是指不通过操作系统(OS)内核以及TCP/IP协议栈在网络上传输数据,因此延迟(latency)非常低,CPU消耗非常少. 下面给出一篇简单介绍RDMA的 ...

  4. [SPDK/NVMe存储技术分析]009 - Introduction to RDMA Send | RDMA Send操作概论

    来源: https://zcopy.wordpress.com/ 说明: 本文不是对原文的逐字逐句翻译,而是摘取核心部分以介绍RDMA Send操作(后面凡是提到RDMA send, 都对应于IBA里 ...

  5. RDMA调研报告&一点随笔

    计算所科研实践随笔 被淹没在论文海里的两个星期. 早上7:10分起床,草草洗漱,7:30出发,开始漫长的1小时通勤.从地铁站的安检口起,队便排的极长,让人看得头皮发麻.下到了轨道旁稍好,但每趟呼啸而来 ...

  6. 基于SoftRoCE 了解RDMA

    RDMA是基于IB技术的内存直接传送,无需内核参与,硬件网卡搞定.IB需要HPC领域的专用硬件,ROCE则是RDMA协议在普通以太网卡的实现,RoCEv1是在MAC上的二层封装,局域网内可以,要通过路 ...

  7. 容器网络启用RDMA高速通讯-Freeflow

    容器网络启用RDMA高速通讯-Freeflow 容器网络启用RDMA高速通讯-Freeflow 本文编译自: Freeflow,https://github.com/openthings/Freefl ...

  8. Revisiting Network Support for RDMA

    重新审视RDMA的网络支持 本文为SIGCOMM 2018会议论文. 笔者翻译了该论文.由于时间仓促,且笔者英文能力有限,错误之处在所难免:欢迎读者批评指正. 本文及翻译版本仅用于学习使用.如果有任何 ...

  9. RDMA的基础概念

    一张图可以简单明确的说明,目前RDMA的几种技术的差别: RDMA是remote Direct memory access的简称,有几个最基本的特点: CPU offload kernel bypas ...

随机推荐

  1. PB笔记之第一行数据不能删除的解决方法

    如果第一行数据不能删除,则单独写SQL进行删除 window lw_sheet //dw_1.event pfc_deleterow()long i String ls_manidInteger ls ...

  2. js中__proto__和prototype的区别和关系?(转)

    转自知乎:https://www.zhihu.com/question/34183746

  3. java.lang.RuntimeException: Unable to instantiate activity ComponentInfo异常解决

    不能实例化activity有如下三种情况: 1.没有在Manifest.xml 清单中注册该activity,或者在创建完activity后,修改了包名或者activity的类名,而配置清单中没有修改 ...

  4. dev 从表处理

    从表列名,从表选中行和主表选中一样,var selectrow = detailView.GetRow(detailView.FocusedRowHandle) as obj; private voi ...

  5. Mac Book触摸板失灵的解决办法(触摸板按下失灵)

    1. 先关机 2. 同时按住 command+option+R+P 3. 按电源键开机,同时手指保持按住前几个按钮的姿势. 4. 等待电脑发出四下“deng”的声音后松开即可.每次发声间隔大概6~7秒 ...

  6. body onload()事件和table insertRow()、tr insertCell()

    onload事件: 定义和用法: onload 事件会在页面或图像加载完成后立即发生. onload 通常用于 <body> 元素,在页面完全载入后(包括图片.css文件等等.)执行脚本代 ...

  7. Image Processing and Computer Vision_Review:A Performance Evaluation of Local Descriptors——2005.08

    翻译 本地描述符的性能评估——http://tongtianta.site/paper/56756 摘要 - 在本文中,我们比较了为局部感兴趣区域计算的描述符的性能,例如,由Harris-Affine ...

  8. uboot中setenv和saveenv分析

    转:https://blog.csdn.net/weixin_34355715/article/details/85751477 Env在u-boot中通常有两种存在方式,在永久性存储介质中(flas ...

  9. 《Python编程:从入门到实践》第四章 操作列表 习题答案

    #4.1 pizzas = ['KFC','MDL','DKS'] ''' for pizza in pizzas: print(pizza); ''' for pizza in pizzas: pr ...

  10. Python + Selenium 主要实现的功能

    selenium 技术 元素定位的几种方法 WebDriver API ,selenium IDE,selenium grid python 技术 函数.类.方法: 读写文件, unitest单元测试 ...