C基础 带你手写 redis ae 事件驱动模型

引言 - 整体认识

　　redis ae 事件驱动模型, 网上聊得很多. 但当你仔细看完一篇又一篇之后, 可能你看的很舒服, 但对于

作者为什么要这么写, 出发点, 好处, 缺点 ... 可能还是好模糊, 不是吗?

我们这里基于阅读的人已经了解了 IO 复用大致流程且抄写过 ae 的全部代码. 好, 那开始吧, 希望后面的

点拨, 给同学们醍醐灌顶一下.

　　先看看 ae.h 设计

/* A simple event-driven programming library. Originally I wrote this code
 * for the Jim's event-loop (Jim is a Tcl interpreter) but later translated
 * it in form of a library for easy reuse.
 *
 * Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *
 *   * Redistributions of source code must retain the above copyright notice,
 *     this list of conditions and the following disclaimer.
 *   * Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *   * Neither the name of Redis nor the names of its contributors may be used
 *     to endorse or promote products derived from this software without
 *     specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
 * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 * POSSIBILITY OF SUCH DAMAGE.
 */

#ifndef __AE_H__
#define __AE_H__

#include <time.h>

#define AE_OK 0
#define AE_ERR -1

#define AE_NONE 0       /* No events registered. */
#define AE_READABLE 1   /* Fire when descriptor is readable. */
#define AE_WRITABLE 2   /* Fire when descriptor is writable. */
#define AE_BARRIER 4    /* With WRITABLE, never fire the event if the
                           READABLE event already fired in the same event
                           loop iteration. Useful when you want to persist
                           things to disk before sending replies, and want
                           to do that in a group fashion. */

#define AE_FILE_EVENTS 1
#define AE_TIME_EVENTS 2
#define AE_ALL_EVENTS (AE_FILE_EVENTS|AE_TIME_EVENTS)
#define AE_DONT_WAIT 4
#define AE_CALL_AFTER_SLEEP 8

#define AE_NOMORE -1
#define AE_DELETED_EVENT_ID -1

/* Macros */
#define AE_NOTUSED(V) ((void) V)

struct aeEventLoop;

/* Types and data structures */
typedef void aeFileProc(struct aeEventLoop *eventLoop, int fd, void *clientData, int mask);
typedef int aeTimeProc(struct aeEventLoop *eventLoop, long long id, void *clientData);
typedef void aeEventFinalizerProc(struct aeEventLoop *eventLoop, void *clientData);
typedef void aeBeforeSleepProc(struct aeEventLoop *eventLoop);

/* File event structure */
typedef struct aeFileEvent {
    int mask; /* one of AE_(READABLE|WRITABLE|BARRIER) */
    aeFileProc *rfileProc;
    aeFileProc *wfileProc;
    void *clientData;
} aeFileEvent;

/* Time event structure */
typedef struct aeTimeEvent {
    long long id; /* time event identifier. */
    long when_sec; /* seconds */
    long when_ms; /* milliseconds */
    aeTimeProc *timeProc;
    aeEventFinalizerProc *finalizerProc;
    void *clientData;
    struct aeTimeEvent *prev;
    struct aeTimeEvent *next;
} aeTimeEvent;

/* A fired event */
typedef struct aeFiredEvent {
    int fd;
    int mask;
} aeFiredEvent;

/* State of an event based program */
typedef struct aeEventLoop {
    int maxfd;   /* highest file descriptor currently registered */
    int setsize; /* max number of file descriptors tracked */
    long long timeEventNextId;
    time_t lastTime;     /* Used to detect system clock skew */
    aeFileEvent *events; /* Registered events */
    aeFiredEvent *fired; /* Fired events */
    aeTimeEvent *timeEventHead;
    int stop;
    void *apidata; /* This is used for polling API specific data */
    aeBeforeSleepProc *beforesleep;
    aeBeforeSleepProc *aftersleep;
    int flags;
} aeEventLoop;

/* Prototypes */
aeEventLoop *aeCreateEventLoop(int setsize);
void aeDeleteEventLoop(aeEventLoop *eventLoop);
void aeStop(aeEventLoop *eventLoop);
int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask,
        aeFileProc *proc, void *clientData);
void aeDeleteFileEvent(aeEventLoop *eventLoop, int fd, int mask);
int aeGetFileEvents(aeEventLoop *eventLoop, int fd);
long long aeCreateTimeEvent(aeEventLoop *eventLoop, long long milliseconds,
        aeTimeProc *proc, void *clientData,
        aeEventFinalizerProc *finalizerProc);
int aeDeleteTimeEvent(aeEventLoop *eventLoop, long long id);
int aeProcessEvents(aeEventLoop *eventLoop, int flags);
int aeWait(int fd, int mask, long long milliseconds);
void aeMain(aeEventLoop *eventLoop);
char *aeGetApiName(void);
void aeSetBeforeSleepProc(aeEventLoop *eventLoop, aeBeforeSleepProc *beforesleep);
void aeSetAfterSleepProc(aeEventLoop *eventLoop, aeBeforeSleepProc *aftersleep);
int aeGetSetSize(aeEventLoop *eventLoop);
int aeResizeSetSize(aeEventLoop *eventLoop, int setsize);
void aeSetDontWait(aeEventLoop *eventLoop, int noWait);

#endif

很多朋友首次看, 或者第一次手写完毕 ae.h 结构设计文件, 印象里 60% 是模糊不可描述 ~ 也许大致知

道这宏有点感觉应该是和 IO Event 事件有关吧 ...

我这里先稍微要剧透点, 带大家快速了解这个库的结构设计的意图. C 先看结构, 比先看接口设计更容

易获取到核心信息. 上面代码中最重要四个结构分别是

　　aeFileEvent, aeTimeEvent, aeFiredEvent, aeEventLoop

aeFileEvent 是文件描述符 Event, 注册在 aeEventLoop 中, 当触发后会生成事件结构 aeFiredEvent,

用于后续处理.  aeTimeEvent 是 timer Event 同样注册在  aeEventLoop 中用于触发定时事件. (太懒,

懒画图, 有兴趣朋友可以自行理解画出好理解的图) 对于  aeEventLoop 内部字段的设计,  先不剧透了.

后面正文部分会讨论一些.

前言 - 底层解密

　　ae 文件整体结构如下

很清晰的看出 epoll, evport, kqueue, select IO 复用的核心包装. 但写完整个 ae.c 发现对其设计影响

最深可能就是 ae_select.c 中兼容 select 思路.

/* Select()-based ae.c module.
 *
 * Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *
 *   * Redistributions of source code must retain the above copyright notice,
 *     this list of conditions and the following disclaimer.
 *   * Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *   * Neither the name of Redis nor the names of its contributors may be used
 *     to endorse or promote products derived from this software without
 *     specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
 * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 * POSSIBILITY OF SUCH DAMAGE.
 */

#include <sys/select.h>
#include <string.h>

typedef struct aeApiState {
    fd_set rfds, wfds;
    /* We need to have a copy of the fd sets as it's not safe to reuse
     * FD sets after select(). */
    fd_set _rfds, _wfds;
} aeApiState;

static int aeApiCreate(aeEventLoop *eventLoop) {
    aeApiState *state = zmalloc(sizeof(aeApiState));

    if (!state) return -;
    FD_ZERO(&state->rfds);
    FD_ZERO(&state->wfds);
    eventLoop->apidata = state;
    return ;
}

static int aeApiResize(aeEventLoop *eventLoop, int setsize) {
    /* Just ensure we have enough room in the fd_set type. */
    if (setsize >= FD_SETSIZE) return -;
    return ;
}

static void aeApiFree(aeEventLoop *eventLoop) {
    zfree(eventLoop->apidata);
}

static int aeApiAddEvent(aeEventLoop *eventLoop, int fd, int mask) {
    aeApiState *state = eventLoop->apidata;

    if (mask & AE_READABLE) FD_SET(fd,&state->rfds);
    if (mask & AE_WRITABLE) FD_SET(fd,&state->wfds);
    return ;
}

static void aeApiDelEvent(aeEventLoop *eventLoop, int fd, int mask) {
    aeApiState *state = eventLoop->apidata;

    if (mask & AE_READABLE) FD_CLR(fd,&state->rfds);
    if (mask & AE_WRITABLE) FD_CLR(fd,&state->wfds);
}

static int aeApiPoll(aeEventLoop *eventLoop, struct timeval *tvp) {
    aeApiState *state = eventLoop->apidata;
    int retval, j, numevents = ;

    memcpy(&state->_rfds,&state->rfds,sizeof(fd_set));
    memcpy(&state->_wfds,&state->wfds,sizeof(fd_set));

    retval = select(eventLoop->maxfd+,
                &state->_rfds,&state->_wfds,NULL,tvp);
    if (retval > ) {
        for (j = ; j <= eventLoop->maxfd; j++) {
            int mask = ;
            aeFileEvent *fe = &eventLoop->events[j];

            if (fe->mask == AE_NONE) continue;
            if (fe->mask & AE_READABLE && FD_ISSET(j,&state->_rfds))
                mask |= AE_READABLE;
            if (fe->mask & AE_WRITABLE && FD_ISSET(j,&state->_wfds))
                mask |= AE_WRITABLE;
            eventLoop->fired[numevents].fd = j;
            eventLoop->fired[numevents].mask = mask;
            numevents++;
        }
    }
    return numevents;
}

static char *aeApiName(void) {
    return "select";
}

作者实现这个 select 思路不是很好, 他把 ae_select.c 当做局部文件去设计, 没有想拆出来独挡一面.

其次对于 select 的第四个参数 error fds 集合没有处理(ae_epoll.c 中 EPOLLHUB 和 EPOLLERR 是

处理). 实现层面 aeApiPoll 也不够好, 推荐采用下面实现

#include "ae.h"
#include <string.h>
#include <sys/select.h>

static int aeApiPoll(aeEventLoop * eventLoop, struct timeval * tvp) {
    aeApiState * state = eventLoop->apidata;
    int retval, j, numevents = ;

    memcpy(&state->_rfds, &state->rfds, sizeof(fd_set));
    memcpy(&state->_wfds, &state->wfds, sizeof(fd_set));

    retval = select(eventLoop->maxfd+, &state->_rfds, &state->_wfds, NULL, tvp);
    for (j = ; j <= eventLoop->maxfd && numevents < retval; j++) {
        int mask = AE_NONE;
        aeFileEvent * fe = &eventLoop->events[j];

        if (fe->mask == AE_NONE) continue;
        if (fe->mask & AE_READABLE && FD_ISSET(j, &state->_rfds))
            mask |= AE_READABLE;
        if (fe->mask & AE_WRITABLE && FD_ISSET(j, &state->_wfds))
            mask |= AE_WRITABLE;
        if (mask == AE_NONE) continue;

        eventLoop->fired[numevents].fd = j;
        eventLoop->fired[numevents].mask = mask;
        numevents++;
    }

    return numevents;
}

降低不需要处理的 AE_NONE 空事件.  随后的 epoll kqueue 都差不多(evport 不熟, 有心的朋友也别看)

正文 - 细节点拨

　　整体看 ae 事件模型设计, 还是有些简陋的. 我猜测是 redis 重IO和内存操作, 对很多文件描述符需求

较固定,  一个文件描述符多数自始至终. 应对的场景不是那种大量的创建, 交互, 关闭. 所以整体设计也能

接受.

1.  setsize maxfd event fired 到底想表达什么?

#include <time.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <poll.h>
#include <unistd.h>
#include <sys/time.h>
#include <sys/types.h>

#include "ae.h"
#include "config.h"
#include "zmalloc.h"

/* Include the best multiplexing layer supported by this system.
 * The following should be ordered by performances, descending. */
#ifdef HAVE_EVPORT
#include "ae_evport.c"
#else
    #ifdef HAVE_EPOLL
    #include "ae_epoll.c"
    #else
        #ifdef HAVE_KQUEUE
        #include "ae_epoll.c"
        #else
        #include "ae_select.c"
        #endif
    #endif
#endif

aeEventLoop * aeCreateEventLoop(int setsize) {
    aeEventLoop * eventLoop;
    int i;

    if (!(eventLoop = zmalloc(sizeof(*eventLoop)))) goto err;

    eventLoop->events = zmalloc(sizeof(aeFileEvent)*setsize);
    eventLoop->fired = zmalloc(sizeof(aeFiredEvent)*setsize);
    if (!eventLoop->events || !eventLoop->fired) goto err;

    eventLoop->setsize = setsize;
    eventLoop->lastTime = time(NULL);
    eventLoop->timeEventHead = NULL;
    eventLoop->timeEventNextId = ;
    eventLoop->stop = ;
    eventLoop->maxfd = -;
    eventLoop->beforesleep = NULL;
    eventLoop->aftersleep = NULL;
    eventLoop->flags = ;
    if (aeApiCreate(eventLoop) == -) goto err;
    /* Events with mask == AE_NONE are not set. So let's initialize the
     * vector with it. */
    for (i = ; i < setsize; i++)
        eventLoop->events[i].mask = AE_NONE;
    return eventLoop;

err:
    if (eventLoop) {
        zfree(eventLoop->events);
        zfree(eventLoop->fired);
        zfree(eventLoop);
    }
    return NULL;
}

有心的同学可以关注  eventLoop->events 和 eventLoop->fired zmalloc 这块, 这基本已经

把之前的 ae_select.c ae_epoll.c ae_kqueue.c ... 串起来了. 分别用于存要监控的事件和有变动的事件.

对于 setsize 也是个看点我们分别看 server.c server.h config.c 局部代码

[server.c]
server.el = aeCreateEventLoop(server.maxclients+CONFIG_FDSET_INCR);

[server.h]
#define CONFIG_MIN_RESERVED_FDS 32

/* When configuring the server eventloop, we setup it so that the total number
* of file descriptors we can handle are server.maxclients + RESERVED_FDS +
* a few more to stay safe. Since RESERVED_FDS defaults to 32, we add 96
* in order to make sure of not over provisioning more than 128 fds. */
#define CONFIG_FDSET_INCR (CONFIG_MIN_RESERVED_FDS+96)

[config.c]
/* Unsigned int configs */
createUIntConfig("maxclients", NULL, MODIFIABLE_CONFIG, , UINT_MAX, server.maxclients, , INTEGER_CONFIG, NULL, updateMaxclients),

可以看出来 setsize 分为两部分, 一分部分是配置的, 默认是 10000; 另外一部分是预留 128个.

(128 分为两部分 CONFIG_MIN_RESERVED_FDS = 32 + 96, 前者是 redis fd 保留的最少个数)

和上面 aeCreateEventLoop 相似的功能有 aeResizeSetSize

/* Resize the maximum set size of the event loop.
 * If the requested set size is smaller than the current set size, but
 * there is already a file descriptor in use that is >= the requested
 * set size minus one, AE_ERR is returned and the operation is not
 * performed at all.
 *
 * Otherwise AE_OK is returned and the operation is successful. */
int aeResizeSetSize(aeEventLoop *eventLoop, int setsize) {
    int i;

    if (setsize == eventLoop->setsize) return AE_OK;
    if (eventLoop->maxfd >= setsize) return AE_ERR;
    if (aeApiResize(eventLoop,setsize) == -) return AE_ERR;

    eventLoop->events = zrealloc(eventLoop->events,sizeof(aeFileEvent)*setsize);
    eventLoop->fired = zrealloc(eventLoop->fired,sizeof(aeFiredEvent)*setsize);
    eventLoop->setsize = setsize;

    /* Make sure that if we created new slots, they are initialized with
     * an AE_NONE mask. */
    for (i = eventLoop->maxfd+; i < setsize; i++)
        eventLoop->events[i].mask = AE_NONE;
    return AE_OK;
}

透过这两个函数希望你对 aeEventLoop 中 setsize maxfd event fired 这些字段能了解透彻.

2. aeTimeEvent 怎么用, 怎么设计的?

redis 中 timer Event 设计比较简单, 单纯的无序时间链表. 下面这段作者意图表达的很清晰.

/* Search the first timer to fire.
 * This operation is useful to know how many time the select can be
 * put in sleep without to delay any event.
 * If there are no timers NULL is returned.
 *
 * Note that's O(N) since time events are unsorted.
 * Possible optimizations (not needed by Redis so far, but...):
 * 1) Insert the event in order, so that the nearest is just the head.
 *    Much better but still insertion or deletion of timers is O(N).
 * 2) Use a skiplist to have this operation as O(1) and insertion as O(log(N)).
 */
static aeTimeEvent *aeSearchNearestTimer(aeEventLoop *eventLoop)
{
    aeTimeEvent *te = eventLoop->timeEventHead;
    aeTimeEvent *nearest = NULL;

    while(te) {
        if (!nearest || te->when_sec < nearest->when_sec ||
                (te->when_sec == nearest->when_sec &&
                 te->when_ms < nearest->when_ms))
            nearest = te;
        te = te->next;
    }
    return nearest;
}

而其中到底怎么跑起来的呢, 我截取 processTimeEvents 中部分代码, 帮读者了然于心

/* Process time events */
static int processTimeEvents(aeEventLoop *eventLoop) {
    int processed = ;
    aeTimeEvent *te;
    long long maxId;
    time_t now = time(NULL);
...
  {
        ...
        aeGetTime(&now_sec, &now_ms);
        if (now_sec > te->when_sec ||
            (now_sec == te->when_sec && now_ms >= te->when_ms))
        {
            int retval;

            id = te->id;
            retval = te->timeProc(eventLoop, id, te->clientData);
            processed++;
            if (retval != AE_NOMORE) {
                aeAddMillisecondsToNow(retval,&te->when_sec,&te->when_ms);
            } else {
                te->id = AE_DELETED_EVENT_ID;
            }
        }
        ...
  }
...
    return processed;
}

从 retval = te->timeProc -> if 那段. 对于 id 打标为  AE_DELETED_EVENT_ID 标识轮循到的时候要删除.

一旦 retval != AE_NOMORE 就再次修改这个timer Event 相关时间, 方便下次接着跑. 同样我们抽一个例

子出来, 同样核心也在 server.c 中

[server.c]
    /* Create the timer callback, this is our way to process many background
     * operations incrementally, like clients timeout, eviction of unaccessed
     * expired keys and so forth. */
    if (aeCreateTimeEvent(server.el, , serverCron, NULL, NULL) == AE_ERR) {
        serverPanic("Can't create event loop timers.");
        exit();
    }

[server.c]
int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
...
    return /server.hz;
}

整体看他这个 timer Event 很骚, 返回毫秒时间后, 继续注入进去, 继续当循环轮循定时器事件使用

static void aeAddMillisecondsToNow(long long milliseconds, long *sec, long *ms) {
    long cur_sec, cur_ms, when_sec, when_ms;

    aeGetTime(&cur_sec, &cur_ms);
    when_sec = cur_sec + milliseconds/;
    when_ms = cur_ms + milliseconds%;
    if (when_ms >= ) {
        when_sec ++;
        when_ms -= ;
    }
    *sec = when_sec;
    *ms = when_ms;
}

设计的思路挺巧妙的. 多数正常思路通过类型特殊处理, 或者特殊地方再次主动注册.

3. EventLoop 是怎么跑的?

EventLoop 奔跑思路很简单, 一个地方轮循, 内部先跑 aeFileEvent, 再跑 aeTimeEvent

[ae.c]
void aeMain(aeEventLoop *eventLoop) {
    eventLoop->stop = ;
    while (!eventLoop->stop) {
        if (eventLoop->beforesleep != NULL)
            eventLoop->beforesleep(eventLoop);
        aeProcessEvents(eventLoop, AE_ALL_EVENTS|AE_CALL_AFTER_SLEEP);
    }
}

[server.c]
int main(int argc, char **argv) {
...

    aeSetBeforeSleepProc(server.el,beforeSleep);
    aeSetAfterSleepProc(server.el,afterSleep);
    aeMain(server.el);
    aeDeleteEventLoop(server.el);
    return ;
}

/* The End */

整体而言 redis ae 模型还是非常简单, 处理的这些的事情完全是为 redis io 定制的. 够用了.

后续有机会我再大家分析 redis 中特定的 socket io 是怎么处理的.

后记 - 为爱展望

❤ 错误是难免的, 欢迎有心同学指正和补充图, 文字是干瘪的 ~

　Here We Are Again - https://music.163.com/#/song?id=27876900