Building a Non-blocking TCP server using OTP principles
转自:https://erlangcentral.org/wiki/index.php/Building_a_Non-blocking_TCP_server_using_OTP_principles
Building a Non-blocking TCP server using OTP principles
From ErlangCentral Wiki
Contents |
Author
Serge Aleynikov <saleyn at gmail.com>
Overview
A reader of this tutorial is assumed to be familiar with gen_server and gen_fsm behaviours, TCP socket communications using gen_tcp module, active and passive socket modes, and OTP supervision principles.
OTP provides a convenient framework for building reliable applications. This is in part accomplished by abstracting common functionality into a set of reusable behaviours such as gen_server and gen_fsm that are linked to OTP's supervision hierarchy.
There are several known TCP server designs. The one we are going to cover involves one process listening for client connections and spawning an FSM process per connecting client. While there is support for TCP communications available in OTP through the gen_tcp module, there is no standard behavior for building non-blocking TCP servers using OTP standard guidelines. By non-blocking we imply that the listening process and the client-handling FSMs should not make any blocking calls and be readily responsive to incoming control messages (such as changes in system configuration, restart requests, etc.) without causing timeouts. Note that blocking in the context of Erlang means blocking an Erlang process rather than the emulator's OS process(es).
In this tutorial we will show how to build a non-blocking TCP server using gen_server and gen_fsm behaviours that offers flow control and is fully compliant with OTP application design principles.
A reader who is new to the OTP framework is encouraged to read Joe Armstrong's tutorial on how to build A Fault-tolerant Server using blocking gen_tcp:connect/3 and gen_tcp:accept/1 calls without involving OTP.
This tutorial was inspired by several threads (e.g. one, two) on the Erlang Questions mailing list mentioning an approach to building non-blocking asynchronous TCP servers.
Server Design
The design of our server will include the main application's supervisor tcp_server_app process with one_for_one restart strategy and two child specifications. The first one being a listening process implemented as a gen_server behaviour that will wait for asynchronous notifications of client socket connections. The second one is another supervisor tcp_client_sup responsible for starting client handling FSMs and logging abnormal disconnects via standard SASL error reports.
For the sake of simplicity of this tutorial, the client handling FSM (tcp_echo_fsm) will implement an echo server that will echo client's requests back to the client.
+----------------+
| tcp_server_app |
+--------+-------+
| (one_for_one)
+----------------+---------+
| |
+-------+------+ +-------+--------+
| tcp_listener | + tcp_client_sup |
+--------------+ +-------+--------+
| (simple_one_for_one)
+-----|---------+
+-------|--------+|
+--------+-------+|+
| tcp_echo_fsm |+
+----------------+
Application and Supervisor behaviours
In order to build an OTP application we need to construct modules implementing an application and supervisor behaviour callback functions. While traditionally these functionalities are implemented in separate modules, given their succinctness we'll combine them in one module.
As an added bonus we implement a get_app_env function that illustrates how to process configuration options as well as command-line options given to the emulator at start-up.
The two instances of init/1 function are for two tiers of supervision hierarchy. Since two different restart strategies for each supervisor are needed, we implement them at different tiers.
Upon application's startup the tcp_server_app:start/2 callback function calls supervisor:start_link/2 that creates main application's supervisor calling tcp_server_app:init([Port, Module]) callback. This supervisor creates a tcp_listener process and a child supervisor tcp_client_sup responsible for spawning client connections. The Module argument in the init function is the name of client-connection handling FSM (in this case tcp_echo_fsm).
|
TCP Server Application (tcp_server_app.erl) |
||
|
Listener Process
One of the shortcomings of the gen_tcp module is that it only exports interface to a blocking accept call. This leads most of developers working on an implementation of a TCP server build a custom process linked to a supervisor using proc_lib or come up with some other proprietary design.
Examining prim_inet module reveals an interesting fact that the actual call to inet driver to accept a client socket is asynchronous. While this is a non-documented property, which means that the OTP team is free to change this implementation, we will exploit this functionality in the construction of our server.
The listener process is implemented as a gen_server behaviour:
|
TCP Listener Process (tcp_listener.erl) |
||
|
In this module init/1 call takes two parameters - the port number that the TCP listener should be started on and the name of a protocol handling module for client connections. The initialization function opens a listening socket in passive {active, false} mode. This is done so that we have flow control of the data received on the connected client sockets that will inherit this option from the listening socket.
The most interesting part of this code is the prim_inet:async_accept/2 call as well as the handling of asynchronous inet_async messages. In order to get this working we also needed to copy some of the internal OTP code encapsulated in the set_sockopt/2 function that handles socket registration with inet database and copying some options to the client socket.
As soon as a client socket is connected inet driver will notify the listening process using {inet_async, ListSock, Ref, {ok, CliSocket}} message. At this point we'll instantiate a new client socket handling process and set its ownership of the CliSocket.
Client Socket Handling Process
While tcp_listener is a generic implementation, tcp_echo_fsm is a mere stub FSM for illustrating how to write TCP servers. This modules needs to export two functions - one start_link/0 for a tcp_client_sup supervisor and another set_socket/2 for the listener process to notify the client connection handling FSM process that it is now the owner of the socket, and can begin receiving messages by setting the {active, once} or {active, true} option.
We would like to highlight the synchronization pattern used between the listening process and client connection-handling FSM to avoid possible message loss due to dispatching some messages from the socket to the wrong (listening) process. The process owning the listening socket has it open with {active, false}. After accepting the client's socket that socket inherits its socket options (including {active, false}) from the listener, transfers ownership of the socket to the newly spawned client connection-handling FSM by calling gen_tcp:controlling_process/2 and calls Module:set_socket/2 to notify the FSM that it can start receiving messages from the socket. Until the FSM process enables message delivery by setting the active mode on the socket by calling inet:setopts(Socket, [{active, once}]), the data sent by the TCP sender stays in the socket buffer.
When socket ownership is transfered to FSM in the 'WAIT_FOR_SOCKET' state the FSM sets {active, once} option to let inet driver send it one TCP message at a time. This is the OTP way of preserving flow control and avoiding process message queue flooding with TCP data and crashing the system in case of a fast-producer-slow-consumer case.
The FSM states are implemented by special functions in the tcp_echo_fsm module that use a naming convention with capital case state names enclosed in single quotes. The FSM consists of two states. 'WAIT_FOR_SOCKET' is the initial state in which the FSM is waiting for assignment of socket ownership, and 'WAIT_FOR_DATA' is the state that represents awaiting for TCP message from a client. In this state FSM also handles a special 'timeout' message that signifies no activity from a client and causes the process to stop and close client connection.
|
TCP Client Socket Handling FSM (tcp_echo_fsm.erl) |
||
|
Application File
Another required part of building an OTP application is creation of an application file that includes application name, version, startup module and environment.
|
Application File (tcp_server.app) |
||
|
Compiling
Create the following directory structure for this application:
./tcp_server
./tcp_server/ebin/
./tcp_server/ebin/tcp_server.app
./tcp_server/src/tcp_server_app.erl
./tcp_server/src/tcp_listener.erl
./tcp_server/src/tcp_echo_fsm.erl
$ cd tcp_server/src
$ for f in tcp*.erl ; do erlc +debug_info -o ../ebin $f
Running
We are going to start an Erlang shell with SASL support so that we can view all progress and error reports for our TCP application. Also we are going to start appmon application in order to examine visually the supervision hierarchy.
$ cd ../ebin
$ erl -boot start_sasl
...
1> appmon:start().
{ok,<0.44.0>}
2> application:start(tcp_server).
ok
Now click on the tcp_server button in the appmon's window in order to display supervision hierarchy of the tcp_server application.
3> {ok,S} = gen_tcp:connect({127,0,0,1},2222,[{packet,2}]).
{ok,#Port<0.150>}
The step above initiated a new client connection to the echo server.
4> gen_tcp:send(S,<<"hello">>).
ok
5> f(M), receive M -> M end.
{tcp,#Port<0.150>,"hello"}
We verified that the echo server works as expected. Now let's try to crash the client connection on the server and watch for the supervisor generating an error report entry on screen.
6> [{_,Pid,_,_}] = supervisor:which_children(tcp_client_sup).
[{undefined,<0.64.0>,worker,[]}]
7> exit(Pid,kill).
true
=SUPERVISOR REPORT==== 31-Jul-2007::14:33:49 ===
Supervisor: {local,tcp_client_sup}
Context: child_terminated
Reason: killed
Offender: [{pid,<0.77.0>},
{name,undefined},
{mfa,{tcp_echo_fsm,start_link,[]}},
{restart_type,temporary},
{shutdown,2000},
{child_type,worker}]
Note that if you are putting this server under a stress test with many incoming connections, the listener process may fail to accept new connections after the number of open file descriptors reaches the limit set by the operating system. In that case you will see the error:
"too many open files"
If you are running Linux/UNIX, google for a solution (which ultimately boils down to increasing the per-process limit by setting "ulimit -n ..." option).
Conclusion
OTP provides building blocks for constructing non-blocking TCP servers. This tutorial showed how to create a simple TCP server with flow control using standard OTP behaviours. As an exercise the reader is encouraged to try abstracting generic non-blocking TCP server functionality into a stand-along behaviour.
Sample Implementations
View source |
Discuss this page |
Page history |
What links here |
Related changes
Building a Non-blocking TCP server using OTP principles的更多相关文章
- 【转载】C# Tutorial - Simple Threaded TCP Server
http://tech.pro/tutorial/704/csharp-tutorial-simple-threaded-tcp-server In this tutorial I'm going t ...
- socket - socketserver - start TCP server
前面提到如何使用socket模块启动tcpserver: 创建socket:sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 绑定ip: ...
- [转] 3个学习Socket编程的简单例子:TCP Server/Client, Select
以前都是采用ACE的编写网络应用,最近由于工作需要,需要直接只用socket接口编写CS的代码,重新学习这方面的知识,给出自己所用到的3个简单例子,都是拷贝别人的程序.如果你能完全理解这3个例子,估计 ...
- [转]一个基于完成端口的TCP Server Framework,浅析IOCP
[转]一个基于完成端口的TCP Server Framework,浅析IOCP http://www.cppblog.com/adapterofcoms/archive/2010/06/26/1187 ...
- Socket TCP Server一个端口可以有多少个长连接?受到什么影响?linux最大文件句柄数量总结
Socket TCP Server一个端口可以有多少个长连接? 网上答案很多,不知道那个才是正确的 理论上是无限的 16.Linux中,一个端口能够接受tcp链接数量的理论上限是? A.1024 B. ...
- 基于 LWIP 建立 TCP Server 与主机通信实验
LWIP 版本:2.0.3 上一篇文章是写如何将 LWIP 移植到板子上,今天晚上记录基于 LWIP 实现与主机的网络通信. 先是打开了原子的实验例程,大概浏览了一遍,觉得 TCP 网络网络通信也就是 ...
- swoole深入学习 2. tcp Server和tcp Client
这节来学习Swoole最基础的Server和Client.会通过创建一个tcp Server来讲解. server <?php class Server { private $serv; pub ...
- Modbus库开发笔记之九:利用协议栈开发Modbus TCP Server应用
前面我们已经完成了Modbus协议栈的开发,但这不是我们的目的.我们开发它的目的当然是要使用它来解决我们的实际问题.接下来我们就使用刚开发的Modbus协议栈开发一个Modbus TCP Server ...
- Modbus库开发笔记之三:Modbus TCP Server开发
在完成了前面的工作后,我们就可以实现有针对性的应用了,首先我们来实现Modbus TCP的服务器端应用.当然我们不是做具体的应用,而是对Modbus TCP的服务器端应用进行封装以供有需要时调用. 这 ...
随机推荐
- C 字符串常量和字符串变量定义和区别
字符串常量 定义:在一个双引号""内的字符序列或者转义字符序列称为字符串常量 例如:"HA HA!" "abc" "\n\t& ...
- ios 开源免费接口
ios 开源免费接口 国家气象局提供的天气预报接口 接口地址: http://www.weather.com.cn/data/sk/101010100.html http://www.weather. ...
- 一文读懂什么是Java中的自动拆装箱
基本数据类型 基本类型,或者叫做内置类型,是Java中不同于类(Class)的特殊类型.它们是我们编程中使用最频繁的类型. Java是一种强类型语言,第一次申明变量必须说明数据类型,第一次变量赋值称为 ...
- Wamp错误: Forbidden You don't have permission to access / on this server.
找到php的配置文件httpd.conf(找不到的话看这篇:http://www.cnblogs.com/liulangmao/p/3569807.html) 在原有的位置文件中找到配置节 <D ...
- StarUML使用说明-指导手册
1.综述 StarUML是一种生成类图和其他类型的统一建模语言(UML)图表的工具.这是一个用Java语言描述的创建类图的简明手册. StarUML(简称SU),是一种创建UML类图,并能够自动生成J ...
- 命令运行带参数的jar
一.打包(此处用的是eclipse) 代码如下,此如引用了某博主的代码,因忘记地址,如博主发现此文,可私信我 package com.example.Open; import java.io.File ...
- SOCKET编程需要注意的问题
1.socket编程首先要做的就是加载库,有两种方法: 1.不需要加载库文件 if(!AfxSocketInit()) { AfxMessageBox("加载套接字库失败!"); ...
- 【Maven学习】Maven打包生成普通jar包、可运行jar包、包含所有依赖的jar包
http://blog.csdn.net/u013177446/article/details/54134394 ******************************************* ...
- 安卓手机数据恢复软件-DiskDigger Pro
以前的手机是支持大容量存储模式,可以被分配到盘符,但是自从手机不支持U盘大容量存储模式只能MTP模式之后,想要做数据恢复麻烦了很多啊! 经过多方查找,终于找到了这个能在手机上用的软件DiskDigge ...
- c++多态之——vptr指针
之前做过一个测试,在一个类中定义一个virtual修饰的函数时,sizeof这个类,发现类的大小多了恰好一个指针的字节大小,当初不明白,只是记住有这么一个特性.后来,发现它就是c++编译器给我们添加的 ...