透过 Go 语言探索 Linux 网络通信的本质

前言

各种编程语言百花齐放、百家争鸣，但是 “万变不离其中”。对于网络通信而言，每一种编程语言的实现方式都不一样；但其实，调用的底层逻辑都是一样的。linux 系统底层向上提供了统一的 Socket 通信系统函数，动态链接库 /lib64/libc.so 中就是实现网络通信的关键类库。下面我们会以 Go 语言为例，来分析网络通信数据传输的路径；最终揭开各大编程语言网络通信的神秘面纱。

演示程序

1、使用 Go 编写一个简单的 Socket 程序

package main

import (
    "fmt"
    "net"
)

func main() {
    // 监听本地端口
    listener, err := net.Listen("tcp", ":8080")
    if err != nil {
        fmt.Println("Error listening:", err.Error())
        return
    }
    defer listener.Close()

    fmt.Println("Listening on localhost:8080")

    for {
        // 接收客户端连接
        conn, err := listener.Accept()
        if err != nil {
            fmt.Println("Error accepting:", err.Error())
            return
        }

        // 处理客户端请求
        go handleRequest(conn)
    }
}

func handleRequest(conn net.Conn) {
    // 读取请求数据
    buffer := make([]byte, 1024)
    n, err := conn.Read(buffer)
    if err != nil {
        fmt.Println("Error reading:", err.Error())
        return
    }

    // 处理请求数据
    message := string(buffer[:n])
    fmt.Println("Received message:", message)

    // 发送响应数据
    reply := "Hello, client!"
    conn.Write([]byte(reply))

    // 关闭连接
    conn.Close()
}

2、编译成二进制文件。

go build main.go

[yxh@dev01 demo]$ ls -l
total 2536
-rwxr-xr-x 1 yangxionghai devops 2590837 Jun  2 15:42 main
-rw-r--r-- 1 yangxionghai devops    1023 Jun  2 15:39 main.go

3、执行 main 二进制文件

[yxh@dev01 demo]$ ./main
Listening on localhost:8080

跟踪进程数据

1、跟踪 main 进程

# 找到进程ID
[yxh@dev01 demo]$ ps -aux | grep main
yxh+ 32270  0.0  0.0 816460  1732 pts/3    Sl+  16:47   0:00 ./main
yxh+ 32404  0.0  0.0 112816   968 pts/13   S+   16:47   0:00 grep --color=auto main

2、使用 strace 跟踪进程

strace -f -s 2048 -i -T -o trace.log -p 32270

命令各参数的解释：
-f: 跟踪在运行时从父进程派生出来的子进程，包括进程创建和退出等操作。
-s: 指定在输出中显示的字符串的最大长度为 2048 字节，这样可以避免过长的输出导致日志文件过大。
-i: 在输出中同时显示系统调用的入口和返回地址。
-T: 在输出中显示每个系统调用花费的时间。
-o: 将输出的结果写入到名为 trace.log 的文件中，而不是输出到控制台。
-p: 指定要跟踪的进程 ID 是 32270。

3、使用 telnet 发生数据

# 客户端发送数据
[yxh@dev01 ~]$ telnet 127.0.0.1 8080
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
^]

telnet>
# 客户端发生的数据
hello world
# 服务端返回的数据
Hello, client!
Connection closed by foreign host.

4、服务端接收到数据

# 服务端接收到数据
[yxh@dev01 demo]$ ./main
Listening on localhost:8080
Received message: hello world

深度分析

1、分析跟踪信息

我们先分析刚刚使用 strace 工具跟踪到的信息，可以看到里面有很多的系统调用。具体每个系统函数的说明及用法，可以去搜索引擎上查找资料学习。

# 跟踪到的信息
[yxh@dev01 demo]$ cat trace.log
32275 [0000000000462943] futex(0x5e0058, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
32274 [0000000000462943] futex(0xc000080150, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
32273 [0000000000462943] futex(0xc000044d50, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
32272 [0000000000462943] futex(0xc000044950, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
32271 [0000000000462943] restart_syscall(<... resuming interrupted restart_syscall ...> <unfinished ...>
32270 [0000000000462b60] epoll_pwait(5,  <unfinished ...>
32271 [0000000000462943] <... restart_syscall resumed>) = -1 ETIMEDOUT (Connection timed out) <25.495631>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0 <0.000107>
32271 [0000000000462943] futex(0x5b1bd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0}) = -1 ETIMEDOUT (Connection timed out) <60.000108>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0 <0.000100>
32271 [0000000000462943] futex(0x5b1bd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0}) = -1 ETIMEDOUT (Connection timed out) <60.000112>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0 <0.000123>
32271 [0000000000462943] futex(0x5b1bd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
32270 [0000000000462b60] <... epoll_pwait resumed>[{EPOLLIN, {u32=2087155416, u64=140464697603800}}], 128, -1, NULL, 0) = 1 <188.828350>
32270 [0000000000462943] futex(0x5b1bd8, FUTEX_WAKE_PRIVATE, 1) = 1 <0.000028>
32271 [0000000000462943] <... futex resumed>) = 0 <43.331766>
32270 [000000000047f08a] accept4(3,  <unfinished ...>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
32270 [000000000047f08a] <... accept4 resumed>{sa_family=AF_INET6, sin6_port=htons(9622), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [112->28], SOCK_CLOEXEC|SOCK_NONBLOCK) = 4 <0.000019>
32271 [00000000004623bd] <... nanosleep resumed>NULL) = 0 <0.000076>
32270 [0000000000462b38] epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2087155184, u64=140464697603568}} <unfinished ...>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
32270 [0000000000462b38] <... epoll_ctl resumed>) = 0 <0.000017>
32270 [000000000047f0f6] getsockname(4, {sa_family=AF_INET6, sin6_port=htons(8080), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [112->28]) = 0 <0.000017>
32270 [000000000047f08a] setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4 <unfinished ...>
32271 [00000000004623bd] <... nanosleep resumed>NULL) = 0 <0.000103>
32270 [000000000047f08a] <... setsockopt resumed>) = 0 <0.000016>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
32270 [000000000047f08a] setsockopt(4, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0 <0.000017>
32270 [000000000047f08a] setsockopt(4, SOL_TCP, TCP_KEEPINTVL, [15], 4) = 0 <0.000017>
32271 [00000000004623bd] <... nanosleep resumed>NULL) = 0 <0.000090>
32270 [000000000047f08a] setsockopt(4, SOL_TCP, TCP_KEEPIDLE, [15], 4 <unfinished ...>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
32270 [000000000047f08a] <... setsockopt resumed>) = 0 <0.000016>
32270 [0000000000462943] futex(0xc000080150, FUTEX_WAKE_PRIVATE, 1) = 1 <0.000020>
32274 [0000000000462943] <... futex resumed>) = 0 <188.828982>
32270 [000000000047f08a] accept4(3,  <unfinished ...>
32274 [0000000000462b60] epoll_pwait(5,  <unfinished ...>
32271 [00000000004623bd] <... nanosleep resumed>NULL) = 0 <0.000101>
32274 [0000000000462b60] <... epoll_pwait resumed>[{EPOLLOUT, {u32=2087155184, u64=140464697603568}}], 128, 0, NULL, 0) = 1 <0.000014>
32270 [000000000047f08a] <... accept4 resumed>0xc000053c10, [112], SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable) <0.000037>
32274 [0000000000462943] futex(0xc000044950, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
32274 [0000000000462943] <... futex resumed>) = 1 <0.000018>
32270 [0000000000462b60] epoll_pwait(5,  <unfinished ...>
32274 [000000000047f01b] read(4,  <unfinished ...>
32272 [0000000000462943] <... futex resumed>) = 0 <188.829100>
32274 [000000000047f01b] <... read resumed>0xc00008e400, 1024) = -1 EAGAIN (Resource temporarily unavailable) <0.000017>
32270 [0000000000462b60] <... epoll_pwait resumed>[], 128, 0, NULL, 2) = 0 <0.000039>
32274 [0000000000462b60] epoll_pwait(5,  <unfinished ...>
32272 [0000000000462b60] epoll_pwait(5,  <unfinished ...>
32274 [0000000000462b60] <... epoll_pwait resumed>[], 128, 0, NULL, 2) = 0 <0.000019>
32271 [00000000004623bd] <... nanosleep resumed>NULL) = 0 <0.000120>
32270 [0000000000462b60] epoll_pwait(5,  <unfinished ...>
32274 [0000000000462943] futex(0xc000080150, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
32272 [0000000000462b60] <... epoll_pwait resumed>[], 128, 0, NULL, 0) = 0 <0.000054>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
32272 [0000000000462943] futex(0xc000044950, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
32271 [00000000004623bd] <... nanosleep resumed>NULL) = 0 <0.000112>
32271 [0000000000462943] futex(0x5b1bd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
32270 [0000000000462b60] <... epoll_pwait resumed>[{EPOLLIN|EPOLLOUT, {u32=2087155184, u64=140464697603568}}], 128, -1, NULL, 0) = 1 <8.305983>
32270 [0000000000462943] futex(0x5b1bd8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
32271 [0000000000462943] <... futex resumed>) = 0 <8.305878>
32270 [0000000000462943] <... futex resumed>) = 1 <0.000035>
32271 [0000000000462aa7] sched_yield( <unfinished ...>
32270 [000000000047f01b] read(4,  <unfinished ...>
32271 [0000000000462aa7] <... sched_yield resumed>) = 0 <0.000015>
# 接收到来自客户端的数据
32270 [000000000047f01b] <... read resumed>"hello world\r\n", 1024) = 13 <0.000015>
32271 [0000000000462943] futex(0x5b1ad8, FUTEX_WAKE_PRIVATE, 1) = 0 <0.000021>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
32270 [000000000047f01b] write(1, "Received message: hello world\r\n\n", 32) = 32 <0.000025>
# 服务端返回给客户端的数据
32270 [000000000047f01b] write(4, "Hello, client!", 14 <unfinished ...>
32271 [00000000004623bd] <... nanosleep resumed>NULL) = 0 <0.000092>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
32270 [000000000047f01b] <... write resumed>) = 14 <0.000040>
32270 [0000000000462b38] epoll_ctl(5, EPOLL_CTL_DEL, 4, 0xc00004ee3c <unfinished ...>
32271 [00000000004623bd] <... nanosleep resumed>NULL) = 0 <0.000092>
32270 [0000000000462b38] <... epoll_ctl resumed>) = 0 <0.000067>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
32270 [000000000047f01b] close(4)       = 0 <0.000033>
32270 [0000000000462b60] epoll_pwait(5, [], 128, 0, NULL, 824634044416) = 0 <0.000017>
32271 [00000000004623bd] <... nanosleep resumed>NULL) = 0 <0.000101>
32270 [0000000000462b60] epoll_pwait(5,  <unfinished ...>
32271 [0000000000462943] futex(0x5b1bd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0}) = -1 ETIMEDOUT (Connection timed out) <60.000106>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0 <0.000116>
32271 [0000000000462943] futex(0x5b1bd8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0}) = -1 ETIMEDOUT (Connection timed out) <60.000107>
32271 [00000000004623bd] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0 <0.000119>

strace 命令是 Linux 系统下的一个系统调用跟踪工具，其主要作用是打印出程序执行时调用的所有系统调用以及相应的返回值。

2、分析 main 编译的可执行二进制文件

使用 ldd 查看二进制文件的动态链接调用库。

[yxh@dev01 demo]$ ldd main
	linux-vdso.so.1 =>  (0x00007ffd761d2000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6cb5857000)
	# libc.so 库的主要作用是为开发者提供一些常用的、通用的基础函数，例如字符串处理、文件操作、进程管理、网络通信等，同时也提供了一些系统调用的封装接口。
	libc.so.6 => /lib64/libc.so.6 (0x00007f6cb5489000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6cb5a73000)

ldd 命令是 Linux 系统下的一个动态链接库依赖检查工具，用于显示可执行程序或共享库文件所依赖的动态链接库列表。

分析 libc.so 动态链接库

[yxh@dev01 demo]$ nm /lib64/libc.so.6 | grep read
000000000010c550 W pthread_setcancelstate
000000000010c580 T pthread_setcanceltype
000000000010c430 T pthread_setschedparam
000000000013e6c0 t __pthread_unwind
# 读函数
00000000000ef990 W read
00000000000ef990 W __read
00000000000fe9a0 W readahead
00000000000fe9a0 t __readahead
0000000000033180 t read_alias_file

[yxh@dev01 demo]$ nm /lib64/libc.so.6 | grep write
00000000000f5510 T pwritev
00000000000f5510 T pwritev64
# 写函数
00000000000ef9f0 W write
00000000000ef9f0 W __write
0000000000100250 t write_gmon
00000000000ef9f9 t __write_nocancel
00000000001009a0 t __write_profiling
000000000012f440 t writetcp

nm 是一个 Linux 系统下的二进制文件分析工具，用于查看目标文件或者可执行文件的符号表信息以及相关的重定位信息等。

除了 read、write 还有 accept、sendto、recvfrom、setsockopt、getsockopt、epoll 等函数。

总结

唯一不变的是变化，新技术层出不穷。对于我们技术人来说，不断学习新的技术是永无止境的，时间长了会陷入疲惫不堪。我们只有在不断变化中 “寻找不变化的东西”，通过掌握本质的东西，来以不变来应万变。这篇文章以 Go 语言为例，来逐步的从应用层到系统层的跟踪剖析，挖掘网络通信的本质，深入了解 Socket 通信的底层逻辑。希望大家可以以本文中的 Go 语言为例，举一反三。如果有什么问题，可以评论留言。