Linux System Programming 学习笔记(二) 文件I/O

1.每个Linux进程都有一个最大打开文件数，默认情况下，最大值是1024

文件描述符不仅可以引用普通文件，也可以引用套接字socket，目录，管道(everything is a file)

默认情况下，子进程会获得其父进程文件表的完整拷贝

 

2.打开文件

open系统调用必须包含 O_RDONLY，O_WRONLY，O_RDWR 三种存取模式之一

注意 O_NONBLOCK模式

int fd = open(filename, O_WRONLY | O_CREAT | O_TRUNC, )
int fd = creat(filename, )

3.读文件

read系统调用会有以下结果：

(1)返回值与请求数len相同，所有len字节都存储在buf内

 

(2)返回值小于请求数len，但是大于0。发生此种情况有很多原因：

a.read系统调用中途被信号中断

b.read系统调用中途发生错误

c.可读字节数大于0，但是小于len

d.在读完len字节之前遇到EOF

 

(3)返回0，表示EOF

(4)调用被阻塞，因为当前没有可读，这种情况不会发生在非阻塞模式

(5)返回-1，errno设置为EINTR，表示在读任一字节之前就接收到信号

(6)返回-1，errno设置为EAGAIN，表示读操作被阻塞，因为当前并没有可读字节，这只发生在 非阻塞模式

(7)返回-1，errno设置为 EINTR，EAGAIN之外的值，表示发生其他更严重的错误

 

读完所有字节：

size_t readn(int fd, void* buf, size_t len)
{
    size_t  tmp = len;
    ssize_t ret = ;
    while (len !=  && (ret = read(fd, buf, len)) != ) {
        if (ret == -) {
            if (errno == EINTR) {
                continue;
            }
            fprintf(stderr, "read error\n");
            break;
        }
        len -= ret;
        buf += ret;
    }
    return tmp - len;
}

非阻塞读：

有时我们并不希望当没有可读数据时read系统调用被阻塞，而是希望调用可以立即返回，表明没有数据可读，这就是非阻塞I/O

 

4.写文件

write系统调用没有EOF，对于普通文件，write默认操作是全部写，除非是发生错误返回-1

对于其他文件就有可能发生部分写，最典型的是网络编程中socket读写时，应该如下写：

size_t writen(int fd, void* buf, size_t len)
{
    ssize_t ret = ;
    size_t  tmp = len;
    while (len !=  && (ret = write(fd, buf, len)) != ) {
        if (ret == -) {
            if (errno == EINTR) {
                continue;
            }
            fprintf(stderr, "write error\n");
            break;
        }
        len -= ret;
        buf += ret;
    }
    return tmp - len;
}

追加模式可以确保文件的当前位置总是位于文件末尾，并且可以把文件偏移更新操作看成原子操作，所以该模式对于多任务追加写非常有用

 

5.文件同步

当调用write时，内核从用户buffer拷贝数据到内核buffer，但并不一定是立即写到目的地，内核通常是执行一些检查，将数据从用户buffer拷贝到一个dirty buffer，后而内核收集所有这些dirty buffer(contain data newer than what is on disk)，最后才写回磁盘。

这种延迟写并没有改变POSIX语义，反而可以提升读写性能

if a read is issued for just-written data that lives in a dirty buffer and is not yet on disk, the request will be satisfied from the buffer and not cause a read from the "stale" data on disk. so the read is satisfied from an in-memory cache without having to go to disk.

 

延迟写可以大幅提升性能，但是有时候需要控制写回磁盘的文件，这是需要确保文件同步

fsync系统调用确保fd关联的文件数据已经写回到磁盘

int ret = fsync(fd);

open调用时 O_SYNC标志表示 文件必须同步

int fd = open(file, O_WRONLY | O_SYNC);

O_SYNC导致I/O等待时间消耗巨大，一般地，需要确保文件写回到磁盘时我们使用 fsync函数

6.文件定位

显式的文件定位函数：

a. 将文件偏移定位到1825

off_t ret = lseek(fd, (off_t)1825, SEEK_SET);

b. 将文件便宜定位到文件末尾处

off_t ret = lseek(fd, 0, SEEK_END);

c. 将文件偏移定位到文件开始处

off_t ret = lseek(fd, 0, SEEK_CUR)

文件定位是可以超出文件末尾的，此时对该文件写操作会填补0，形成空洞，空洞是不占有物理磁盘空间的。

This implies that the total size of all files on a filesystem can add up to more than the physical size of the disk

 

7.截断文件

int ftruncate(int fd, off_t len);  

将给定文件截断为给定长度，这里的给定长度是可以小于文件大小，也可以大于文件大小(会造成空洞)

8.多路I/O

阻塞I/O：如果read系统调用时，文件(例如管道输入)没有可读数据，这时进程会一直阻塞等待，直到有可读数据。效率低下，不能同时进行多个文件读写操作

多路I/O可以允许程序并发地阻塞在多个文件上，并且当任一文件变为可读或可写的时候会立马接收到通知

Multiplexed I/O becomes the pivot point for the application，designed similarly to the following activity：
a. Multiplexed I/O ： Tell me when any of these file descriptors becomes ready for I/O
b. Nothing ready? Sleep until one or more file descriptors are ready.
c. Woken up ! What is ready?
d. Handle all file descriptors ready for I/O, without bolocking
e. Go back to step a

9.select

int select(int nfds, fd_set* readfds, fd_set* writefds, fd_set* exceptfds, struct timeval* timeout);
FD_CLR(int fd, fd_set* set);      // removes a fd from a given set
FD_ISSET(int fd, fd_set* set);   // test whether a fd is part of a given set
FD_SET(int fd, fd_set* set);      // adds a fd to a given set
FD_ZERO(int fd, fd_set* set);  // removes all fds from specified set. shoule be called before every invocation of select()

因为fd_set是静态分配的，系统有一个文件描述符的最大打开数 FD_SETSIZE，在Linux中，该值为 1024

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>

#define TIMEOUT 5     /* select timeout in seconds */
#define BUFLEN  1024  /* read buffer in bytes */

int main(int argc, char* argv[])
{
    struct timeval tv;
    tv.tv_sec = TIMEOUT;
    tv.tv_usec = ;

    /* wait on stdin for input */
    fd_set readfds;
    FD_ZERO(&readfds);
    FD_SET(STDIN_FILENO, &readfds);

    int ret = select(STDIN_FILENO + , &readfds, NULL, NULL, &tv);
    if (ret == -) {
        fprintf(stderr, "select error\n");
        return ;
    } else if (!ret) {
        fprintf(stderr, "%d seconds elapsed.\n", TIMEOUT);
        return ;
    }
    if (FD_ISSET(STDIN_FILENO, &readfds)) {
        char buf[BUFLEN + ];
        int len = read(STDIN_FILENO, buf, BUFLEN);
        if (len == -) {
            fprintf(stderr, "read error\n");
            return ;
        }
        if (len != ) {
            buf[BUFLEN] = '\0';
            fprintf(stdout, "read:%s\n", buf);
        }
        return ;
    } else {
        fprintf(stderr, "This should not happen\n");
        return ;
    }

}

10. poll

int poll(struct pollfd* fds, nfds_t  nfds, int timeout);

This is a program that uses poll() to check whether a read from stdin and a write to stdout will block

#include <unistd.h>
#include <poll.h>

#define TIMEOUT 5

int main(int argc, char* argv[])
{
    struct pollfd fds[];

    /* watch stdin for input */
    fds[].fd = STDIN_FILENO;
    fds[].events = POLLIN;

    /* watch stdout for alibity to write */
    fds[].fd = STDOUT_FILENO;
    fds[].events = POLLOUT;

    int ret = poll(fds, , TIMEOUT * );
    if (ret == -) {
        fprintf(stderr, "poll error\n");
        return ;
    }

    if (!ret) {
        fprintf(stdout, "%d seconds elapsed.\n", TIMEOUT);
        return ;
    }

    if (fds[].revents & POLLIN) {
        fprintf(stdout, "stdin is readable\n");
    }
    if (fds[].revents & POLLOUT) {
        fprintf(stdout, "stdout is writable\n");
    }
    return ;
}

poll vs select 

a. poll不需要用户计算并传递文件描述符参数(select中必须将该值设为最大描述符数加1)

b. select的fd_set是静态分配的，有一个最大文件数限制FD_SETSIZE，poll就没有这个限制，只需要创建一个合适大小的结构体数组

c. select移植性更好，支持select的unix更多

d. select支持更精细的timeout，poll只支持毫秒

11.内核实现

Linux内核主要由 virtual filesystem, page cache, page write-back 来支持有效且强大的I/O机制

(1) virtual filesystem

The virtual filesystem (also called a virtual file switch) is a mechanism of abstraction that allows the Linux kernel to call filesystem functions and manipulate filesystem data without knowing the specific type of filesystem being used.

So, a single system call can read any filesystem on any medium, All filesystems support the same concepts, the same interfaces, and the same calls

 

(2) page cache

The page cache is an in-memory store of recently accessed data from an on-disk filesystem.

Storing requested data in memory allows the kernel to fulfill subsequent requests for the same data  from memory, avoiding repeated disk access

The page cache exploits the concept of temporal locality, which says that a resource accessed at one point has a high probability of being accessed again in the near future

 

时间局部性：

The page cache is the first place that kernel looks for filesystem data. The first time any item of sata is read, it is transferred from the disk into the page cache, and is returned to the application from the cache. 

 

空间局部性：

The data is often referenced sequentially. The kernel implements page cache  readahead（预读）. Readahead is the act of reading extra data off the disk and into the page cache following each read request. In effect, reading a little bit ahead. 

 

(3) page write-back

When a process issues a write request, the data is copied into a buffer, and the buffer is marked dirty, denoting that the in-memory copy is newer than the on-disk copy.

Eventually, the dirty buffers need to be committed to disk, sync the on-disk files with the data in memory.