Notes of O_DIRECT flag

What is O_DIRECT

Starting with kernel 2.4, Linux allows an application to bypass the buffer cache when performing disk I/O, thus transferring data directly from user space to a file or disk device. This is sometimes termed direct I/O or raw I/O.

Direct I/O is sometimes misunderstood as being a means of obtaining fast I/O performance. However, for most applications, using direct I/O can considerably degrade performance. This is because the kernel applies a number of optimiza- tions to improve the performance of I/O done via the buffer cache, including per- forming sequential read-ahead, performing I/O in clusters of disk blocks, and allowing processes accessing the same file to share buffers in the cache. All of these optimizations are lost when we use direct I/O. Direct I/O is intended only for applications with specialized I/O requirements. For example, database systems that perform their own caching and I/O optimizations don’t need the kernel to consume CPU time and memory performing the same tasks.

We can perform direct I/O either on an individual file or on a block device (e.g., a disk). To do this, we specify the O_DIRECT flag when opening the file or device with open().

The O_DIRECT flag is effective since kernel 2.4.10. Not all Linux file systems and kernel versions support the use of this flag. Most native file systems support O_DIRECT, but many non-UNIX file systems (e.g., VFAT) do not. It may be necessary to test the file system concerned (if a file system doesn’t support O_DIRECT, then open() fails with the error EINVAL) or read the kernel source code to check for this support.

If a file is opened with O_DIRECT by one process, and opened normally (i.e., so that the buffer cache is used) by another process, then there is no coherency between the contents of the buffer cache and the data read or written via direct I/O. Such scenarios should be avoided.

The raw(8) manual page describes an older (now deprecated) technique for obtaining raw access to a disk device.

Alignment restrictions for direct I/O

Because direct I/O (on both disk devices and files) involves direct access to the disk, we must observe a number of restrictions when performing I/O:

• The data buffer being transferred must be aligned on a memory boundary that is a multiple of the block size.

• The file or device offset at which data transfer commences must be a multiple of the block size.

• The length of the data to be transferred must be a multiple of the block size.

Failure to observe any of these restrictions results in the error EINVAL. In the above list, block size means the physical block size of the device (typically 512 bytes).

Notes of O_DIRECT

1，数据对齐

从2.6.0内核开始，O_DIRECT要求的对齐基本单位是底层块设备的逻辑块大小（Logical Block Size），此处一定要注意，是底层块设备的逻辑块大小，而不是文件系统（如ext4）的block size。底层块设备的块大小也叫Sector Size，可以用两种方式获取它：

一种是使用ioctl系统调用的BLKSSZGET指令来获取:

#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/ioctl.h>
#include <sys/mount.h>

void getblocksize(char *dev)
{
    std::cout << "Params for " << dev << std::endl;
    int fd;
    fd = open(dev, O_RDWR);
    if (fd == -1) {
        std::cout << "open error " << errno << std::endl;
        return;
    }

    long size = 0;
    if (ioctl(fd, BLKSSZGET, &size) >= 0)
        std::cout << "BLKSSZGET: " << size << std::endl;
    else
        std::cout << "error BLKSSZGET " << errno << std::endl;
    close(fd);
}

另一种是脚本:

blockdev --getss

值得一提的是，blockdev命令有两个block size：

#blockdev --help

Usage:
 blockdev -V
 blockdev --report [devices]
 blockdev [-v|-q] commands devices

Available commands:
 --getsz                   get size in 512-byte sectors
 --setro                   set read-only
 --setrw                   set read-write
 --getro                   get read-only
 --getdiscardzeroes        get discard zeroes support status
 --getss                   get logical block (sector) size
 --getpbsz                 get physical block (sector) size
 --getiomin                get minimum I/O size
 --getioopt                get optimal I/O size
 --getalignoff             get alignment offset in bytes
 --getmaxsect              get max sectors per request
 --getbsz                  get blocksize
 --setbsz <bytes>          set blocksize on file descriptor opening the block device
 --getsize                 get 32-bit sector count (deprecated, use --getsz)
 --getsize64               get size in bytes
 --setra <sectors>         set readahead
 --getra                   get readahead
 --setfra <sectors>        set filesystem readahead
 --getfra                  get filesystem readahead
 --flushbufs               flush buffers
 --rereadpt                reread partition table

　--getss和--getpbsz是指获取设备的逻辑块大小和物理块大小，sector的含义：

Mass storage devices (hard disks, CD-ROMs, tapes) operate on chunks of data, usually called sectors. The size of these device sectors varies, but is fixed for any one device. Hard disks and floppies usually use 512 bytes, while data CDs and DVDs use 2048 bytes. Today, it is customary to number all sectors sequentially and leave the details to the device.

--getbsz是指文件系统的逻辑块大小，block的含义：

File systems also operate on chunks at a time, but they don't need to be the same size as the device's sectors. The chunks used by the file system are usually called blocks, but cluster, allocation block, and allocation unit are also common.

O_DIRECT对齐的单位就是上面的sector size。而不是操作系统的block size。这个地方大家往往混淆为操作系统的blocksize（尽管这样做O_DIRECT不会报错）。

2，并行操作

Applications should avoid mixing O_DIRECT and normal I/O to the same file, and especially to overlapping byte regions in the same file. Even when the filesystem correctly handles the coherency issues in this situation, overall I/O throughput is likely to be slower than using either mode alone. Likewise, applications should avoid mixing mmap(2) of files with direct I/O to the same files.

3，O_DIRECT和fork

O_DIRECT I/Os should never be run concurrently with the fork(2)
system call, if the memory buffer is a private mapping (i.e., any
mapping created with the mmap(2) MAP_PRIVATE flag; this includes
memory allocated on the heap and statically allocated buffers). Any
such I/Os, whether submitted via an asynchronous I/O interface or
from another thread in the process, should be completed before
fork(2) is called. Failure to do so can result in data corruption
and undefined behavior in parent and child processes. This
restriction does not apply when the memory buffer for the O_DIRECT
I/Os was created using shmat(2) or mmap(2) with the MAP_SHARED flag.
Nor does this restriction apply when the memory buffer has been
advised as MADV_DONTFORK with madvise(2), ensuring that it will not
be available to the child after fork(2).