openshift上使用devicemapper

环境：openshift v3.6.173.0.5

openshift上devicemapper与官方文档中的描述略有不同，在官方文档的描述中，容器使用的lvm文件系统挂载在/var/lib/devicemapper下，因此可以在/var/lib/devicemapper/metadata中找到容器对应的devicemapper的配置信息，在/var/lib/devicemapper/mnt中可以看到容器的文件系统，描述如下：

The /var/lib/docker/devicemapper/metadata/ directory contains metadata about the Devicemapper configuration itself and about each image and container layer that exist. The devicemapper storage driver uses snapshots, and this metadata include information about those snapshots. These files are in JSON format.

The /var/lib/devicemapper/mnt/ directory contains a mount point for each image and container layer that exists. Image layer mount points are empty, but a container’s mount point shows the container’s filesystem as it appears from within the container.

在openshfit中使用docker inspect查看一个容器的devicemapper信息如下，使用的块设备为 docker-253:0-101504694-13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f

"GraphDriver": {
    "Name": "devicemapper",
    "Data": {
        "DeviceId": "",
        "DeviceName": "docker-253:0-101504694-13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f",
        "DeviceSize": ""
    }
},

但在/var/lib/docker/devicemapper/mnt中该容器对应的文件挂载为空，系统mount命令也查找不到该容器对应的块设备挂载信息

# pwd
/var/lib/docker/devicemapper
# du -d |grep 13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f
       ./mnt/13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f-init
       ./mnt/13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f

查看该容器对应的进程，pid为19422

# docker inspect 17ba06eb4578|grep Pid
  "Pid": ,
  "PidMode": "",
  "PidsLimit": ,

在/proc/19422/ns中可以看到其命名空间相关的信息，对比pid 为1的mnt ns(mnt -> mnt:[4026531840])，可以看到该容器与根进程不属于同一个mount 命名空间，因此在根进程所在的mount命名空间中无法查看到该容器的挂载信息

# ll
total
lrwxrwxrwx.   root  Apr  : ipc -> ipc:[]
lrwxrwxrwx.   root  Apr  : mnt -> mnt:[]
lrwxrwxrwx.   root  Apr  : net -> net:[]
lrwxrwxrwx.   root  Apr  : pid -> pid:[]
lrwxrwxrwx.   root  Apr  : user -> user:[]
lrwxrwxrwx.   root  Apr  : uts -> uts:[]

使用nsenter -t 19422 -m mnt -u命令进入到进程所在的mount和uts命名空间，使用mount命令可以看到容器的块设备挂载到了该mount命名空间的根目录

/dev/mapper/docker-:--13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f on / type xfs (rw,relatime,context="system_u:object_r:container_file_t:s0:c2,c3",nouuid,attr2,inode64,sunit=,swidth=,noquota)

当然也可以在/proc/19422/mounts和/proc/19422/mountinfo中看到与该进程相关的mount信息，其中也包括容器的块设备挂载信息

使用lsns可以看系统中的命名空间与进程的对应关系，下面列出了pid为19422的相关命名空间，NPROCS表示该命名空间下面的进程数目，PID表示该命名空间下的最小PID值。可以看出19422与根进程属于同一个user命名空间。其中mnt命名空间中有2个进程，一个是19422，另一个是19422的父进程(此处为java)。因此在容器异常退出后，可以通过进入未退出进程的命名空间定位问题。

[root@lab-node1 proc]# lsns -p
        NS TYPE  NPROCS   PID USER       COMMAND
 user           root       /usr/lib/systemd/systemd --switched-root --system --deserialize
 ipc                 /usr/bin/pod
 net                 /usr/bin/pod
 mnt           /bin/sh /opt/eap/bin/standalone.sh -Djavax.net.ssl.keyStore=/opt/hawkular/auth/hawkular-metrics.keystore -Djavax.net.ssl.tru
 uts           /bin/sh /opt/eap/bin/standalone.sh -Djavax.net.ssl.keyStore=/opt/hawkular/auth/hawkular-metrics.keystore -Djavax.net.ssl.tru
 pid           /bin/sh /opt/eap/bin/standalone.sh -Djavax.net.ssl.keyStore=/opt/hawkular/auth/hawkular-metrics.keystore -Djavax.net.ssl.tru

参考：

https://xuxinkun.github.io/2019/04/02/deviemapper-docker/

https://docs.docker.com/engine/reference/commandline/dockerd/