docker镜像中文件丢失

背景介绍

笔者创建了一个镜像，然后在不同的主机上启动，发现有的能启动，有的却不行，报错信息为找不到文件。

犹记得当初有人介绍，只要docker镜像做好了，拿到任何地方都可以用，此处好像不成呢，好诡异的事。

初步怀疑docker存储驱动导致的问题。

制作镜像步骤还原

1、在某一台主机做dockerfile，该主机环境如下

[root@ ~]# docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 46
Server Version: 1.13.1
Storage Driver: btrfs
 Build Version: Btrfs v4.9.1
 Library Version: 102
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 9c3c5f853ebf0ffac0d087e94daef462133b69c7 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-514.16.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 40
Total Memory: 125.7 GiB
Name: cndh1325-10-12.ptmind.com
ID: 2AHA:F2JK:ECEX:U2WY:OAET:BFOS:CY77:MNAK:2Y2P:7DRS:KU72:6VI5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

[root@test ~]# xfs_info /
meta-data=/dev/sda5              isize=256    agcount=4, agsize=116621952 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=466487808, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=227777, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

可以看到，制作镜像的主机存储驱动为 Storage Driver: btrfs ，文件系统xfs不支持d_type。

将镜像上传镜像库，已提供使用。

2、在A主机拉取镜像，镜像不能启动

笔者是创建的elasticsearch:6.2.4版本镜像，从官方拉取，官方镜像默认包含ingest-geoip插件,在dockerfile中已经将插件删除

dockfile如下

FROM docker.elastic.co/elasticsearch/elasticsearch:6.2.4

COPY elasticsearch.yml /usr/share/elasticsearch/config/
COPY jvm.options /usr/share/elasticsearch/config/

RUN /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-geoip --purge \
    && /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-user-agent \
    && chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/config \

报错信息如下：

Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/ingest-geoip
Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/ingest-geoip
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)

A主机环境如下

[root@A ~]# docker info
Containers: 20
 Running: 20
 Paused: 0
 Stopped: 0
Images: 21
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: false
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 5eda6f6fd0c2884c2c8e78a6e7119e8d0ecedb77 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-862.9.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 16
Total Memory: 62.74 GiB
Name: jp33e505-10-14.ptengine.com
ID: PDSY:4YPU:AUSW:HA4V:223D:YBSG:E67E:TUJF:3WUJ:EL7J:VNGH:LG3O
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

[root@A ~]# xfs_info /
meta-data=/dev/sdc2              isize=256    agcount=4, agsize=6553600 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=26214400, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=12800, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

检查发现存储驱动为overlay2，文件系统xfs不支持d_type。

3、在B主机拉取镜像，镜像能启动

B主机环境如下

[root@B ~]# docker info
Containers: 21
 Running: 21
 Paused: 0
 Stopped: 0
Images: 22
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: df5c38a9167e87f53a9894d77c0950e178a745e7 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-957.10.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 16
Total Memory: 62.75 GiB
Name: jp33e506-10-19.ptengine.com
ID: OFVX:ELT3:RNT4:OM6W:L7GC:OC47:R5QJ:XBVL:T5HW:UGIL:PJUJ:66ZO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

[root@B ~]# xfs_info /
meta-data=/dev/md126p3           isize=512    agcount=4, agsize=7169408 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=28677632, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=14002, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

现在已经很明显了，A和B唯一不同的就是文件系统xfs是否支持d_type，而官方已明确说明如果使用xfs文件系统，需要d_type的支持，即 ftype=1，具体参考  https://docs.docker.com/storage/storagedriver/overlayfs-driver/

修改A主机的d_type

mkfs.xfs -f -n ftype=1 /dev/sdg5

重启docker，一切正常。