docker-run-failed

Phenomenon

overlay2 not such file or directory.

overlay2 not such file or directory.
network not such file or directory.

docker run ****

/usr/bin/docker-current: open /data0/docker-data/tmp/GetImageBlob742939759: no such file or directory.

Job for docker.service failed

[root@localhost file-center-wf-monitor]# systemctl  stopstop docker

[root@localhost file-center-wf-monitor]# service docker start
Redirecting to /bin/systemctl start docker.service
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

[root@localhost file-center-wf-monitor]# journalctl -xe
Apr 08 16:46:56 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 08 16:46:56 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
Apr 08 16:46:56 localhost.localdomain systemd[1]: Unit docker.service entered failed state.
Apr 08 16:46:56 localhost.localdomain systemd[1]: docker.service failed.
Apr 08 16:46:56 localhost.localdomain polkitd[378650]: Unregistered Authentication Agent for unix-process:615774:734710755 (system bus name :1.130780, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

[root@localhost file-center-wf-monitor]# rm -rf /data0/docker_images/*
rm: cannot remove ‘/data0/docker_images/containers’: Device or resource busy
[root@localhost file-center-wf-monitor]# mount | grep /data0/docker_images
/dev/sda7 on /data0/docker_images/containers type ext4 (rw,relatime,data=ordered)
/dev/sda7 on /data0/docker_images/overlay type ext4 (rw,relatime,data=ordered)
[root@localhost file-center-wf-monitor]# umount /data0/docker_images/containers
[root@localhost file-center-wf-monitor]# rm -rf /data0/docker_images

init/merged: invalid argument.

[root@localhost file-center-wf-monitor]# service docker start
Error response from daemon: Cannot kill container file-center-wf-monitor-20200408: No such container: file-center-wf-monitor-20200408
Error response from daemon: No such container: file-center-wf-monitor-20200408
/usr/bin/docker-current: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/dd9b026441e21f3b74bd00dbb9cba667129b8aba342413fa9fa56f68202b582b-init/merged: invalid argument.
See '/usr/bin/docker-current run --help'.

[root@localhost data0]# systemctl status docker.service
● docker.service - Docker Application Container Engine

{
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2020-04-08 16:54:09 CST; 6s ago
     Docs: http://docs.docker.com
  Process: 617751 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE)
 Main PID: 617751 (code=exited, status=1/FAILURE)

Apr 08 16:54:08 localhost.localdomain systemd[1]: Starting Docker Application Container Engine...
Apr 08 16:54:08 localhost.localdomain dockerd-current[617751]: time="2020-04-08T16:54:08.821786068+08:00" level=info msg="libcontainerd: new containerd process, pid: 617763"
Apr 08 16:54:09 localhost.localdomain dockerd-current[617751]: Error starting daemon: error initializing graphdriver: invalid argument
Apr 08 16:54:09 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 08 16:54:09 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
Apr 08 16:54:09 localhost.localdomain systemd[1]: Unit docker.service entered failed state.
Apr 08 16:54:09 localhost.localdomain systemd[1]: docker.service failed.

solution

• 停止docker

[root@localhost file-center-wf-monitor]# systemctl  stopstop docker

• 重装docker

  • 卸载docker及关键依赖

    yum list installed | grep docker
    yum remove docker-engine.x86_64
    yum remove docker-engine-selinux.noarch
    yum remove docker-engine.x86_64 docker-client.x86_64 docker-common.x86_64
    yum remove container-selinux.noarch
    

    • 删除docker 相关文件

    rm -rf /var/lib/docker/
    rm -rf /var/lib/docker
    rm -rf /data0/docker_images/overlay2/*
    rm -rf /data0/docker_images/*
    rm -rf /etc/sysconfig/docker*
    rm -rf /etc/systemd/system/docker.service.d
    

    • 不卸载会导致安装不上

    Package(s) docker available, but not installed.
    

    • 不删除可能导致卸载不完全

  • 安装docker

    yum -y install docker
    

• 调整配置

[root@localhost file-center-wf-monitor]# vim /etc/docker/daemon.json
{
    "graph": "/data0/docker_images",
    "storage-driver": "overlay"
}

• 重载服务

systemctl daemon-reload

• 重启docker

service [start|restart] docker

• 查看docker状态

systemctl status docker

conclusion

• 本次问题根源是因为机器上某目录不可用，推测是由于历史docker版本残留导致
• 调整docker graph后，问题解决。
• 本文记录为解决当前问题过程中，所遇到的一系列问题和对应的操作。
• yum安装docker，如果机器比较干净，安装会很顺利；如果历史安装过docker，会出现一系列问题。可以分别卸载docker的子package，并手动清除残留信息解决。
• docker 镜像存储路径一定调整到系统数据盘，不然分分钟打满系统盘。
• journalctl -xe 可以看到service信息（docker/mesos）
• docker invalid argument不一定是启动的参数错误，一般不必调整/etc/systemd/system/docker.service.d/storage.conf 、/etc/sysconfig/docker-storage、/etc/sysconfig/docker 和 /etc/docker/daemon.json等文件。
• /etc/sysconfig/docker配置可以关闭docker的selinux模式

# OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
OPTIONS='--log-driver=journald --signature-verification=false'

• /etc/sysconfig/docker-storage配置可以调整docker存储（挂载目录）模式

DOCKER_STORAGE_OPTIONS=

• /etc/sysconfig/docker-network配置可以调整docker网络模式
• 通过/etc/systemd/system/docker.service.d/***.conf下的对应文件进行配置，同样生效
• docker启动时，本地目录权限、状态、挂载都应确认清楚，避免发生异常。