Docker Registry V2 with Nginx

安装 nginx

修改/etc/yum.repos.d/nginx.repo

[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

其中 releasever basearch 到 http://nginx.org/packages/centos/ 地址上去查

yum install nginx

配置ssl证书

注意CN的地址要与你的registry地址一致，可以修改/etc/hosts来绑定一个地址

修改 /etc/docker/daemon.json

增加 "insecure-registries":["xxx(CN里写的地址):port"]

正常启动registry 不需要任何ssl参数，权限交由nginx控制

docker run -d -p 6000:5000 --restart=always --name registry registry:2.5.1

配置nginx

user  root root;
worker_processes  1;

error_log   /var/log/nginx/error.log debug;
#pid         /var/log/nginx/nginx.pid;

worker_rlimit_nofile 51200;

events {
    use epoll;
    worker_connections  512;
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$http_host $remote_user [$time_local] $request '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" $remote_addr $request_time $upstream_response_time';

    access_log  /var/log/nginx/access.log  main;

    sendfile       on;
    tcp_nopush     on;
    tcp_nodelay    on;

    keepalive_timeout  0;
    #keepalive_timeout  65;

    #gzip  on;

    upstream registry {
        server tf56:6000;
    }

    server {
        listen       8080;
        server_name  registry.lenovo.com;

        ssl          on;
        ssl_certificate /root/registry/certs/registry.lenovo.com.crt;
        ssl_certificate_key /root/registry/certs/registry.lenovo.com.key;
        ssl_client_certificate /root/registry/certs/registry.lenovo.com.crt;

        # Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
        ssl_protocols TLSv1.1 TLSv1.2;
        ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;

        client_max_body_size 0;

        chunked_transfer_encoding on;

        location / {
          auth_basic "Registry realm";
          auth_basic_user_file /root/registry/nginx/nginx.htpasswd;
          add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

          proxy_pass                          http://registry;
          proxy_set_header  Host              $http_host;   # required for docker client's sake
          proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
          proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
          proxy_set_header  Authorization     ""; # see https://github.com/dotcloud/docker-registry/issues/170
          proxy_read_timeout                  900;
         # proxy_redirect off;
          proxy_set_header  X-Forwarded-Proto $scheme;
          proxy_set_header X-Forwarded-Protocol $scheme;

         # proxy_set_header  X-Forwarded-Proto "https";
         # proxy_set_header  X-Forwarded-Protocol "https";
        }
    }
}

　　

其中

/root/registry/nginx/nginx.htpasswd
生成方法 htpasswd -cb nginx.htpasswd tester1 123456

登陆验证

docker login xxxx:port

curl -i -k -v https://uname:password@xxx:port/v2/_ping

问题解决：

502 问题

nginx后台日志

connect() failed (111: Connection refused) while connecting to upstream

而且upstream总是80端口

修改proxy_pass，这里修改成了http://registry 是一个不存在的地址

client sent plain HTTP request to HTTPS port while reading client request headers

peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking,

或者registry log ： registry first record does not look like a TLS handshake

关闭所有registry的ssl选项，所有SSL控制交给nginx处理

registry报错 Invalid token

需要在启动时设置REGISTRY_HTTP_SECRET,所有replica都使用相同的值即可（-e REGISTRY_HTTP_SECRET=mysk）

registry报错 blob unknown

后端数据要存放在一个数据源上，比如NFS，ceph