GitLab 502问题的解决

问题：

502 Whoops, GitLab is taking too much time to respond.

日志：

[root@cs12-66-gitlab ~]# my gitlab-ctl tail

-bash: my: command not found

[root@cs12-66-gitlab ~]# gitlab-ctl tail

==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==

# Logfile created on 2018-02-02 14:34:26 +0800 by logger.rb/56438

==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-02-02-14-34-39.log <==

   -> 0.3182s

== Seed from /opt/gitlab/embedded/service/gitlab-rails/db/fixtures/production/001_admin.rb

Administrator account created:

login:    root

password: You'll be prompted to create one on your first visit.

== Seed from /opt/gitlab/embedded/service/gitlab-rails/db/fixtures/production/010_settings.rb

==> /var/log/gitlab/gitlab-rails/production.log <==

Started GET "/" for 10.100.16.218 at 2018-02-02 14:54:06 +0800

Started GET "/-/metrics" for 127.0.0.1 at 2018-02-02 14:54:06 +0800

Processing by MetricsController#index as HTML

Completed 200 OK in 3ms (Views: 0.8ms | ActiveRecord: 0.0ms)

Processing by RootController#index as HTML

Redirected to http://git.tuandai888.com/users/sign_in

Filter chain halted as :redirect_unlogged_user rendered or redirected

Completed 302 Found in 9ms (ActiveRecord: 0.0ms)

Started GET "/users/sign_in" for 10.100.16.218 at 2018-02-02 14:54:10 +0800

Processing by SessionsController#new as HTML

==> /var/log/gitlab/gitlab-rails/grpc.log <==

# Logfile created on 2018-02-02 14:41:50 +0800 by logger.rb/56438

==> /var/log/gitlab/gitlab-rails/production_json.log <==

{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":3.63,"view":0.5,"db":0.0,"time":"2018-02-02T06:52:06.991Z","params":{},"remote_ip":null,"user_id":null,"username":null}

{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":4.42,"view":0.96,"db":0.0,"time":"2018-02-02T06:52:21.998Z","params":{},"remote_ip":null,"user_id":null,"username":null}

{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":17.59,"view":13.91,"db":0.0,"time":"2018-02-02T06:52:38.632Z","params":{},"remote_ip":null,"user_id":null,"username":null}

{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":4.71,"view":1.08,"db":0.0,"time":"2018-02-02T06:52:52.001Z","params":{},"remote_ip":null,"user_id":null,"username":null}

{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":3.6,"view":0.55,"db":0.0,"time":"2018-02-02T06:53:06.992Z","params":{},"remote_ip":null,"user_id":null,"username":null}

{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":4.08,"view":0.54,"db":0.0,"time":"2018-02-02T06:53:21.984Z","params":{},"remote_ip":null,"user_id":null,"username":null}

{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":3.64,"view":0.42,"db":0.0,"time":"2018-02-02T06:53:36.985Z","params":{},"remote_ip":null,"user_id":null,"username":null}

{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":4.94,"view":0.73,"db":0.0,"time":"2018-02-02T06:53:51.985Z","params":{},"remote_ip":null,"user_id":null,"username":null}

{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":4.72,"view":0.84,"db":0.0,"time":"2018-02-02T06:54:06.991Z","params":{},"remote_ip":null,"user_id":null,"username":null}

{"method":"GET","path":"/","format":"html","controller":"RootController","action":"index","status":302,"duration":10.83,"view":0.0,"db":0.0,"location":"http://git.tuandai888.com/users/sign_in","time":"2018-02-02T06:54:10.479Z","params":{},"remote_ip":"10.100.16.218","user_id":null,"username":null}

==> /var/log/gitlab/gitlab-rails/api_json.log <==

# Logfile created on 2018-02-02 14:41:51 +0800 by logger.rb/56438

==> /var/log/gitlab/gitlab-rails/application.log <==

# Logfile created on 2018-02-02 14:42:32 +0800 by logger.rb/56438

February 02, 2018 14:42: User "Administrator" (admin@example.com) was created

==> /var/log/gitlab/logrotate/current <==

==> /var/log/gitlab/unicorn/current <==

2018-02-02_06:42:37.56441 starting new unicorn master

2018-02-02_06:49:48.90547 adopted new unicorn master 3906

==> /var/log/gitlab/unicorn/unicorn_stderr.log <==

I, [2018-02-02T14:49:47.581121 #4594]  INFO -- : worker=0 ready

I, [2018-02-02T14:49:47.612613 #4597]  INFO -- : worker=1 ready

I, [2018-02-02T14:49:47.674536 #4600]  INFO -- : worker=2 ready

I, [2018-02-02T14:49:47.710865 #4603]  INFO -- : worker=3 ready

I, [2018-02-02T14:49:47.756704 #4606]  INFO -- : worker=4 ready

I, [2018-02-02T14:49:47.825869 #3906]  INFO -- : master process ready

I, [2018-02-02T14:49:47.836401 #4609]  INFO -- : worker=5 ready

I, [2018-02-02T14:49:47.887782 #4612]  INFO -- : worker=6 ready

I, [2018-02-02T14:49:47.909891 #4615]  INFO -- : worker=7 ready

I, [2018-02-02T14:49:47.940642 #4618]  INFO -- : worker=8 ready

==> /var/log/gitlab/unicorn/unicorn_stdout.log <==

==> /var/log/gitlab/sidekiq/current <==

2018-02-02_06:52:59.51350 127.0.0.1 - - [02/Feb/2018:14:52:59 CST] "GET /metrics HTTP/1.1" 200 25

2018-02-02_06:52:59.51358 - -> /metrics

2018-02-02_06:53:14.51410 127.0.0.1 - - [02/Feb/2018:14:53:14 CST] "GET /metrics HTTP/1.1" 200 25

2018-02-02_06:53:14.51417 - -> /metrics

2018-02-02_06:53:29.51384 127.0.0.1 - - [02/Feb/2018:14:53:29 CST] "GET /metrics HTTP/1.1" 200 25

2018-02-02_06:53:29.51390 - -> /metrics

2018-02-02_06:53:44.51387 127.0.0.1 - - [02/Feb/2018:14:53:44 CST] "GET /metrics HTTP/1.1" 200 25

2018-02-02_06:53:44.51392 - -> /metrics

2018-02-02_06:53:59.51637 127.0.0.1 - - [02/Feb/2018:14:53:59 CST] "GET /metrics HTTP/1.1" 200 25

2018-02-02_06:53:59.51640 - -> /metrics

==> /var/log/gitlab/gitlab-workhorse/current <==

2018-02-02_06:46:40.87969 10.100.12.66 @ - - [2018-02-02 14:46:40.877704109 +0800 CST] "GET / HTTP/1.1" 502 2916 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" 0.001910

2018-02-02_06:46:40.89513 2018/02/02 14:46:40 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/favicon.ico" ("") for GET "/favicon.ico"

2018-02-02_06:46:40.89566 10.100.12.66 @ - - [2018-02-02 14:46:40.89417661 +0800 CST] "GET /favicon.ico HTTP/1.1" 200 5430 "http://10.100.12.66/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" 0.001187

2018-02-02_06:47:25.68625 2018/02/02 14:47:25 error: GET "/": badgateway: failed after 0s: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory

2018-02-02_06:47:25.68697 git.tuandai888.com @ - - [2018-02-02 14:47:25.683237091 +0800 CST] "GET / HTTP/1.1" 502 2916 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" 0.003579

2018-02-02_06:47:25.87230 2018/02/02 14:47:25 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/favicon.ico" ("") for GET "/favicon.ico"

2018-02-02_06:47:25.87285 git.tuandai888.com @ - - [2018-02-02 14:47:25.871579483 +0800 CST] "GET /favicon.ico HTTP/1.1" 200 5430 "http://git.tuandai888.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" 0.001192

2018-02-02_06:47:45.63429 2018/02/02 14:47:45 error: GET "/": badgateway: failed after 0s: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory

2018-02-02_06:47:45.63501 git.tuandai888.com @ - - [2018-02-02 14:47:45.631742962 +0800 CST] "GET / HTTP/1.1" 502 2916 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" 0.003081

2018-02-02_06:54:10.49117 git.tuandai888.com @ - - [2018-02-02 14:54:06.631802992 +0800 CST] "GET / HTTP/1.1" 302 105 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" 3.859047

==> /var/log/gitlab/nginx/current <==

==> /var/log/gitlab/nginx/error.log <==

==> /var/log/gitlab/nginx/gitlab_access.log <==

10.100.16.218 - - [02/Feb/2018:14:46:40 +0800] "GET / HTTP/1.1" 502 2916 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

10.100.16.218 - - [02/Feb/2018:14:46:40 +0800] "GET /favicon.ico HTTP/1.1" 200 5430 "http://10.100.12.66/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

10.100.16.218 - - [02/Feb/2018:14:46:40 +0800] "GET / HTTP/1.1" 502 2916 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

10.100.16.218 - - [02/Feb/2018:14:46:40 +0800] "GET /favicon.ico HTTP/1.1" 200 5430 "http://10.100.12.66/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

10.100.16.218 - - [02/Feb/2018:14:46:40 +0800] "GET / HTTP/1.1" 502 2916 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

10.100.16.218 - - [02/Feb/2018:14:46:40 +0800] "GET /favicon.ico HTTP/1.1" 200 5430 "http://10.100.12.66/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

10.100.16.218 - - [02/Feb/2018:14:47:25 +0800] "GET / HTTP/1.1" 502 2916 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

10.100.16.218 - - [02/Feb/2018:14:47:25 +0800] "GET /favicon.ico HTTP/1.1" 200 5430 "http://git.tuandai888.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

10.100.16.218 - - [02/Feb/2018:14:47:45 +0800] "GET / HTTP/1.1" 502 2916 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

10.100.16.218 - - [02/Feb/2018:14:54:10 +0800] "GET / HTTP/1.1" 302 105 "" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

安装好GitLab，开启服务，发现有502错误。这是开始寻找解决办法，各种百度。
1.找到/var/log/gitlab/nginx中的错误日志文件，发现有如下错误/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (2: No such file or directory)，然后用 nc命令创建了这个socket文件，最终权限设为srwxrwxrwx，用户和组设置为git:git，但发现这个方法行不通。

2.这时我跑到GitLab的官网去寻找解决办法，https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
ctrl+f 502 找到官方教程中说502出现的问题

Note that on a single-core server it may take up to a minute to restart Unicorn and Sidekiq. Your GitLab instance will give a 502 error until Unicorn is up again.

It is also possible to start, stop or restart individual components.

sudo gitlab-ctl restart sidekiq
Unicorn supports zero-downtime reloads. These can be triggered as follows:

sudo gitlab-ctl hup unicorn
Note that you cannot use a Unicorn reload to update the Ruby runtime.

尝试用上面两个命令解决，发现没有用。
这时我怒了，不断的输入gitlab-ctl status,发现unicorn的pid一直在变大。而其他几个服务的pid没有变化。

3.这时差不多找到了问题的所在了，应该就是unicorn的问题。然后看官方教程，可以使用gitlab-ctl tail unicorn 来跟踪unicorn的状态，这时候悲催的发现原来时权限不够，chmod -R 777 /var/log/gitlab

502 Whoops, GitLab is taking too much time to respond。经搜索梳理网上几篇相同问题的解决方案，归纳原因可能如下：

1、unicorn原8080默认端口被容器中别的进程已经占用，必须调整为没用过的

2、gitlab的timeout设置过小，默认为60

从这两个原因着手，开始调整配置：

1、使用以下命令编辑gitlab配置文件：

sudo docker exec -it gitlab vim /etc/gitlab/gitlab.rb

经试验，应该修改以下配置项：

unicorn['port'] = 8888

gitlab_workhorse['auth_backend'] = "http://localhost:8888"

注意：unicorn['port']与gitlab_workhorse['auth_backend']的端口必须相同

2、调整timout时长，从60秒改为90秒

gitlab_rails['webhook_timeout'] = 90

gitlab_rails['git_timeout']=90

以上配置修改完成后，使用以下命令重启容器：

docker restart gitlab
docker logs gitlab

也可以通过以下命令进入容器，查看gitlab启动状态和动态日志：

sudo docker exec -it gitlab /bin/bash
cd /usr/bin
gitlab-ctl status
gitlab-ctl tail

3、初始镜像、容器的创建步骤是这样的：

sudo docker pull gitlab/gitlab-ce
chmod 777 /var/log/gitlab
sudo docker run --detach --hostname devops --publish 443:443 --publish 80:80 --publish 20022:22 --name gitlab --env GITLAB_OMNIBUS_CONFIG="external_url 'http://devops/'; gitlab_rails['lfs_enabled'] = true;" --restart always --volume /home/linyu/gitlab/config:/etc/gitlab --volume /home/linyu/gitlab/logs:/var/log/gitlab --volume /home/linyu/gitlab/data:/var/opt/gitlab gitlab/gitlab-ce:latest

4、也可以进入容器后，根据容器创建时映射的--volume目录，在里面修改配置，然后重启gitlab：

gitlab-ctl reconfigure
gitlab-ctl retart
gitlab-ctl status
gitlab-ctl tail

GitLab 502问题的解决的更多相关文章

Nginx的500,502,504错误解决方法
Nginx的500,502,504错误解决方法一.解决500错误: 1.500错误指的是服务器内部错误,也就是服务器遇到意外情况,而无法履行请求. 2.500错误一般有几种情况: (1)web脚本错 ...
【自建gitlab服务器】gitlab内存持续增大，出现502错误的解决办法
首先说明笔者的服务器环境,阿里云服务器:8G内存,2核.自从团队运维小伙伴搭建了gitlab之后,git push 代码时不时的就很卡,也经常出现 gitlab 反应超时,返回502错误,严重阻塞了团 ...
gitlab 502
经过一个下午的查找终于发现了错误,原来是在服务器上还开启了一个tomcat服务,占用了8080端口,使GitLab的unicorn服务不能开启. 最后在/etc/gitlab/gitlab.rb 中做 ...
gitlab 502解决方案
gitlab突然出现502 尝试gitlab-ctl restart.gitlab-ctl reconfigure 无效. 之后做法: #查看git日志 gitlab-ctl tail发现日志报错:端 ...
nginx 配置虚拟主机访问PHP文件 502错误的解决方法
最近配置Nginx 服务器虚拟主机访问目录发现报502错误百度了很多方法都不管用我擦各种抓狂----- 原本Nginx配置如下: 网上找了很多方法: 查看日志借助nginx的错误日志 ...
pycharm使用gitlab输错密码解决办法
在pycharm中使用http方式连接gitlab,在测试连接的时候提示输入用户名,密码.密码输错后,以后的每次test都是使用错误的密码,在删除pycharm后也是一样,解决方法是在控制面板\用户帐 ...
ASP.Net Core 发布到IIS Http Error 502.5 官方解决办法
Http Error 502.5 - Process Failure 在IIS上发布.NET Core程序出现这个错误.网上搜索到的办法为什么总行不通呢? 有可能年代久远,现在的环境与当年不同,所以解 ...
Mac下phpstorm 浏览器出现 502 bad gateway 解决办法
问题: 在网上搜了好久,都没有合适的解决办法,于是决定分享下自己鼓捣好久解决了的办法,希望能帮到大家. 出现上述情况,一般在phpstorm里都会出现这个提示我们只需要点击蓝色的进去,或者点phps ...
LNMP : 502 Bad Gateway 解决小记，真正的原因
站点搬迁到新的server.原先一直都是LAMP.如今改为LNMP. 将重写文件 htaccess改成 nginx的 conf.放到了站点.可仅仅能打开首页,其它重写页面一打开都是不停的载入. 载入等 ...

随机推荐

Eclipse SVN 合并分支/主干
可以从主干合并到分支,也可以从分支合并到主干,根据需要可以选择合适的选项,如下图:
dubbo管理控制台搭建
1. 从网上下载dubbo管理控制台:dubbo-admin-2.5.4.war 2. 下载tomcat7,解压缩 3. 将tomcat7 webapps/ROOT的文件全部删除,然后把dubbo-a ...
ShardedJedis的分片原理
ShardedJedisPool xml配置: <bean id="shardedJedisPool" class="redis.clients.jedis.Sha ...
dubbo的超时重试
dubbo的超时分为服务端超时 SERVER_TIMEOUT 和客户端超时 CLIENT_TIMEOUT.本文讨论服务端超时的情形: 超时:consumer发送调用请求后,等待服务端的响应,若超过ti ...
spring AOP自定义注解实现日志管理
今天继续实现AOP,到这里我个人认为是最灵活,可扩展的方式了,就拿日志管理来说,用Spring AOP 自定义注解形式实现日志管理.废话不多说,直接开始!!! 关于配置我还是的再说一遍. 在appli ...
069——VUE中vuex之使用getters高效获取购物车商品总价
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&quo ...
PHP:第一章——PHP中逻辑运算符的使用方法
//逻辑运算符 $a=;$b=;$c=;$d=; //逻辑与(and 和 &&)他们两个的逻辑是一样的,如果两个值都为true,结果才为true,否则是false. //var_dum ...
poj3461
题解: 简单kmp 然而strlen时间号费啊代码: #include<cstdio> #include<cstring> using namespace std; ; #d ...
ReentrantReadWriteLock——写写互斥（二）
"读写" ."写读"."写写"都是同步的.互斥的 1.Service.java package ReentrantReadWriteLock ...
Java通过class文件得到所在jar包
今天遇到一个问题,需要通过知道的class文件得到该文件所在的jar包,试过很多办法都不行,最后在网上找到了一个解决办法,如下: demo.java 1 2 String path = XXX.cla ...

GitLab 502问题的解决

GitLab 502问题的解决的更多相关文章

随机推荐

热门专题