修复gitlab服务器突然停电导致PostgreSQL损坏的数据库

最开始是存储的卷组受损，使用的DRBD，使用了xfs分区格式：

挂载也报错：

mount /dev/drbd0 /var/opt

mount: wrong fs type, bad option, bad superblock on /dev/drbd0,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)

/var/log/message 报：XFS (drbd0): Corruption warning: Metadata has LSN (152:651864) ahead of current LSN (129:294808). Please unmount and run xfs_repair (>= v4.3) to resolve.

修复卷组：xfx_repair -L /dev/drbd0

访问 gitlab web是报500：

gitlab-ctl tail是看到报错：

 /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/users/sign_in","format":"html","controller":"SessionsController","action":"new","status":500,"error":"ActiveRecord::StatementInvalid: PG::InternalError: ERROR:  missing chunk number 0 for toast value 127916 in pg_toast_2619\n: SELECT COUNT(count_column) FROM (SELECT  1 AS count_column FROM \"users\" LIMIT 2) subquery_for_count","duration":9.1,"view":0.0,"db":2.41,"time":"2019-07-26T03:16:02.627Z","params":{},"remote_ip":"10.100.17.191","user_id":null,"username":null}

这是postgresql查询users表示报错

访问GitLab的PostgreSQL数据库，看看出什么问题了

1.登陆gitlab的安装服务查看配置文件

cat /var/opt/gitlab/gitlab-rails/etc/database.yml 

production:
  adapter: postgresql
  encoding: unicode
  collation:
  database: gitlabhq_production  //数据库名
  pool: 10
  username: 'gitlab'  //用户名
  password:
  host: '/var/opt/gitlab/postgresql'  //主机
  port: 5432
  socket:
  sslmode:
  sslrootcert:
  sslca:

查看/etc/passwd文件里边gitlab对应的系统用户

[root@localhost ~]# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
gitlab-www:x:496:493::/var/opt/gitlab/nginx:/bin/false
git:x:495:492::/var/opt/gitlab:/bin/sh
gitlab-redis:x:494:491::/var/opt/gitlab/redis:/bin/false
gitlab-psql:x:493:490::/var/opt/gitlab/postgresql:/bin/sh  //gitlab的postgresql用户

2.根据上面的配置信息登陆postgresql数据库

[root@localhost ~]# su - gitlab-psql     //登陆用户
-sh-4.1$ psql -h /var/opt/gitlab/postgresql -d gitlabhq_production   连接到gitlabhq_production库

gitlabhq_production=# \dt;   // 查看表

gitlabhq_production=# select * from users; // 报如下错
ERROR:  missing chunk number 0 for toast value 127916 in pg_toast_2619

修复：

参考：https://www.postgresql.org/message-id/CAJfPOeDNcnrPLHm%3DuxO8qLL_g14-QG1O6vyLHvO20oWt0JpPgw%40mail.gmail.com

gitlabhq_production=# select 2619::regclass;
gitlabhq_production=# delete from pg_statistic;
reindex table pg_statistic;
vacuum analyze;

就这样有可以愉快的玩耍了