gitlab数据迁移至其他gitlb服务器上

需求：

A : 待迁移服务器，上边存有数据

B：接收项目得服务器，本身存在数据

验证方案：

一，搭建gitlab8.15.2

OS:rhel7.4

yum install policycoreutils-python.x86_64

wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-8.15.2-ce.0.el7.x86_64.rpm

chmod 777 gitlab-ce-8.15.2-ce.0.el7.x86_64.rpm

rpm -ivh gitlab-ce-8.15.2-ce.0.el7.x86_64.rpm

########################更新配置文件########修改数据库为mysql###################

1）安装MySQL

wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm

yum -y install mysql57-community-release-el7-10.noarch.rpm

yum -y install mysql-community-server

yum install mysql-devel

2）修改mysql密码

systemctl start mysqld.service

grep "password" /var/log/mysqld.log   ===》获得临时密码

mysql -uroot -p     # 回车后会提示输入密码

alter user 'root'@'localhost' identified by '123qweASD!';   ===》修改密码

set global validate_password_policy=0;  ===》关闭MySQL密码验证

set global validate_password_length=1;

 

注：MySQL需要开启binlog

vim /etc/my.cnf

添加

log-bin=/var/lib/mysql/mysql-bin   ====>路径注意权限，否则会启动报错

server-id=10

 

3）创建必要数据库

CREATE USER 'git'@'localhost' IDENTIFIED BY '123456';

SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda, innodb_large_prefix=1;

CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_general_ci`;

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES, REFERENCES ON `gitlabhq_production`.* TO 'git'@'localhost';

flush privileges;

4）修改配置文件

vim /etc/gitlab/gitlab.rb

 

修改内容如下

external_url 'http://10.2.177.31:8090'  ===》AIP

gitlab_rails['db_adapter'] = "mysql2"
gitlab_rails['db_encoding'] = "utf8"
gitlab_rails['db_database'] = "gitlabhq_production"
gitlab_rails['db_username'] = "git"
gitlab_rails['db_password'] = "123456"
gitlab_rails['db_host'] = "127.0.0.1"
gitlab_rails['db_port'] = 3306
postgresql['enable'] = false

 

vim /opt/gitlab/embedded/service/gitlab-rails/.bundle/config

修改内容如下：

BUNDLE_WITHOUT: "development:test:postgres"  ===》MySQL改为postgres

 

5）重构

cd /opt/gitlab/embedded/bin/

./gem install mysql2 --version="0.3.20"

./gem install -i /opt/gitlab/embedded/service/gem/ruby/2.3.0/ mysql2 --version="0.3.20"

//gitlab-rake gitlab:setup

gitlab-ctl reconfigure

gitlab-ctl restart

6)检查

gitlab-rake gitlab:check

#################同样搭建另一台#####################

保险起见，先备份gitlab

 gitlab-rake gitlab:backup:create  ====>备份目录/var/opt/gitlab/backups/

##################################################

二，备份库以及其他文件

key目录：

/var/opt/gitlab/.ssh/authorized_keys

====》cp authorized_keys /opt

仓库目录：

/var/opt/gitlab/git-data/repositories

  ===》tar cvf repositories_A.tar repositories   ===》cp repositories_A.tar /opt/

mysql库备份：

先创建一个备份数据库&复制到备份数据库gitlab

mysqldump gitlabhq_production -uroot -p123qweASD! | mysql gitlab -uroot -p123qweASD!

导出备份数据库：

mysqldump --single-transaction -u root -p -B xx > A.sql  ===》cp A.sql /opt

传送文件：将上述3个文件拷贝至接收服务器B：/opt目录下，其他目录也可以

解析binlog

cd mysql-binlog的目录

 mysqlbinlog --base64-output=decode-rows -v -d gitlabhq_production mysql-bin.000001 > t.binlog

 

 

三，迁移文件

gitlab命令参考：

https://wiki.archlinux.org/index.php/GitLab

 

1）合并MySQL数据库

查看t.binlog中的sql语句，按照顺序在mysql 上进行操作

 涉及的表为users,namespaces,routes,audit_events,members,notification_settings,project_features,projects,events,project_authorizations,

　　protected_branches,protected_branch_merge_access_levels,protected_branch_push_access_levels,events 共13个表

插入数据时注意各个表的id不要与已存数据冲突

 

2）合并repository数据库

cd /var/opt/gitlab/git-data/

tar xvf repositories_B.tar  ===>将B仓库合并至A中

之后执行gitlab-rake gitlab:import:repos

 

或者执行

#chmod -R ug+rwX,o-rwx 仓库path
#chmod -R ug-s 仓库path
#find 仓库path -type d -print0 |  xargs -0 chmod g+s
#chown -R git:git 仓库path

 

3）合并authorized_keys  ====>视情况====》我验证时不用

cat authorized_keys_B >> /var/opt/gitlab/.ssh/authorized_keys

 

############################### ########################MySQL数据库表合并##################################

1）创建组涉及4个表

namespaces,routes,members,notification_settings

验证：假设创建空组B，SQL执行顺序如下

　　1.1）namespaces 先为组B分配namespace_id=11，path=B

　　1.2）routes为组B指定path,  routes_id=15,  routes.source_id=namespaces_id ,routes.path=namespaces.path

　　1.3) members 为组B创建member_id=10,members.source_id=namespaces.id

　　1.4) notification_settings为组B创建id=10,notification_settings.source_id=namespace_id

测试，修改namespace_id以及其他关联id,验证能否生成新的组

要点：routes.source_id=namespaces.id=members.source_id=notification_settings.id

　　   routes.path=namespaces.path

####################测试实例############################

  INSERT INTO gitlabhq_production.namespaces values(11,'B','B',NULL,'2018-11-24 04:09:59','2018-11-24 04:09:59','Group','',NULL,0,20,0,NULL,1,NULL,NULL);
  INSERT INTO gitlabhq_production.routes values(15,11,'Namespace','B','2018-11-24 04:09:59','2018-11-24 04:09:59');
  INSERT INTO gitlabhq_production.members values(10,50,11,'Namespace',1,3,'GroupMember','2018-11-24 04:09:59','2018-11-24 04:09:59',NULL,NULL,NULL,NULL,NULL,NULL);
  INSERT INTO gitlabhq_production.notification_settings values(10,1,11,'Namespace',3,'2018-11-24 04:09:59','2018-11-24 04:09:59','{"new_note":false,"new_issue":false,"reopen_issue":false,"close_issue":false,"reassign_issue":false,"new_merge_request":false,"reopen_merge_request":false,"close_merge_request":false,"reassign_merge_request":false,"merge_merge_request":false,"failed_pipeline":false,"success_pipeline":false}');

###################################################

结果：测试通过

2)创建用户涉及3个表

users,namespaces,routes

验证：假设创建用户UserA,密码123qweASD!，SQL执行顺序如下

　　1.1) users为用户创建user_id, email，authentication_token具有唯一性

　　1.2)namespaces创建namespace_id ,namespaces.owner_id=users.id ，namespaces.name=users.username    namespace的name默认=path

　　1.3)routes创建routes_id, routes.source_id=namespaces.id,routes.path=namespaces.path

　　1.4) users更新密码   

　　密码会过期，要执行UPDATE gitlabhq_production.users set password_expires_at=NULL where id= 'x'

测试：修改相关关联id，是否会生成新组

　　要点：users_id=owner_id      routes.source_id=namespaces.id

###############################测试实例####

  INSERT INTO gitlabhq_production.users values(4,'zhangyu512@sohu.com','$2a$10$AibtkRnWuicxnioQ5enWReVnO8upwoVED94Mm3E/U6RQLstggLexm','c601f60659a30dcb49cfd978f5bd0f73b3591cdb562a100f6a0c4d92f65d0ccd','2018-11-24 05:35:52',NULL,0,NULL,NULL,NULL,NULL,'2018-11-24 05:35:52','2018-11-24 05:35:52','zhang',0,10,'','','','CCEPy4qr-kGUUNCzdzmi',2,NULL,0,NULL,'zhangyu',1,0,'active',1,NULL,1,NULL,NULL,NULL,'2018-11-24 05:35:52',NULL,NULL,0,'','zhangyu512@sohu.com',0,0,NULL,NULL,NULL,NULL,0,NULL,'',0,0,NULL,0,0,NULL,NULL,0,0,NULL,'c8ksp6mszdo9is7r9w4cq3t8f',NULL);
  INSERT INTO gitlabhq_production.namespaces values(15,'zhangyu','zhangyu',4,'2018-11-24 05:35:52','2018-11-24 05:35:52',NULL,'',NULL,0,20,0,NULL,NULL,NULL,NULL);
  INSERT INTO gitlabhq_production.routes values(19,15,'Namespace','zhangyu','2018-11-24 05:35:52','2018-11-24 05:35:52');
UPDATE gitlabhq_production.users set encrypted_password='$2a$10$6iZkb/N/mJ4lmJIjUlOSvuRZeAH26yyIY.7OB7RfrsykJ3mpiFDRW',reset_password_token=NULL,reset_password_sent_at=NULL,updated_at='2018-11-24 05:36:06',password_expires_at='2018-11-24 05:36:06' where id= '4'

UPDATE gitlabhq_production.users set encrypted_password='$2a$10$UGo9UflSX9LC9rGGtJqD/Om5b0xBhwPmgBh2Lvapz5T2XE34RK6oO',updated_at='2018-11-24 05:37:13' where id= '4'
UPDATE gitlabhq_production.users set password_expires_at=NULL where id= '4'

##########################################　

测试结果：通过

3）将用户添加至组

涉及2个表 members,notification_settings

　　3.1)members表为用户和组建立关联    members.source_id=namespaces.id    members.user_id=users.id

　　3.2)notification_settings为用户和组建立关联  notificatiion_settings.user_id=users.id   notification_setting.source_id=namespaces.id

 

 

##################测试实例##########

 INSERT INTO gitlabhq_production.members values(13,10,12,'Namespace',2,3,'GroupMember','2018-11-24 05:59:22','2018-11-24 05:59:22',1,NULL,NULL,NULL,NULL,NULL);
 INSERT INTO gitlabhq_production.notification_settings values(13,2,12,'Namespace',3,'2018-11-24 05:59:22','2018-11-24 05:59:22','{"new_note":false,"new_issue":false,"reopen_issue":false,"close_issue":false,"reassign_issue":false,"new_merge_request":false,"reopen_merge_request":false,"close_merge_request":false,"reassign_merge_request":false,"merge_merge_request":false,"failed_pipeline":false,"success_pipeline":false}');

###############################

测试结果：通过

 4）创建project（创建是直接指认组）

涉及project_features,project,routes,events,project_authorizations,users

　　4.1)project_features 创建2个id ,  指定snippets_access_level=0

　　4.2)projects创建project_id ,name,path

　　4.3)routes 创建 routes_id,routes.source_id=projects.id

　　4.4)project_features 设置第二个的project_features.project_id=projects.id

　　4.5)project_authorizations插入 user_id=1,project_id=projects.id

　　　　　　　　　　　　    插入user_id=组.用户的ID，project_id=projects.id

　　4.6)users  设置对应用户authorized_projects_populated=1

　　4.7)projects设置has_external_issue_tracker=0  has_external_wiki=0

######## 测试实例######################

 INSERT INTO gitlabhq_production.project_features values(9,NULL,20,20,20,20,20,'2018-11-24 06:21:40','2018-11-24 06:21:40',20);
 UPDATE gitlabhq_production.project_features set snippets_access_level=0 where id= '9'
 INSERT INTO gitlabhq_production.project_features values(10,NULL,20,20,20,20,20,'2018-11-24 06:22:44','2018-11-24 06:22:44',20);
 UPDATE gitlabhq_production.project_features set snippets_access_level=0 where id= '10'
 INSERT INTO gitlabhq_production.projects values(1,'A','A','','2018-11-24 06:22:44','2018-11-24 06:22:44',1,11,NULL,NULL,20,0,NULL,'none',0                  ,0,NULL,NULL,0,NULL,NULL,1,'BrGVBu9duy_xm5tfpKc6',NULL,1,3600,0,1,NULL,NULL,1,0,NULL,'default',0,NULL,NULL,'',0);
 INSERT INTO gitlabhq_production.routes values(21,1,'Project','B/A','2018-11-24 06:22:44','2018-11-24 06:22:44');
 UPDATE gitlabhq_production.projects set last_activity_at='2018-11-24 06:22:44' where id= '1'
 UPDATE gitlabhq_production.project_features set project_id=1 where id= '10'
 INSERT INTO gitlabhq_production.events values(9,NULL,NULL,NULL,NULL,1,'2018-11-24 06:22:45','2018-11-24 06:22:45',1,1);
 INSERT INTO gitlabhq_production.project_authorizations values(2,1,1,50);
 INSERT INTO gitlabhq_production.project_authorizations values(1,5,1,10);
UPDATE gitlabhq_production.users set authorized_projects_populated=1 where id= '5'
UPDATE gitlabhq_production.projects set has_external_issue_tracker=0 where id= '1'
 UPDATE gitlabhq_production.projects set has_external_wiki=0 where id= '1'

######################################

测试结果：通过