MySQL SQL审核平台 inception+archer2.0（亲测）

docker run -d --privileged -v `pwd`/archer_data:/data -p 9306:3306 --name archer --hostname archer --net staticnet --ip 192.168.0.200 eiki/mysql:5.7.24 /usr/sbin/init

docker run -d --privileged -v `pwd`/archer_data:/data -p 9307:3306 -p 9123:9123 --name archer2 --hostname archer2 --net staticnet --ip 192.168.0.201 eiki/mysql:5.7.24 /usr/sbin/init

安装SQLAdvisor

1下载软件包

wget https://www.percona.com/downloads/Percona-Server-5.6/Percona-Server-5.6.29-76.2/binary/redhat/6/x86_64/Percona-Server-5.6.29-76.2-rddf26fe-el6-x86_64-bundle.tar

2 解压tar包

[root@archer5 soft]# tar -xvf Percona-Server-5.6.29-76.2-rddf26fe-el6-x86_64-bundle.tar
Percona-Server-56-debuginfo-5.6.29-rel76.2.el6.x86_64.rpm
Percona-Server-client-56-5.6.29-rel76.2.el6.x86_64.rpm
Percona-Server-devel-56-5.6.29-rel76.2.el6.x86_64.rpm
Percona-Server-server-56-5.6.29-rel76.2.el6.x86_64.rpm
Percona-Server-shared-56-5.6.29-rel76.2.el6.x86_64.rpm
Percona-Server-test-56-5.6.29-rel76.2.el6.x86_64.rpm
Percona-Server-tokudb-56-5.6.29-rel76.2.el6.x86_64.rpm

3 安装

[root@archer5 soft]# yum localinstall Percona-Server-shared-56-5.6.29-rel76.2.el6.x86_64.rpm

[root@archer5 soft]# yum localinstall Percona-Server-client-56-5.6.29-rel76.2.el6.x86_64.rpm

[root@archer5 soft]# yum localinstall Percona-Server-server-56-5.6.29-rel76.2.el6.x86_64.rpm

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h archer5 password 'new-password'

Alternatively you can run:

/usr/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.

See the manual for more instructions.

Please report any problems at
https://bugs.launchpad.net/percona-server/+filebug

The latest information about Percona Server is available on the web at
http://www.percona.com/software/percona-server

Support Percona by buying support at
http://www.percona.com/products/mysql-support

Percona Server is distributed with several useful UDF (User Defined Function) from Percona Toolkit.
Run the following commands to create these functions:
mysql -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'"
mysql -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"
mysql -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"
See http://www.percona.com/doc/percona-server/5.6/management/udf_percona_toolkit.html for more details

vi /etc/my.cnf

/etc/init.d/mysql start

[root@archer5 soft]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.29-76.2 Percona Server (GPL), Release 76.2, Revision ddf26fe

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

[root@localhost][(none)]> drop user root@::1;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '::1' at line 1
[root@localhost][(none)]> drop user root@'
Query OK, 0 rows affected (0.00 sec)

[root@localhost][(none)]> drop user root@'archer5
Query OK, 0 rows affected (0.00 sec)

[root@localhost][(none)]> drop user root@'127.0.0.1
Query OK, 0 rows affected (0.00 sec)

[root@localhost][(none)]> select user,host from mysql.user;
droarcher5
Query OK, 0 rows affected (0.00 sec)

[root@localhost][(none)]> drop user ''@localhost
Query OK, 0 rows affected (0.02 sec)

[root@localhost][(none)]>
[root@localhost][(none)]>
[root@localhost][(none)]> set password for root@localhost = password('rooT_258
Query OK, 0 rows affected (0.00 sec)

[root@localhost][(none)]>
[root@localhost][(none)]> quit
Bye
[root@archer5 soft]# mysql -uroot -p
Enter password:
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
[root@archer5 soft]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.6.29-76.2 Percona Server (GPL), Release 76.2, Revision ddf26fe

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

[root@localhost][(none)]>
[root@localhost][(none)]>
[root@localhost][(none)]> quit
Bye

[root@archer5 data]# git clone https://github.com/Meituan-Dianping/SQLAdvisor.git

[root@archer5 data]# yum install cmake libaio-devel libffi-devel glib2 glib2-devel bison gcc gcc-c++ cmake

[root@archer5 data]# yum install -y make cmake libaio-devel libffi-devel glib2 glib2-devel bison gcc gcc-c++

[root@archer5 data]# find / -name libperconaserverclient_r.so
[root@archer5 data]# find / -name libperconaserverclient_r.so.18
/usr/lib64/libperconaserverclient_r.so.18
[root@archer5 data]#
[root@archer5 data]#
[root@archer5 data]# cd /usr/lib64/
[root@archer5 lib64]# ln -s libperconaserverclient_r.so.18 libperconaserverclient_r.so

[root@archer5 data]# cd SQLAdvisor/

[root@archer5 SQLAdvisor]# cmake -DBUILD_CONFIG=mysql_release -DCMAKE_BUILD_TYPE=debug -DCMAKE_INSTALL_PREFIX=/usr/local/sqlparser -DIGNORE_AIO_CHECK=/usr/share/doc/libaio-0.3.109 ./

[root@archer5 SQLAdvisor]# cd ./sqladvisor/
[root@archer5 sqladvisor]# cmake -DCMAKE_BUILD_TYPE=debug ./

[root@archer5 sqladvisor]# make

[root@archer5 sqladvisor]# cp /data/SQLAdvisor/sqladvisor/sqladvisor /usr/bin/sqladvisor

[root@archer5 sqladvisor]# sqladvisor --help
Usage:
sqladvisor [OPTION…] sqladvisor

SQL Advisor Summary

Help Options:
-?, --help Show help options

Application Options:
-f, --defaults-file sqls file
-u, --username username
-p, --password password
-P, --port port
-h, --host host
-d, --dbname database name
-q, --sqls sqls
-v, --verbose 1:output logs 0:output nothing

安装inception

yum -y install cmake bison ncurses-devel gcc gcc-c++ openssl-devel

手动上传inception文件
tar -xf inception.tar -C /usr/local/

chown -R root:root /usr/local/inception/
vi ~/.bash_profile
增加：path=$path:/usr/local/inception/bin
source ~/.bash_profile

启动及检测
nohup /usr/local/inception/bin/Inception --defaults-file=/usr/local/inception/bin/inc.cnf >/dev/null 2>&1 &

nohup sh /usr/local/archer_web/archer/debug.sh >/dev/null 2>&1 &

验证：
mysql -uroot -h127.0.0.1 -P6669
inception get variables;

wget -O m4-1.4.9.tar.gz http://ftp.gnu.org/gnu/m4/m4-1.4.9.tar.gz
tar -zvxf m4-1.4.9.tar.gz
cd m4-1.4.9
./configure
make
make install

wget http://alpha.gnu.org/gnu/bison/bison-2.5.91.tar.xz
xz -d bison-2.5.91.tar.xz
tar xf bison-2.5.91.tar
cd bison-2.5.91
./configure
make && make install

安装完成后：bison -V

wget http://www.openssl.org/source/openssl-1.1.1.tar.gz
tar -zxvf openssl-1.1.1.tar.gz
cd openssl-1.1.1
./config --prefix=/usr/local/openssl shared zlib
make && make install

安装python3

yum install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev zlib zlib-devel

wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tgz
tar -xvf Python-3.6.6.tgz
cd Python-3.6.6
./configure --prefix=/usr/local/python3 --with-openssl=/usr/local/openssl --with-ssl
make && make install

软连接
cp /usr/bin/python /usr/bin/python2.7
ln -fs /usr/local/python3/bin/python3.6 /usr/bin/python
ln -fs /usr/local/python3/bin/pip3 /usr/bin/pip

--------------------------------------------------------------------------------------------------

wget https://www.python.org/ftp/python/3.4.1/Python-3.4.1.tgz
tar -xvf Python-3.4.1.tgz
cd Python-3.4.1
./configure --prefix=/usr/local/python3 --with-openssl=/usr/local/openssl --with-ssl
make && make install

软连接
ln -fs /usr/local/python3/bin/python3.4 /usr/bin/python
ln -fs /usr/local/python3/bin/pip3 /usr/bin/pip

---yum修改
vi /usr/bin/yum #将头部 #!/usr/bin/python 修改为 #!/usr/bin/python2.7
vi /usr/libexec/urlgrabber-ext-down #将头部 #!/usr/bin/python改为/usr/bin/python2.7。
vi /usr/bin/yum-config-manager #将头部 #!/usr/bin/python换成 #!/usr/bin/python2.7

pip install --upgrade pip

--安装archer
mkdir -p /usr/local/archer_web/
cd /usr/local/archer_web
yum -y install git
git clone https://github.com/jly8866/archer.git

--安装setuptools
cd /opt
wget --no-check-certificate https://pypi.python.org/packages/source/s/setuptools/setuptools-19.6.tar.gz#md5=c607dd118eae682c44ed146367a17e26
tar -zxvf setuptools-19.6.tar.gz
cd setuptools-19.6
python setup.py build
python setup.py install
可以看到安装到python3下面了

--安装Django
cd /opt
wget https://pypi.python.org/packages/44/41/bf93934082e9897a56a591a67bacbd9fb74e71244f3f42253432a9e627e6/Django-1.8.17.tar.gz#md5=e76842cdfbcb31286bd44f51e087a04c
tar -zxvf Django-1.8.17.tar.gz
cd Django-1.8.17
python setup.py install
python
>>> import django
>>> django.VERSION
(1, 8, 17, 'final', 0)
>>> exit()

yum -y install libyaml libyaml-devel

--安装Crypto和pymysql
pip install Crypto
pip install pycrypto
pip install pymysql
这里遇到了错误
# pip3 install Crypto
-bash: ./pip3: /usr/bin/python3: bad interpreter: No such file or directory
解决方法：
python3.4 -m pip install Crypto
或者
vim /opt/python3/bin/pip3
将#!/usr/bin/python3改为#!/opt/python3/bin/python3.4
# vim /opt/python3/lib/python3.4/site-packages/pymysql/connections.py
在if int(self.server_version.split(‘.’, 1)[0]) >= 5: 这一行之前加上以下这一句并保存，记得别用tab键用4个空格缩进：
self.server_version = '5.6.24-72.2-log'

--配置archer
cd /usr/local/archer_web/archer

--创建archer DB和inception DB
--进入安装的mysql5.6
mysql -uroot -p'rooT_258'
--grant all privileges on *.* to root@'%' identified by 'P@ssw0rd';
create database archer default character set utf8;
grant all on archer.* to archer_rw@'%' identified by 'P@ssw0rd';
create database inception_db default character set utf8;
grant all on inception_db.* to incep_rw@'%' identified by 'P@ssw0rd';
grant select,create,insert on *.* to incep_rw@'%' identified by 'P@ssw0rd';

10.200.22.33 3306
#create database test default character set utf8; #测试库建在审核库里了仅做测试
#grant all on test.* to test@'%' identified by 'test';
#flush privileges;

--修改/usr/local/archer_web/archer/archer/settings.py，如下内容：
vi /usr/local/archer_web/archer/archer/settings.py
#该项目本身的mysql数据库地址
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'archer',
'USER': 'archer_rw',
'PASSWORD': 'P@ssw0rd',
'HOST': '127.0.0.1',
'PORT': '3306'
}
}
#inception组件所在的地址
INCEPTION_HOST = '10.10.3.70'
INCEPTION_PORT = '6669'
#查看回滚SQL时候会用到，这里要告诉archer去哪个mysql里读取inception备份的回滚信息和SQL.
#注意这里要和inception组件的inception.conf里的inception_remote_XX部分保持一致.
INCEPTION_REMOTE_BACKUP_HOST='10.10.3.70'
INCEPTION_REMOTE_BACKUP_PORT=3306
INCEPTION_REMOTE_BACKUP_USER='incep_rw'
INCEPTION_REMOTE_BACKUP_PASSWORD='P@ssw0rd' 这个用户就是在部署inception使用的用户。（这用户应该是和上面创建的incep_rw是一个用户，可以不用创建incep_rw）

--通过model创建archer本身的数据库表
cd /usr/local/archer_web/archer
pip install django_admin_bootstrapped
pip install django_apscheduler
python manage.py makemigrations
python manage.py makemigrations sql
python manage.py migrate
进入数据库查看archer库下的表是否存在

--创建django admin管理员
python manage.py createsuperuser
Username: admin 自己填写用户名
Email address: 邮箱 xuaiqi@vcredit.com
Password: 密码 xaq1234!!
Password (again): 确认密码
Superuser created successfully.
该用户可以登录django admin来管理model。

--启动acher
vi debug.sh
#!/bin/bash
python manage.py runserver 10.10.3.70:9123 端口为9123 此处改为8080了
nohup sh /usr/local/archer_web/archer/debug.sh >/dev/null 2>&1 &

--创建archer系统登录用户
pip install simplejson
通过浏览器访问http://10.138.61.13:9123/admin/sql/users/
可以看到django登录界面：
使用上面创建的用户名密码（admin/和密码）登录：
点击右侧Add用户配置，用户名密码自定义，至少创建一个工程师和一个审核人（用admin用户可以登录）后续新的工程师和审核人用户请用LDAP导入sql_users表或django admin增加

--配置主库地址
通过浏览器访问http://10.138.61.13:9123/admin/sql/master_config
点击右侧Add master_config。这一步是为了告诉archer你要用inception去哪些mysql主库里执行SQL，所用到的用户名密码、端口等。

--正式访问主页
http://10.138.61.13:9123/login/
发起脚本如：
use mysql;
create table t (id int(10));
insert into t values(1);
insert into t values(2);
commit;
select * from t;
如报错：/opt/python3/lib/python3.4/site-packages/pymysql/connections.py in _request_authentication, line 1113
解决办法：
vi /opt/python3/lib/python3.4/site-packages/pymysql/connections.py
... ...
def _request_authentication(self):
# https://dev.mysql.com/doc/internals/en/connection-phase-packets.html#packet-Protocol::HandshakeResponse
self.server_version = '5.6.24-72.2-log' #添加此行
if int(self.server_version.split('.', 1)[0]) >= 5:
... ...

如果遇到rollback的问题，就是给权限的问题，inception配置的连接的数据库的用户的权限，在
grant select,create,insert on *.* to incep_rw@'%' identified by 'P@ssw0rd';
也就是我配置的admin的用户。

inception源码地址：https://github.com/mysql-inception/inception
archer 源码地址：https://github.com/jly8866/archer