openstack私有云布署实践【5 数据库MariaDB 集群】

kxcontroller1   kxcontroller2  controller1

这3台作角色

确保服务器的域名解析访问正常

 

确保NTP时间同步

 

配置HAproxy主备访问，其中的一项数据库3306端口项，参考Haproxy配置文档。

 

每台安装如下包

----------------------------------------------------------------------

201511月的安装包如下

yum install -y  mariadb-galera-server xinetd rsync mariadb mariadb-galera-common galera MySQL-python 

-----------------------------------------------------------------------

20160620 数据库集群安装方式变了，缺省只需要2个包：mariadb mariadb-galera-common

 

新版的常规安装只需要用到2个包，以及方法如下

vi /etc/yum.repos.d/Galera.repo

 

[mariadb]

name = MariaDB Galera Cluster

baseurl = http://yum.mariadb.org/5.5-galera/centos7-amd64/

gpgkey = https://yum.mariadb.org/RPM-GPG-KEY-MariaDB

gpgcheck = 1

--------------------------------------------------------------------

 

yum install galera MariaDB-Galera-server  xinetd MySQL-python -y

 

但我们还需要用到xinetd MySQL-python ，第1个包是用来心跳检测，第2个是python访问mysql的插件，我们知道，心跳检测不是简单的telnet 3306端口就完事了，数据库集群异常的判断是你能能否登录数据库；

 

配置数据之间的心跳参数,三台服务器都配，配置的方式都是自己检测自己，所以下文会提及到一个普通登录检查的用户clustercheck

 

# vi /etc/sysconfig/clustercheck

MYSQL_USERNAME="clustercheck"

MYSQL_PASSWORD="venicchina"

MYSQL_HOST="localhost"

MYSQL_PORT="3306"

 

# vi  /etc/xinetd.d/galera-monitor

 

service galera-monitor

{

   port = 9200

   disable = no

   socket_type = stream

   protocol = tcp

   wait = no

   user = root

   group = root

   groups = yes

   server = /usr/bin/clustercheck

   type = UNLISTED

   per_source = UNLIMITED

   log_on_success =

   log_on_failure = HOST

   flags = REUSE

}

 

初始守护进程，以及启动xinetd的9200端口

 

# systemctl daemon-reload

# systemctl enable xinetd

# systemctl start xinetd

 

配置三台服务器数据配置文件

 

# vi /etc/my.cnf

 

[mysqld]

datadir=/home/local/mysql/data

socket=/var/lib/mysql/mysql.sock

# Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

# Settings user and group are ignored when systemd is used.

# If you need to run mysqld under a different user or group,

# customize your systemd unit file for mariadb according to the

# instructions in http://fedoraproject.org/wiki/Systemd

max_connections = 9999

 

[mysqld_safe]

log-error=/var/log/mariadb/mariadb.log

pid-file=/var/run/mariadb/mariadb.pid

 

#

# include all files from the config directory

#

!includedir /etc/my.cnf.d

 

 

新版本的mysql+centos7，要支持max_connections = 9999 参数还需要额外改多一个地方来支持外部访问并发数

vi /usr/lib/systemd/system/mariadb.service

在以下增加红色字体部份,意在修改普通用户的文件句柄数和可使用的进程数

[Service]

Type=simple

User=mysql

Group=mysql

LimitNOFILE=65535

LimitNPROC=65535

 

 

创建三台服务器的目录

# mkdir -p /home/local/mysql/data/

# mkdir -p /home/local/mysql/log/

# chown -R mysql:mysql  /home/local/mysql

 

配置三台服务器数据配置文件

# vi /etc/my.cnf.d/galera.cnf

 

[mysqld]

binlog_format=ROW

default-storage-engine=innodb

innodb_autoinc_lock_mode=2

innodb_file_per_table

collation-server = utf8_general_ci

init-connect = 'SET NAMES utf8'

character-set-server = utf8

#根据每台服务器进行修改

bind-address=10.40.42.1

 

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_cluster_name="my_wsrep_cluster"

wsrep_slave_threads=1

wsrep_certify_nonPK=1

wsrep_max_ws_rows=131072

wsrep_max_ws_size=1073741824

wsrep_debug=0

wsrep_convert_LOCK_to_trx=0

wsrep_retry_autocommit=1

wsrep_auto_increment_control=1

wsrep_drupal_282555_workaround=0

wsrep_causal_reads=0

wsrep_notify_cmd=

wsrep_sst_method=rsync

wsrep_cluster_address = "gcomm://kxcontroller1,kxcontroller2,controller1"

#根据每台服务器进行修改

wsrep_node_name = controller1

#根据每台服务器进行修改

wsrep_node_address=10.40.42.1

 

 

 

此时集群配置后，无需按自由版本提供的配置文件/etc/my.cnf.d/mariadb_openstack.cnf，因为它提到的配置，以上两个文件均有体现

 

启动第一台kxcontroller1数据库

(如第一步有报错，请检查/home/local/mysql目录的权限所有者是否为mysql,改之

/usr/libexec/mysqld --wsrep-new-cluster --user=root &

 

观察日志：

 

[root@node4 ~]# tail -f /var/log/mariadb/mariadb.log

150701 19:54:17 [Note] WSREP: wsrep_load(): loading provider library 'none'

150701 19:54:17 [Note] /usr/libexec/mysqld: ready for connections.

Version: '5.5.40-MariaDB-wsrep'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_25.11.r4026

 

出现ready for connections,证明我们启动成功

 

再初始第一台数据库

 

# mysql_secure_installation （初始化过程中，设置root密码venic8888  开启远程访问）

 

 

 

接下来另外两台（kxcontroller2   controller1）进行

 

systemctl start mariadb

 

可以查看/var/log/mariadb/mariadb.log，在日志可以看到节点均加入了集群中。

警告⚠：--wsrep-new-cluster这个参数只能在初始化集群使用，且只能在一个节点使用，且一般使用一次即可完成。

 

MariaDB [(none)]> show status like 'wsrep_%';

+------------------------------+---------------------------------------------------+

| Variable_name                | Value                                             |

+------------------------------+---------------------------------------------------+

| wsrep_local_state_uuid       | de6a8351-b5fa-11e5-a450-cebe9a9fd835              |

| wsrep_protocol_version       | 5                                                 |

| wsrep_last_committed         | 2                                                 |

| wsrep_replicated             | 0                                                 |

| wsrep_replicated_bytes       | 0                                                 |

| wsrep_repl_keys              | 0                                                 |

| wsrep_repl_keys_bytes        | 0                                                 |

| wsrep_repl_data_bytes        | 0                                                 |

| wsrep_repl_other_bytes       | 0                                                 |

| wsrep_received               | 6                                                 |

| wsrep_received_bytes         | 1146                                              |

| wsrep_local_commits          | 0                                                 |

| wsrep_local_cert_failures    | 0                                                 |

| wsrep_local_replays          | 0                                                 |

| wsrep_local_send_queue       | 0                                                 |

| wsrep_local_send_queue_avg   | 0.000000                                          |

| wsrep_local_recv_queue       | 0                                                 |

| wsrep_local_recv_queue_avg   | 0.000000                                          |

| wsrep_local_cached_downto    | 18446744073709551615                              |

| wsrep_flow_control_paused_ns | 0                                                 |

| wsrep_flow_control_paused    | 0.000000                                          |

| wsrep_flow_control_sent      | 0                                                 |

| wsrep_flow_control_recv      | 0                                                 |

| wsrep_cert_deps_distance     | 0.000000                                          |

| wsrep_apply_oooe             | 0.000000                                          |

| wsrep_apply_oool             | 0.000000                                          |

| wsrep_apply_window           | 0.000000                                          |

| wsrep_commit_oooe            | 0.000000                                          |

| wsrep_commit_oool            | 0.000000                                          |

| wsrep_commit_window          | 0.000000                                          |

| wsrep_local_state            | 4                                                 |

| wsrep_local_state_comment    | Synced                                            |

| wsrep_cert_index_size        | 0                                                 |

| wsrep_causal_reads           | 0                                                 |

| wsrep_cert_interval          | 0.000000                                          |

| wsrep_incoming_addresses     | 10.120.42.1:3306,10.120.42.2:3306,10.40.42.1:3306 |             #关注这个加入集群的主机IP，就是对应的这3台机器的IP

| wsrep_cluster_conf_id        | 16                                                |

| wsrep_cluster_size           | 3                                                 |

| wsrep_cluster_state_uuid     | de6a8351-b5fa-11e5-a450-cebe9a9fd835              |

| wsrep_cluster_status         | Primary                                           |

| wsrep_connected              | ON                                                |

| wsrep_local_bf_aborts        | 0                                                 |

| wsrep_local_index            | 2                                                 |

| wsrep_provider_name          | Galera                                            |

| wsrep_provider_vendor        | Codership Oy <info@codership.com>                 |

| wsrep_provider_version       | 3.5(rXXXX)                                        |

| wsrep_ready                  | ON                                                |

| wsrep_thread_count           | 2                                                 |

+------------------------------+---------------------------------------------------+

 

测试，数据库stop其中1台观察集群show status like 'wsrep_%';状态。

 

接下可测试数据库创建等操作。

 

三台服务器加入开机自启。

 

systemctl enable mariadb

 

使用HA proxy网页服务器，http://10.40.42.10:8008/admin-venic 观察代理是否正常

 

还差一步，进入其中一台数据库。执行。mysql -u root -p

创建一个专用于心跳的用户，参考以上 vi /etc/sysconfig/clustercheck配置的用户名和密码

 

CREATE USER 'clustercheck'@'localhost' IDENTIFIED BY 'venicchina';

flush privileges;

 

其它两个数据库查看是否创建成功这个用户

 

SELECT DISTINCT CONCAT('User: ''',user,'''@''',host,''';') AS query FROM mysql.user;

 

再使用HA proxy网页服务器，http://10.40.42.10:8008/admin-venic观察状态是否变绿。

 

 

--------------------------------------------------------------------------------------------

重点

服务器重启时，不建议全部同时重启数据库。不然，

systemctl start mariadb会命令全部失效

 

如发生此情况。应按以上的方法先启用第一台kxcontroller

#/usr/libexec/mysqld --wsrep-new-cluster --user=root &

 

再其它台

systemctl start mariadb

 

状态同步后，你可以保持这个状态，如果你跟我一样有强迫症的话，可选择在第1台kxcontroller1，kill -9 杀死mysql进程，第1台就可以使用命令，命令就相同了

systemctl start mariadb

-------------------------------------------------------------------------------------------

其它额外配置。赋予远程登录的权限和密码

grant all privileges on *.* to root@'%' identified by '高强度密码';

flush privileges;

 

 

这是是查询数据库当前设置的最大连接数（每台数据的并发数是独立的，所以上面max_connections = 9999  在一开始都配好，因为openstack数据库并发非常高）

mysql> show variables like '%max_connections%';

+-----------------+-------+

| Variable_name   | Value |

+-----------------+-------+

| max_connections | 1000  |----默认1000

+-----------------+-------+