分布式数据库TiDB的部署

转自：https://my.oschina.net/Kenyon/blog/908370

一、环境

CentOS Linux release 7.3.1611 (Core)
172.26.11.91  pd & tidb
172.26.11.92  tikv
172.26.11.93  tikv
172.26.11.94  tikv

二、安装

分别在4台服务器上上传安装包

wget http://download.pingcap.org/tidb-latest-linux-amd64.tar.gz
tar -xzf tidb-latest-linux-amd64.tar.gz
cd tidb-latest-linux-amd64
mkdir -p /data/tidb/log
ln -s /root/software/tidb/tidb-latest-linux-amd64/bin/pd-tso-bench /usr/bin
ln -s /root/software/tidb/tidb-latest-linux-amd64/bin/tikv-server  /usr/bin/
ln -s /root/software/tidb/tidb-latest-linux-amd64/bin/tidb-server  /usr/bin/
ln -s /root/software/tidb/tidb-latest-linux-amd64/bin/pd-server    /usr/bin/
ln -s /root/software/tidb/tidb-latest-linux-amd64/bin/pd-ctl       /usr/bin/

三、配置使用

1.按照顺序启动

在91上启动pd服务
pd-server    --name=pd1                                     --多个pd以不同名字命名
             --data-dir=/data/tidb/pd                       --pd路径
             --client-urls="http://172.26.11.91:2379"
             --peer-urls="http://172.26.11.91:2380"
             --initial-cluster="pd1=http://172.26.11.91:2380" --多个pd以逗号分隔
             --log-file=/data/tidb/log/pd.log  &

在92,93,94上启动tikv
tikv-server   --pd="172.26.11.91:2379" \
              --addr="172.26.11.92:20160" \
              --data-dir=/data/tidb/tikv \
              --log-file=/data/tidb/log/tikv.log

tikv-server   --pd="172.26.11.91:2379" \
              --addr="172.26.11.93:20160" \
              --data-dir=/data/tidb/tikv \
              --log-file=/data/tidb/log/tikv.log

tikv-server   --pd="172.26.11.91:2379" \
              --addr="172.26.11.94:20160" \
              --data-dir=/data/tidb/tikv \
              --log-file=/data/tidb/log/tikv.log &

在91上启动tipd服务
tidb-server    --store=tikv \                  --tikv引擎允许分布式存储，其他如LevelDB等是本地存储
               --path="172.26.11.91:2379" \
               --log-file=/data/tidb/log/tidb.log  &

2.登陆使用

[root@test05 ~]# mysql -h 172.26.11.91 -P 4000 -u root -D test
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 5.7.1-TiDB-1.0 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database db_kenyon;
Query OK, 0 rows affected (2.02 sec)

mysql> use db_kenyon;
Database changed
mysql> create table tbl_kenyon(user_code varchar(64) primary key,user_name varchar(32),ctime timestamp);
Query OK, 0 rows affected (2.03 sec)

mysql> insert into tbl_kenyon values('01','qiaofeng',now()),('02','murong',now());
Query OK, 2 rows affected (0.01 sec)

mysql>

三、高可用

tidb的数据都是保存在tikv节点上面，比如上面配置了3套tikv，每套tikv都是独立的，数据保存的方式和传统关系型不一样的是，在tidb里面或者说tikv里面是映射成kv模式存储的

把92的tikv人为挂掉，此时数据库的使用会受影响，简单的一个查询就会被挂起，直到切换成功

--切换过程
mysql> select * from tbl_kenyon;
+----+-----------+---------------------+
| id | cname     | ctime               |
+----+-----------+---------------------+
|  1 | qiaofeng  | 2017-05-23 10:50:43 |
|  2 | murong    | 2017-05-23 10:50:43 |
|  3 | saodiseng | 2017-05-23 10:50:43 |
+----+-----------+---------------------+
3 rows in set (10.24 sec)

--切换以后
mysql> select * from tbl_kenyon;
+----+-----------+---------------------+
| id | cname     | ctime               |
+----+-----------+---------------------+
|  1 | qiaofeng  | 2017-05-23 10:50:43 |
|  2 | murong    | 2017-05-23 10:50:43 |
|  3 | saodiseng | 2017-05-23 10:50:43 |
+----+-----------+---------------------+
3 rows in set (0.00 sec)

可以发现经过投票，PD已经连到94上去了（93,92上都能看到），此时读取表数据很快

2017/05/23 17:59:12.151 server.rs:153: [INFO] TiKV is ready to serve
2017/05/23 17:59:12.517 raft.rs:846: [INFO] [region 2] 3 [term: 1487] received a MsgHeartbeat message with higher term from 7 [term: 1488]
2017/05/23 17:59:12.517 raft.rs:681: [INFO] [region 2] 3 became follower at term 1488
2017/05/23 17:59:12.525 server.rs:460: [INFO] resolve store 6 address ok, addr 172.26.11.94:20160
2017/05/23 17:59:13.517 apply.rs:621: [INFO] [region 2] 3 execute admin command cmd_type: CompactLog compact_log {compact_index: 6437 compact_term: 1488} at [term: 1488, index: 6439]
2017/05/23 17:59:13.644 raftlog_gc.rs:117: [INFO] [region 2] collected 225 log entries
2017/05/23 17:59:43.517 apply.rs:621: [INFO] [region 2] 3 execute admin command cmd_type: CompactLog compact_log {compact_index: 6498 compact_term: 1488} at [term: 1488, index: 6500]
2017/05/23 17:59:43.643 raftlog_gc.rs:117: [INFO] [region 2] collected 61 log entries

这是因为92是Region中的leader，假如不是leader的tikv服务器受影响如93,94，数据因为默认做了三个副本（也可以配置5个或者7个副本），服务并不会受影响，但是在日志中会不停地告警

四、水平扩展

其实主要是以上组件模块的扩展，对于tidb来说，本身是无状态的，比较容易扩展，pd也可以部署成集群的模式，通过Haproxy、F5或者其他第三方软件来实现，比较难的Tikv的水平扩展。在每个TiKV的节点里，逻辑上划分了一个或多个store，每个store里又划了一个或多个Region，数据就是存放在Region里面，每个Region的默认值是64M，扩容的过程类似以下细胞分裂的过程，比传统的RDBMS采用的Sharding方式以及中间件模式要透明很多，也许以后市面上的诸多中间件日子要难过了。

1.添加pd

动态添加pd
pd-server --name=pd2 \
          --client-urls="http://172.26.11.95:2379" \
          --peer-urls="http://172.26.11.95:2380" \
          --join="http://172.26.11.91:2379"              --要加入的原pd集群
动态删除pd
pd-ctl -u http://172.26.11.91:2379
>> member delete pd2

2.添加tikv

添加tikv，比较简单，直接注册一个新的tikv，剩下的数据迁移工作就交给pd，以下在91上新注册一个tikv
tikv-server --pd="172.26.11.91:2379" \
            --addr="172.26.11.91:20160" \
            --data-dir=/data/tidb/tikv \
            --log-file=/data/tidb/log/tikv.log &

查看store，新增了一个1001的store，另外也能看出当前的leader在93上面
[root@test05 ~]# pd-ctl -u http://172.26.11.91:2379
» store
{
  "count": 4,
  "stores": [
    {
      "store": {
        "id": 6,
        "address": "172.26.11.94:20160",
        "state": 0,
        "state_name": "Up"
      },
      "status": {
        "store_id": 6,
        "capacity": "21 GB",
        "available": "21 GB",
        "leader_count": 0,
        "region_count": 1,
        "sending_snap_count": 0,
        "receiving_snap_count": 0,
        "applying_snap_count": 0,
        "is_busy": false,
        "start_ts": "2017-05-23T18:05:07+08:00",
        "last_heartbeat_ts": "2017-05-24T17:52:03.842239159+08:00",
        "uptime": "23h46m56.842239159s"
      }
    },
    {
      "store": {
        "id": 1001,
        "address": "172.26.11.91:20160",
        "state": 0,
        "state_name": "Up"
      },
      "status": {
        "store_id": 1001,
        "capacity": "21 GB",
        "available": "15 GB",
        "leader_count": 0,
        "region_count": 0,
        "sending_snap_count": 0,
        "receiving_snap_count": 0,
        "applying_snap_count": 0,
        "is_busy": false,
        "start_ts": "2017-05-24T17:50:12+08:00",
        "last_heartbeat_ts": "2017-05-24T17:52:03.290658649+08:00",
        "uptime": "1m51.290658649s"
      }
    },
    {
      "store": {
        "id": 1,
        "address": "172.26.11.92:20160",
        "state": 0,
        "state_name": "Up"
      },
      "status": {
        "store_id": 1,
        "capacity": "21 GB",
        "available": "21 GB",
        "leader_count": 0,
        "region_count": 1,
        "sending_snap_count": 0,
        "receiving_snap_count": 0,
        "applying_snap_count": 0,
        "is_busy": false,
        "start_ts": "2017-05-23T17:59:12+08:00",
        "last_heartbeat_ts": "2017-05-24T17:52:06.843194072+08:00",
        "uptime": "23h52m54.843194072s"
      }
    },
    {
      "store": {
        "id": 4,
        "address": "172.26.11.93:20160",
        "state": 0,
        "state_name": "Up"
      },
      "status": {
        "store_id": 4,
        "capacity": "21 GB",
        "available": "21 GB",
        "leader_count": 1,
        "region_count": 1,
        "sending_snap_count": 0,
        "receiving_snap_count": 0,
        "applying_snap_count": 0,
        "is_busy": false,
        "start_ts": "2017-05-23T17:22:48+08:00",
        "last_heartbeat_ts": "2017-05-24T17:52:09.766282426+08:00",
        "uptime": "24h29m21.766282426s"
      }
    }
  ]
}

3.删除tikv

查看1001的store状态是0，也就是up
» store 1001
{
  "store": {
    "id": 1001,
    "address": "172.26.11.91:20160",
    "state": 0,
    "state_name": "Up"
  },
  "status": {
    "store_id": 1001,
    "capacity": "21 GB",
    "available": "15 GB",
    "leader_count": 0,
    "region_count": 0,
    "sending_snap_count": 0,
    "receiving_snap_count": 0,
    "applying_snap_count": 0,
    "is_busy": false,
    "start_ts": "2017-05-24T17:50:12+08:00",
    "last_heartbeat_ts": "2017-05-24T17:58:57.490156968+08:00",
    "uptime": "8m45.490156968s"
  }
}

--删除过程，state=1表示正在下线
» store delete 1001
Success!
» store 1001
{
  "store": {
    "id": 1001,
    "address": "172.26.11.91:20160",
    "state": 1,
    "state_name": "Offline"
  },
  "status": {
    "store_id": 1001,
    "capacity": "21 GB",
    "available": "15 GB",
    "leader_count": 0,
    "region_count": 0,
    "sending_snap_count": 0,
    "receiving_snap_count": 0,
    "applying_snap_count": 0,
    "is_busy": false,
    "start_ts": "2017-05-24T17:50:12+08:00",
    "last_heartbeat_ts": "2017-05-24T17:59:17.690136502+08:00",
    "uptime": "9m5.690136502s"
  }
}

--state=2表示数据已经清理，可以关闭
» store 1001
{
  "store": {
    "id": 1001,
    "address": "172.26.11.91:20160",
    "state": 2,
    "state_name": "Tombstone"
  },
  "status": {
    "store_id": 0,
    "capacity": "0 B",
    "available": "0 B",
    "leader_count": 0,
    "region_count": 0,
    "sending_snap_count": 0,
    "receiving_snap_count": 0,
    "applying_snap_count": 0,
    "is_busy": false,
    "start_ts": "1970-01-01T08:00:00+08:00",
    "last_heartbeat_ts": "0001-01-01T00:00:00Z",
    "uptime": "0s"
  }
}

»  

五、总结：

1.维护相对很简单，官方文档是有中英文版本，资料更新相对及时，但是实际使用者提供的资料较少
2.Scale Out相比较传统的RDBMS方案上来看简化很多，特别是可以摒弃五花八门的中间件
3.从架构上来看，小数据量的性能应该一般，不建议使用，大数据量理论上会较好
4.期待GA版本