【原创】大数据基础之Kudu（2）移除dead tsever

当kudu有tserver下线或者迁移或者修改hostname之后，旧的tserver会一直以dead状态出现，并且tserver日志中会有大量的连接重试日志，一天的错误日志会有几个G，

W0322 22:13:59.202749 16927 tablet_service.cc:290] Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: e2f80a1fcf0c47f6b7f220a44d69297f. Requested UUID: 45bfb5b3e3ff41d9b1b1d2afab78d65c: from {username='kudu'} at 192.168.0.1:34724: tablet_id: "9933f18e59554ae6b5354e2a948469e9" caller_uuid: "9b164f37d04a484c8634ea86eae1b048" caller_term: 3 preceding_id { term: 2 index: 1873 } ops { id { term: 3 index: 1874 } timestamp: 6359719759241142272 op_type: NO_OP noop_request { } } dest_uuid: "45bfb5b3e3ff41d9b1b1d2afab78d65c" committed_index: 1874 all_replicated_index: 0 safe_timestamp: 6359719761707556864 last_idx_appended_to_leader: 1874

这时如果想要把这些dead状态的tserver去掉，并没有直接的命令，官方给出的方法如下：

Kudu does not currently have an automated way to remove a tablet server from a cluster permanently. Instead, use the following steps:

• 1 Ensure the cluster is in good health using ksck. See Checking Cluster Health with ksck.
  • 　　首先保证集群是健康的（通过ksck命令）
• 2 If the tablet server contains any replicas of tables with replication factor 1, these replicas must be manually moved off the tablet server prior to shutting it down. The kudu tablet change_config move_replica tool can be used for this.
  • 　　将dead状态的server上的副本进行迁移，如果有replication factor设置为1的数据，必须在下线前手工移动数据；
• 3 Shut down the tablet server. After -follower_unavailable_considered_failed_sec, which defaults to 5 minutes, Kudu will begin to re-replicate the tablet server’s replicas to other servers. Wait until the process is finished. Progress can be monitored using ksck.
  • 　　只要tserver处于下线状态超过5分钟以上会自动进行副本迁移；
• 4 Once all the copies are complete, ksck will continue to report the tablet server as unavailable. The cluster will otherwise operate fine without the tablet server. To completely remove it from the cluster so ksck shows the cluster as completely healthy, restart the masters. In the case of a single master, this will cause cluster downtime. With multimaster, restart the masters in sequence to avoid cluster downtime.
  • 　　当所有副本都迁移完之后，ksck依然会显示有tserver不可用，如果想完全去掉这些dead状态的server，需要重启master；

Do not shut down multiple tablet servers at once. To remove multiple tablet servers from the cluster, follow the above instructions for each tablet server, ensuring that the previous tablet server is removed from the cluster and ksck is healthy before shutting down the next.

最后，重启master之后在保证集群健康的前提下逐一重启tserver；

如果这样操作之后还是报错，说明可能有leader副本丢失，比如ksck报错

Tablet c58cef3f36a846b4bdf58447f77a6bcf of table 'impala::impala.test_kudu' is unavailable: 2 replica(s) not RUNNING
  a46f0fd38eba4a5286098ff7fe260eb1: TS unavailable
  45bfb5b3e3ff41d9b1b1d2afab78d65c: TS unavailable
  9b164f37d04a484c8634ea86eae1b048 (server02:7050): RUNNING [LEADER]
All reported replicas are:
  A = a46f0fd38eba4a5286098ff7fe260eb1
  B = 45bfb5b3e3ff41d9b1b1d2afab78d65c
  C = 9b164f37d04a484c8634ea86eae1b048
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | [config not available] |              |              |
 B             | [config not available] |              |              |
 C             | [config not available] |              |              |

可用的副本可能存在同步延迟会丢失部分数据，这时如果已经确定leader副本不可恢复，则可以强制指定剩下的可用副本为leader，恢复tablet到健康状态；

The remaining replica is not the leader, so the leader replica failed as well. This means the chance of data loss is higher since the remaining replica on tserver-00 may have been lagging.

$ sudo -u kudu kudu remote_replica unsafe_change_config tserver-00:7150 <tablet-id> <tserver-00-uuid>

where <tablet-id> is e822cab6c0584bc0858219d1539a17e6 and <tserver-00-uuid> is the uuid of tserver-00,638a20403e3e4ae3b55d4d07d920e6de.

<tablet-id>为非健康的tablet，tserver-00:7150为可用副本所在的tserver，<tserver-00-uuid>为可用副本所在的tserver的uuid，这样就可以在可能丢失少量数据的情况下恢复tablet；

如果有问题的tablet非常多，可以参考如下命令：

$ kudu cluster ksck localhost|grep -e '^Tablet '|awk '{print $2}'|xargs -i echo "sudo -u kudu kudu remote_replica unsafe_change_config tserver-00:7150 {} <tserver-00-uuid>"

参考：

https://kudu.apache.org/docs/administration.html#tablet_server_decommissioning

https://kudu.apache.org/docs/administration.html#tablet_majority_down_recovery