注:本文主要内容均来源 StarRocks 官网 https://docs.starrocks.com/zh-cn/main/administration/stargo

StarGo 是一个用于管理多个 StarRocks 集群的命令行工具。通过 StarGo,您可以使用简单的命令行实现多集群的部署、查看、升级、启动与停止等操作。

StarGo 使用 go 开发,github 地址 https://github.com/wangtianyi2004/starrocks-controller

部署 StarGo

在当前用户路径下下载并解压 StarGo 二进制安装包。

wget https://raw.githubusercontent.com/wangtianyi2004/starrocks-controller/main/stargo-pkg.tar.gz
tar -xzvf stargo-pkg.tar.gz

安装包包含以下文件。

stargo:StarGo 二进制文件,无需安装。

deploy-template.yaml:部署配置文件模板。

repo.yaml:指定 StarRocks 安装包下载库的配置文件。

StarGo 支持的版本参见: http://starrocks-thirdparty.oss-cn-zhangjiakou.aliyuncs.com/packageVersion.list

目前支持版本如下:

v2.0.1
v2.1.3
v2.1.6
v2.2.0
v2.2.1
v2.2.2
v2.2.3
v2.3.0

部署集群

前提条件

  • 待部署集群至少需要一个中控机节点和三个部署机节点,所有节点可以混合部署于同一台机器。
  • 中控机上需部署 StarGo。
  • 中控机与部署机间需创建 SSH 互信。

以下示例创建了中控机 sr-dev@r0 与部署机 starrocks@r1、starrocks@r2 以及 starrocks@r3 间的 SSH 互信。

创建 sr-dev@r0 到 starrocks@r1、r2、r3 的 ssh 互信。

[sr-dev@r0 ~]$ ssh-keygen
[sr-dev@r0 ~]$ ssh-copy-id starrocks@r1
[sr-dev@r0 ~]$ ssh-copy-id starrocks@r2
[sr-dev@r0 ~]$ ssh-copy-id starrocks@r3 ## 验证 sr-dev@r0 到 starrocks@r1、r2、r3 的 ssh 互信。
[sr-dev@r0 ~]$ ssh starrocks@r1 date
[sr-dev@r0 ~]$ ssh starrocks@r2 date
[sr-dev@r0 ~]$ ssh starrocks@r3 date

创建配置文件

根据以下 YAML 模板,创建部署 StarRocks 集群的拓扑文件。具体配置项参考参数配置。

[starrocks@bigdata11 stargo-pkg]$ cat deploy-star3.yaml
global:
user: "starrocks"
ssh_port: 22
fe_servers:
- host: xx.xx.xx.229
ssh_port: 22
http_port: 38030
rpc_port: 39020
query_port: 39030
edit_log_port: 39010
deploy_dir: /opt/StarRocks1/fe
meta_dir: /data/starrocks1/fe/meta
log_dir: /data/starrocks1/fe/log
priority_networks: xx.xx.xx.229
config:
sys_log_level: "INFO"
be_servers:
- host: xx.xx.xx.229
ssh_port: 22
be_port: 39060
webserver_port: 38040
heartbeat_service_port: 39050
brpc_port: 38060
deploy_dir : /opt/StarRocks1/be
storage_dir: /data/starrocks1/be/storage
log_dir: /data/starrocks1/be/log
priority_networks: xx.xx.xx.229/24
config:
create_tablet_worker_count: 3
  • 注:服务器上已经有一套 StarRocks,再部署一套单机的,修改了下程序、端口

创建部署目录(可选)

如果您在配置文件中设定的部署路径不存在,且您有创建该路径的权限,StarGo 将根据配置文件自动创建部署目录。如果路径已存在,请确保您有在该路径下拥有写入的权限。您也可以通过以下命令,在各部署节点分别创建部署路径。

  • 在 FE 节点安装目录下上创建 meta 路径。
mkdir -p /opt/StarRocks1/be
  • 在 BE 节点安装目录下上创建 storage 路径。
mkdir -p /data/starrocks1/be/storage
  • 注意: 请确保以上创建的路径与配置文件中的 meta_dir 和 storage_dir 相同。

部署 StarRocks

通过以下命令部署 StarRocks 集群。

./stargo cluster deploy <cluster_name> <version> <topology_file>
参数 描述
cluster_name 创建的集群名
version StarRocks 的版本
topology_file 配置文件名

创建成功后,集群将会自动启动。当返回 beStatus 和feStatus 为 true 时,集群部署启动成功。

示例:

[starrocks@bigdata11 stargo-pkg]$ ./stargo cluster deploy star3 v2.2.2 deploy-star3.yaml
[20220812-113235 OUTPUT] Deploy cluster [clusterName = star3, clusterVersion = v2.2.2, metaFile = deploy-star3.yaml] [20220812-113236 OUTPUT] PRE CHECK DEPLOY ENV:
PreCheck FE:
server id ssh auth meta dir deploy dir http port rpc port query port edit log port open files count
-------------------- --------------- ------------------------------ ------------------------------ --------------- --------------- --------------- --------------- ---------------
xx.xx.xx.229:39010 PASS PASS PASS PASS PASS PASS PASS PASS PreCheck BE:
server id ssh auth storage dir deploy dir webSer port heartbeat port brpc port be port open files count
-------------------- --------------- ------------------------------ ------------------------------ --------------- --------------- --------------- --------------- ---------------
xx.xx.xx.229:39060 PASS PASS PASS PASS PASS PASS PASS PASS [20220812-113236 OUTPUT] PreCheck successfully. RESPECT
[20220812-113236 OUTPUT] Create the deploy folder ...
[20220812-113237 OUTPUT] Download StarRocks package & jdk ...
[20220812-113322 INFO] The file starrocks-2.2.2-quickstart.tar.gz [1695364308] download successfully
[20220812-113322 OUTPUT] Download done.
[20220812-113322 OUTPUT] Decompress StarRocks pakcage & jdk ...
[20220812-113325 INFO] The tar file /home/starrocks/.stargo/download/starrocks-2.2.2-quickstart.tar.gz has been decompressed under /home/starrocks/.stargo/download
[20220812-113349 INFO] The tar file /home/starrocks/.stargo/download/StarRocks-2.2.2.tar.gz has been decompressed under /home/starrocks/.stargo/download
[20220812-113354 INFO] The tar file /home/starrocks/.stargo/download/jdk-8u301-linux-x64.tar.gz has been decompressed under /home/starrocks/.stargo/download
[20220812-113354 OUTPUT] Distribute FE Dir ...
[20220812-113402 INFO] Upload dir feSourceDir = [/home/starrocks/.stargo/download/StarRocks-2.2.2/fe] to feTargetDir = [/opt/StarRocks1/fe] on FeHost = [xx.xx.xx.229]
[20220812-113406 INFO] Upload dir JDKSourceDir = [/home/starrocks/.stargo/download/jdk1.8.0_301] to JDKTargetDir = [/opt/StarRocks1/fe/jdk] on FeHost = [xx.xx.xx.229]
[20220812-113406 INFO] Modify JAVA_HOME: host = [xx.xx.xx.229], filePath = [/opt/StarRocks1/fe/bin/start_fe.sh]
[20220812-113406 OUTPUT] Distribute BE Dir ...
[20220812-113417 INFO] Upload dir BeSourceDir = [/home/starrocks/.stargo/download/StarRocks-2.2.2/be] to BeTargetDir = [/opt/StarRocks1/be] on BeHost = [xx.xx.xx.229]
[20220812-113417 OUTPUT] Modify configuration for FE nodes & BE nodes ...
############################################# START FE CLUSTER #############################################
############################################# START FE CLUSTER #############################################
[20220812-113417 INFO] Starting leader FE node [host = xx.xx.xx.229, editLogPort = 39010]
[20220812-113438 WARN] The FE node doesn't start, wait for 10s [FeHost = xx.xx.xx.229, FeQueryPort = 39030, error = Process exited with status 1]
[20220812-113438 INFO] Starting leader FE node [host = xx.xx.xx.229, editLogPort = 39010]
[20220812-113454 INFO] The FE node start succefully [host = xx.xx.xx.229, queryPort = 39030]
[20220812-113454 INFO] List all FE status:
feHost = xx.xx.xx.229 feQueryPort = 39030 feStatus = true ############################################# START BE CLUSTER #############################################
############################################# START BE CLUSTER #############################################
[20220812-113454 INFO] Starting BE node [BeHost = xx.xx.xx.229 HeartbeatServicePort = 39050]
[20220812-113515 INFO] The BE node start succefully [host = xx.xx.xx.229, heartbeatServicePort = 39050]
[20220812-113515 OUTPUT] List all BE status:
beHost = xx.xx.xx.229 beHeartbeatServicePort = 39050 beStatus = true

如果执行检查失败,会有相应提示,按提示操作即可

[starrocks@bigdata12 stargo-pkg]$ ./stargo cluster deploy star3 v2.2.2 deploy-star3.yaml
[20220812-102653 OUTPUT] Deploy cluster [clusterName = star3, clusterVersion = v2.2.2, metaFile = deploy-star3.yaml] [20220812-102658 OUTPUT] PRE CHECK DEPLOY ENV:
PreCheck FE:
server id ssh auth meta dir deploy dir http port rpc port query port edit log port open files count
-------------------- --------------- ------------------------------ ------------------------------ --------------- --------------- --------------- --------------- ---------------
xx.xx.xx.228:39010 PASS FAILED: Priv failed FAILED: Priv failed PASS PASS PASS PASS PASS PreCheck BE:
server id ssh auth storage dir deploy dir webSer port heartbeat port brpc port be port open files count
-------------------- --------------- ------------------------------ ------------------------------ --------------- --------------- --------------- --------------- ---------------
xx.xx.xx.228:39060 PASS FAILED: Dir exist/Priv failed FAILED: Dir exist/Priv failed PASS PASS PASS PASS PASS
xx.xx.xx.229:39060 PASS FAILED: Dir exist/Priv failed FAILED: Dir exist/Priv failed PASS PASS PASS PASS PASS
xx.xx.xx.230:39060 PASS FAILED: Dir exist FAILED: Dir exist PASS PASS PASS PASS PASS [20220812-102658 ERROR] Please use bellowing promption to fix the issue for FE servers:
Detect the FE META FOLDER exist or no privilege. Use bellowing command to check or fix the issue:
[Host = xx.xx.xx.228] chown -R starrocks /data/starrocks1/fe/meta
Detect the FE DEPLOY FOLDER exist or no privilege. Use bellowing command to check or fix the issue:
[Host = xx.xx.xx.228] chown -R starrocks /opt/StarRocks1/fe [20220812-102658 ERROR] Please use bellowing promption to fix the issue for BE servers:
Detect the BE STORAGE FOLDER exist or no privilege. Use bellowing command to check or fix the issue:
[Host = xx.xx.xx.228] mkdir /data/starrocks1/be/storage.bak && mv /data/starrocks1/be/storage/* /data/starrocks1/be/storage.bak/ && chown -R starrocks /data/starrocks1/be/storage
[Host = xx.xx.xx.229] mkdir /data/starrocks1/be/storage.bak && mv /data/starrocks1/be/storage/* /data/starrocks1/be/storage.bak/ && chown -R starrocks /data/starrocks1/be/storage
[Host = xx.xx.xx.230] mkdir /data/starrocks1/be/storage.bak && mv /data/starrocks1/be/storage/* /data/starrocks1/be/storage.bak/
Detect the BE DEPLOY FOLDER exist or no privilege. Use bellowing command to check or fix the issue:
[Host = xx.xx.xx.228] mkdir /opt/StarRocks1/be.bak && mv /opt/StarRocks1/be/* /opt/StarRocks1/be.bak/ && chown -R starrocks /opt/StarRocks1/be
[Host = xx.xx.xx.229] mkdir /opt/StarRocks1/be.bak && mv /opt/StarRocks1/be/* /opt/StarRocks1/be.bak/ && chown -R starrocks /opt/StarRocks1/be
[Host = xx.xx.xx.230] mkdir /opt/StarRocks1/be.bak && mv /opt/StarRocks1/be/* /opt/StarRocks1/be.bak/ [20220812-102658 ERROR] PreCheck failed.

查看集群信息

[starrocks@bigdata11 stargo-pkg]$ ./stargo cluster list
[20220812-115246 OUTPUT] List all clusters
ClusterName Version User CreateDate MetaPath PrivateKey
--------------- ---------- ---------- ------------------------- ------------------------------------------------------------ --------------------------------------------------
star3 v2.2.2 starrocks 2022-08-12 11:34:17 /home/starrocks/.stargo/cluster/star3 /home/starrocks/.ssh/id_rsa

查看指定集群信息

[starrocks@bigdata11 stargo-pkg]$ ./stargo cluster display star3
[20220812-143023 OUTPUT] Display cluster [clusterName = star3]
clusterName = star3
clusterVerison = v2.2.2
ID ROLE HOST PORT STAT DATADIR DEPLOYDIR
-------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- --------------------------------------------------
xx.xx.xx.229:39010 FE xx.xx.xx.229 39010/39030 UP/L /opt/StarRocks1/fe /data/starrocks1/fe/meta
xx.xx.xx.229:39060 BE xx.xx.xx.229 39060/39050 UP /opt/StarRocks1/be /data/starrocks1/be/storage
[starrocks@bigdata11 stargo-pkg]$

升级版本

starrocks 2.2.0 升级到 2.3.0

[starrocks@bigdata11 stargo-pkg]$ ./stargo cluster upgrade star3 v2.3.0
[20220812-143227 OUTPUT] Upgrade cluster. [ClusterName = star3, TargetVersion = v2.3.0]
[20220812-143227 OUTPUT] Upgrade StarRocks Cluster star3, from version v2.2.2 to version v2.3.0
[20220812-143227 OUTPUT] Download StarRocks package & jdk ...
[20220812-143340 INFO] The file starrocks-2.3.0-quickstart.tar.gz [1726845129] download successfully
[20220812-143340 OUTPUT] Download done.
[20220812-143340 OUTPUT] Decompress StarRocks pakcage & jdk ...
[20220812-143346 INFO] The tar file /home/starrocks/.stargo/download/starrocks-2.3.0-quickstart.tar.gz has been decompressed under /home/starrocks/.stargo/download
[20220812-143411 INFO] The tar file /home/starrocks/.stargo/download/StarRocks-2.3.0.tar.gz has been decompressed under /home/starrocks/.stargo/download
[20220812-143411 INFO] The tar file /home/starrocks/.stargo/download/jdk-8u301-linux-x64.tar.gz has been decompressed under /home/starrocks/.stargo/download
[20220812-143413 OUTPUT] Starting upgrade BE node. [beId = 0]
[20220812-143413 INFO] upgrade be node - backup be lib. [host = xx.xx.xx.229, sourceDir = /opt/StarRocks1/be/lib, targetDir = /opt/StarRocks1/be/lib.bak-20220812143413]
[20220812-143423 INFO] upgrade be node - upload new be lib. [host = xx.xx.xx.229, sourceDir = /home/starrocks/.stargo/download/StarRocks-2.3.0/be/lib, targetDir = /opt/StarRocks1/be/lib]
[20220812-143423 INFO] Waiting for stoping BE node [BeHost = xx.xx.xx.229]
[20220812-143429 INFO] upgrade be node - stop be node. [host = xx.xx.xx.229, beDeployDir = /opt/StarRocks1/be]
[20220812-143430 INFO] upgrade be node - start be node. [host = xx.xx.xx.229, beDeployDir = /opt/StarRocks1/be]
[20220812-143443 INFO] upgrade be node - start be node. [host = xx.xx.xx.229, beDeployDir = /opt/StarRocks1/be]
[20220812-143443 OUTPUT] The Be node upgrade successfully. [beId = 0, currentVersion = v2.3.0-a9bdb09]
[20220812-143443 OUTPUT] Starting upgrade FE node. [feId = 0]
[20220812-143445 INFO] upgrade FE node - backup FE lib. [host = xx.xx.xx.229, sourceDir = /opt/StarRocks1/fe/lib, targetDir = /opt/StarRocks1/fe/lib.bak-20220812143443]
[20220812-143448 INFO] upgrade FE node - upload new FE lib. [host = xx.xx.xx.229, sourceDir = /home/starrocks/.stargo/download/StarRocks-2.3.0/fe/lib, targetDir = /opt/StarRocks1/fe/lib]
[20220812-143448 INFO] Waiting for stoping FE node [FeHost = xx.xx.xx.229]
[20220812-143451 INFO] upgrade FE node - stop FE node. [host = xx.xx.xx.229, feDeployDir = /opt/StarRocks1/fe]
[20220812-143452 INFO] upgrade FE node - start FE node. [host = xx.xx.xx.229, feDeployDir = /opt/StarRocks1/fe]
[20220812-143452 ERROR] Error in ping db [dbPath = root:@tcp(xx.xx.xx.229:39030)/], error = dial tcp xx.xx.xx.229:39030: connect: connection refused
[20220812-143502 INFO] upgrade FE node - start FE node. [host = xx.xx.xx.229, feDeployDir = /opt/StarRocks1/fe]
[20220812-143503 ERROR] The FE node upgrade failed. [feId = 0, targetVersion = v2.3.0, currentVersion = v]
  • 注: 是有报错,但是升级成功了

升级后查看集群


[20220812-143555 OUTPUT] List all clusters
ClusterName Version User CreateDate MetaPath PrivateKey
--------------- ---------- ---------- ------------------------- ------------------------------------------------------------ --------------------------------------------------
star3 v2.3.0 starrocks 2022-08-12 14:35:03 /home/starrocks/.stargo/cluster/star3 /home/starrocks/.ssh/id_rsa

已知缺陷

  1. StarGo 不会部署 Broker
  2. StarGo 安装包包含 jdk,不会替换安装

欢迎关注Flink菜鸟公众号,会不定期更新Flink(开发技术)相关的推文

StarRocks 运维工具 StarGo的更多相关文章

  1. CheungSSH国产自动化运维工具开源Web界面

    CheungSSH web2.0 发布文档 CheungSSH 简介 CheungSSH是一款国人自主研发的Linux运维自动化管理服务器软件,秉着为企业降低运营成本,解放管理员双手和自动化生产的理念 ...

  2. 自动化运维工具Ansible详细部署 (转载)

    自动化运维工具Ansible详细部署 标签:ansible 原创作品,允许转载,转载时请务必以超链接形式标明文章 原始出处 .作者信息和本声明.否则将追究法律责任.http://sofar.blog. ...

  3. 【运维工具】Git代码发布系统

    引言 代码发布系统是互联网公司必备的运维系统,作用主要用户发布业务代码 到 业务服务器 为什么需要代码发布系统 有的同学可能说,我们公司服务器就那么一台,做个发布系统太麻烦了? 不认同这说法 发布系统 ...

  4. Python自动化运维工具fabric的安装

    使用shell命令进行复杂的运维时,代码往往变得复杂难懂,而使用python脚本语言来编写运维程序,就相当于开发普通的应用一样,所以维护和扩展都比较简单,更重要的是python运维工具fabric能自 ...

  5. distri.lua的web运维工具

    我的新手游项目很快就要进入到寻找发行商的环节,最近几天相对较空闲,逐将工作重心转移到服务器组运维工具的制作上. 回想一年之前经历的那个不算成功的端游项目,因为运维工具设计得不合理,使用十分不方便,游戏 ...

  6. 自动化运维工具之 Ansible 介绍及安装使用

    一.初识Ansible 介绍: Absible 使用 模块(Modules)来定义配置任务.模块可以用标准脚本语言(Python,Bash,Ruby,等等)编写,这是一个很好的做法,使每个模块幂等.A ...

  7. 【运维工具】logrotate 日志管理神器

    服务器经常会产生各种各样的日志文件,我们需要定期清理 日志的分类 系统日志 应用日志 系统日志 例如系统的history 历史信息   crontab的运行日志  一般系统日志系统都帮我们运维好了,不 ...

  8. 自动化运维工具之ansible

    自动化运维工具之ansible   一,ansible简介 ansible是新出现的自动化运维工具,基于Python开发,集合了众多运维工具(puppet.cfengine.chef.func.fab ...

  9. 自动化运维工具Ansible详细部署 - 人生理想在于坚持不懈 - 51CTO技术博客

    自动化运维工具Ansible详细部署 - 人生理想在于坚持不懈 - 51CTO技术博客 自动化运维工具Ansible详细部署

随机推荐

  1. SpringBoot官方支持任务调度框架,轻量级用起来也挺香!

    大家好,我是二哥呀.定时任务的应用场景其实蛮常见的,比如说: 数据备份 订单未支付则自动取消 定时爬取数据 定时推送信息 定时发布文章 等等(想不出来了,只能等等来凑,,反正只要等的都需要定时,怎么样 ...

  2. STM32内存知识

    在了解STM32内存之前需要了解 MCU 的型号和MDK 中的.map 文件,很多刚学习 stm32 时都不会过多的去了解 MCU 的选型,是在太枯燥了.这里在从新了解一下,久了就熟悉了. 一.STM ...

  3. camunda开源版与商业版的差异

    Camunda流程引擎分社区版和企业版,社区版实际上是开源版,是Apache2.0协议,企业版实际上是商业收费版本,需要购买授权才能使用,那么社区版和企业版的差异有哪些呢,社区版本是否能满足我们日常的 ...

  4. flink处理延迟

    flink处理延迟 flink主要是处理实时数据的,在处理实时数据的过程中,难免会遇到乱序的存在.以事件时间举例,先发生的事件后到处理算子.flink针对乱序数据的处理主要有三种方式: 拨慢水位线的生 ...

  5. markdown常用到的语法

    一.标题 后加文字,几个#代表几级标题,最高为6 ,标准语法一般在#后跟个空格再写文字. 二.分割线 三个或者三个以上的 - 或者 * 三.图片 格式: ![A](B "C") A ...

  6. Linux-Centos快速安装Docker

    卸载之前的docker sudo yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-l ...

  7. 快速保存Win10锁屏壁纸,收获美丽瞬间

    对于写程序而言,每天接触得最多的就是电脑了 所以保持一种开放乐观,豁达美丽的心情是十分有必要的 使用"Everything"工具,输入"LocalState\Assets ...

  8. MAC帧的格式&&wireshark分析MAC帧

    MAC帧的格式 MAC帧较为简单,由五个字段组成 目的地址:6字节 源地址:6字节 类型字段:2字节,用来标志上一层使用的是什么协议,以便把收到的MAC地址帧的数据交给上一层的这个协议. 数据字段:其 ...

  9. 使用Runnable和Callable接口实现多线程的区别

    使用Runnable和Callable接口实现多线程的区别 先看两种实现方式的步骤: 1.实现Runnable接口 public class ThreadDemo{ public static voi ...

  10. 常用API(Java)

    常用API Object toString方法 场景:当我们使用toString方法想要输出对象变量时,官方提供的toString方法会直接输出对象所在的地址,而不是我们想要的对象变量,所以我们要把t ...