同步工具-腾讯EMR表治理工具安装使用

一、安装

1、root用户上传文件

cd wang
rz -bey luoshu-1.0-bin.tar.gz

2、解压文件到服务目录

重新安装洛书需执行：rm -rf /usr/local/service/luoshu

mkdir /usr/local/service/luoshu

tar -zxf luoshu-1.0-bin.tar.gz -C /usr/local/service/luoshu

3、（仅初次安装洛书执行）连接hive metastore,执行初始化脚本。脚本会新建数据库luoshu_db

mysql -h 127.0.0.1 -u账号 -p'密码' < config/luoshu-mysql-init.sql

4、（仅初次安装洛书执行）查看本地有无jdk-17.0.6版本，没有则上传。解压到/usr/local/jdk-17.0.6

rz -bey jdk-17.0.6.tar.gz

tar -zxf jdk-17.0.6.tar.gz -C  /usr/local/

5、修改luoshu/config下面的luoshu-env.sh

#!/usr/bin/env bash

# Options for luoshu server
# - JAVA_HOME, to set java home path
#export JAVA_HOME=
export JAVA_HOME=/usr/local/jdk-17.0.6

# - JAVA_EXTRA_OPTS, set java extra opts for luoshu server
#export JAVA_OPTS="-server -Xms512m -Xmx512m"
export JAVA_OPTS="-server -Xms512m -Xmx512m -Djava.security.krb5.conf=/etc/krb5.conf --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED"

# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
#export HADOOP_CONF_DIR=
export HADOOP_CONF_DIR=/usr/local/service/hadoop/etc/hadoop

# - HIVE_CONF_DIR, to point Spark towards Hive configuration files
#export HIVE_CONF_DIR=
export HIVE_CONF_DIR=/usr/local/service/hive/conf

# - LUOSHU_LOG_DIR, to set where log files are stored. (Default: ${LUOSHU_HOME}/logs)
#export LUOSHU_LOG_DIR=

# - SERVER_PORT, to set luoshu web server listen port. (Default: 8080)
#export SERVER_PORT=8080
export SERVER_PORT=9090

# - DB_DRIVER, to set datastore engine driver class name. (Default: com.mysql.cj.jdbc.Driver)
#export DB_DRIVER=com.mysql.cj.jdbc.Driver
export DB_DRIVER=com.mysql.cj.jdbc.Driver

# - DB_URL, to set datastore engine connection url
#export DB_URL="jdbc:mysql://localhost:3306/db?useSSL=false&characterEncoding=UTF-8"
export DB_URL="jdbc:mysql://localhost:3306/luoshu_db?useSSL=false&characterEncoding=UTF-8"

# - DB_USER, to set datastore engine connection username
#export DB_USER=
export DB_USER=账号

# - DB_PASS, to set datastore engine connection password
#export DB_PASS=
export DB_PASS=密码

6、配置luoshu-config.properties新增如下配置

#本地ip和spark做rpc通信，不能用127.0，用本机ip
thrift.server.host=10.10.10.10
thrift.server.port=9091

7、修改spark配置

/usr/local/service/spark/conf/spark-env.sh

新增：export JAVA_HOME=/usr/local/jdk

8、krb5.conf增加配置

原因：洛书使用了jdk17，不支持低版本加密方式，需要开启兼容

vim /etc/krb5.conf

[libdefaults]下面新增
    allow_weak_crypto = true

9、增加jar包

/usr/local/service/hadoop/share/hadoop/common/lib下面拷贝包

cp chdfs_hadoop_plugin_network-2.7.jar /usr/local/service/luoshu/lib/

cp gson-2.2.4.jar /usr/local/service/luoshu/lib/

cp goosefs-1.1.0-SNAPSHOT-client.jar /usr/local/service/luoshu/lib/

10、修改文件夹权限

chown hadoop:hadoop luoshu -R

11、多询问一下，有没有数据库变更

有的更新可能有数据库变更，询问有没有数据库字段变更，确保启动正常

12、启动luoshu（hadoop用户）

sudo su - hadoop

cd /usr/local/service/luoshu/bin

sh start.sh

13、验证服务

（1）查看启动日志

tail -f /usr/local/service/luoshu/logs/luoshu.log -n 500

（2）ps -ef |grep -i luoshu

查看是否是hadoop登录

（3）登录洛书页面，新增任务，查看是否验证正常

data_lake_ods.order_info1

14、附加-luoshu定时提交任务

默认是创建时间，每天一次全表优化

优化命令：

/usr/local/service/spark/bin/spark-submit --master yarn --deploy-mode cluster --queue default --class com.tencent.luoshu.optimizer.SparkBatchOptimizer --conf spark.yarn.submit.waitAppCompletion=false --conf spark.yarn.dist.files=/usr/local/service/luoshu/config/luoshu-config.properties --conf spark.app.name=spark-luoshu --conf spark.yarn.tags=spark-20240103164142 --conf spark.yarn.applicationType=spark-20240103164142 /usr/local/service/luoshu/spark/luoshu-spark-1.0-jar-with-dependencies.jar -1

15、附加-测试kerberos连接工具

/usr/local/jdk-17.0.6/bin/java -Djava.security.krb5.conf=/etc/krb5.conf --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -jar kerberos-test-1.0-jar-with-dependencies2.jar

二、使用

1、页面管理地址

127.0.0.1:9090/luoshu

2、配置定时清理任务