cn2 lab 笔记

Ubuntu 18.04

Kafka

先启动kafka自带的zookeeper

在data/kafka_2.13-3.3.1bin目录下执行

./zookeeper-server-start.sh ../config/zookeeper.properties

再启动kafka

./kafka-server-start.sh ../config/server.properties

查看topic列表

./kafka-topics.sh --bootstrap-server localhost:9092 --list

查看该topic的具体内容(既可以看producer也可以看consumer)

cjy@cjy-CP65S:/data/kafka_2.13-3.3.1/bin$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic wordCount-input --from-beginning
this is my first job graph
this is my third GF, my second GF is a foreigner
Hi. Nice to meet you. I am glad that I have a great time with you
That's all

cjy@cjy-CP65S:/data/kafka_2.13-3.3.1/bin$ ./kafka-console-consumer.sh --bootp-server localhost:9092 --topic wordCount-output --from-beginning
(this,1)
(is,1)
(my,1)
(first,1)
(job,1)
(graph,1)
(this,2)
(is,2)
(my,2)
(third,1)
(gf,,1)
(my,3)
(second,1)
(gf,1)
(is,3)
(a,1)
(foreigner,1)
(hi.,1)
(nice,1)
(to,1)
(meet,1)
(you.,1)
(i,1)
(am,1)
(glad,1)
(that,1)
(i,2)
(have,1)
(a,2)
(great,1)
(time,1)
(with,1)
(you,1)

WordCount测试

用maven打包flinkAdapter，再执行

./bin/flink run -c WordCount.WordCount /data/MMCLab_T4_SS2023/mmc-lab-t4-ss2023/flinkAdapter/target/flinkAdapter-1.0-SNAPSHOT.jar --parallelism 2 --mode kafka --input /data/MMCLab_T4_SS2023/mmc-lab-t4-ss2023/Data/task2/wordCount.dat --output /data/MMCLab_T4_SS2023/mmc-lab-t4-ss2023/Data/task2/wordCountOut.txt --kafka-server localhost:9092 --num-record 20 --throughput 1

GoogleCloudMonitoring测试

./bin/flink run -c GoogleCloudMonitoring.GoogleCloudMonitoring /data/MMCLab_T4_SS2023/mmc-lab-t4-ss2023/flinkAdapter/target/flinkAdapter-1.0-SNAPSHOT.jar --parallelism 2 --mode kafka --input /data/MMCLab_T4_SS2023/mmc-lab-t4-ss2023/Data/task2/googleCloudMonitoring.csv --output /data/MMCLab_T4_SS2023/mmc-lab-t4-ss2023/Data/task2/googleCouldMonitorOutput.txt  --kafka-server localhost:9092 --query 1 --size 5 --slide 1 --lateness 10

Kafka-Producer测试

Gradle编译和打包

gradle build
gradle jar

运行kafka-producer

java -jar kafka-producer.jar --topic inputTopic --num-records 100 --producer-props bootstrap.servers=localhost:9092 --payload-file /data/MMCLab_T4_SS2023/mmc-lab-t4-ss2023/Data/task1/smart-grid.xlsx --throughput 2 --shuffle --shuffle-size 10

Windows 11

Maven（不添加环境变量）

maven打包

C:\apache-maven-3.8.8\bin\mvn package

Kafka

先启动kafka自带的zookeeper

在C:\kafka_2.13-3.3.1\bin\windows目录下执行

.\zookeeper-server-start.bat ../../config/zookeeper.properties

再启动kafka

.\kafka-server-start.bat ../../config/server.properties

Spring server

在maven打包后，在pom.xml同级目录下执行

java -jar <path to jar file>

Front end

在package.json同级目录下执行

npm start

Flink

启动cluster

PS C:\flink-1.16.0\bin> .\start-cluster.bat

关闭cluster

关掉自动运行的java cmd窗口即可

WordCount测试

用maven打包flinkAdapter，再执行

PS C:\flink-1.16.0> .\bin\flink.bat run -c WordCount.WordCount D:\VS_project\mmc-lab-t4-ss2023\flinkAdapter\target\flinkAdapter-1.0-SNAPSHOT.jar  --parallelism 2 --mode kafka --input D:\VS_project\mmc-lab-t4-ss2023\Data\task2\wordCount.dat   --output D:\VS_project\mmc-lab-t4-ss2023\Data\task2\wordCountOut --num-record 20 --throughput 1  --kafka-server localhost:9092