基于Java的支持可变QPS的http负载生成器，提供交互界面和RMI接口
Load generator
The load generator is a Java maven project which is implemented using httpclient+threadpool that works in open-loop [1], it has a web GUI for the realtime latency watch (e.g., 99th tail-latency, QPS, RPS, etc.). Meanwhile, the load generator provides RMI interface for external call and supports dynamically changeable request loads
- support thousands level concurrent request per second in single node
- support web GUI for real-time latency watch
- support dynamiclly changeable QPS in open-loop
- support RMI interface for external call
Code architecture
/LoadGen
|----/build/sdcloud.war # the executable war package
|----/src/main
|----/java/scs/
| |----/controller/* # MVC controller layer
| |----/pojo/* # entity bean layer
| +----/util
| |----/format/* # format time and data
| |----/loadGen
| | |----/loadDriver/* # generate request loads
| | |----/recordDriver/* # record request metrics
| | +----/strategy/*
| |----/respository/* # in-memory data storage
| |----/rmi/* # RMI service and interfaces
| +----/tools/* # some tools
|----/resources/* # configuration files
+----/webapp/* # GUI pages
Hardware environment
In our experiment environment, the configuration of nodes are shown as below:
| hostname | description | IP | role |
|---|---|---|---|
| tank-node1 | where the load generator is deployed | 192.168.3.110 | k8s master |
| tank-node3 | where the inference service is deployed | 192.168.3.130 | k8s slave |
Build load generator
The load generator is writen in Java, it can be deployed in container or host machine, and we need install Java JDK and apache tomcat before using it
Step 1: Install Java JDK
Download java jdk and install to /usr/local/java/
$ wget https://download.oracle.com/otn/java/jdk/8u231-b11/5b13a193868b4bf28bcb45c792fce896/jdk-8u231-linux-x64.tar.gz
$ tar -zxvf jdk-8u231-linux-x64.tar.gz /usr/local/java/
Modify the /etc/profile file
$ vi /erc/profile
Config Java environment variables, append the following content into the file
export JAVA_HOME=/usr/local/java/jdk1.8.0_231
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=$PATH:${JAVA_HOME}/bin
Enable the configuration
$ source /etc/profile
$ java -version
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b12, mixed mode)
Step 2: Install apache tomcat
Download apache tomcat and install to /usr/local/tomcat/
$ http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.47/bin/apache-tomcat-8.5.47.tar.gz
$ tar -zxvf apache-tomcat-8.5.47.tar.gz /usr/local/tomcat/
Step 3: Deploy the load generator into tomcat
An executable war package has been provided in loadGen/build/, you can also build the source code uses
jaroreclipse IDE
Deploy the web package into tomcat webapp/
$ mv loadGen/build/sdcloud.war /usr/local/tomcat/apache-tomcat-8.5.47/webapp
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/startup.sh
Validate if the depolyment is successful
$ curl http://localhost:8080/sdcloud/
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
...
welcome to the load generator page!
...
</body>
The command will output the content of a welcome page, means the war package has been deployed successfully
Then modify the configuration file in loadGen and restart tomcat
$ vi LoadGen/src/main/resources/conf/sys.properties
Modify the content of sys.properties as below:
# URL of web inference service that can be accessed by http
imageClassifyBaseURL=http://192.168.3.130:31500/gpu
# node IP that deployed load generator
serverIp=192.168.3.110
# the port of RMI service provided by load generator
rmiPort=22222
# window size of latency recorder, which can be seen from GUI page
windowSize=60
# record interval of latency, default: 1000ms
recordInterval=1000
Then restart the tomcat
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/shutdown.sh
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/startup.sh
If error occured when restart like java.rmi.server.ExportException: Port already in use: 22222, we need to kill the process that uses this port and restart tomcat
$ netstat -apn |grep 22222
tcp6 1 0 202.113.8.12:36284 127.0.0.1:22222 USING 35487/java
$ kill 35487
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/startup.sh
Step 4: Test the load generator
Open the exlporer and visit url http://192.168.3.110:8080/sdcloud/
The GUI page is shown as below:
Type 1: Using the URL interfaces
We provide four URL interfaces to control the load generator as below:
| url interface | description | type | parameter |
|---|---|---|---|
startOnlineQuery.do?intensity=1&serviceId=0 |
start to generate the request load | GET | intensity: concurrent requests per second (RPS) serviceId: inference service index id |
goOnlineQuery.do?serviceId=0 |
visit the real-time latency page | GET | serviceId: inference service index id |
setIntensity.do?intensity=20&serviceId=0 |
change the RPS dynamically | GET | intensity: RPS serviceId: inference service index id |
stopOnlineQuery.do?serviceId=0 |
stop the load generator | GET | --- |
The index id of mobilenet-coco300-tf is 0, and the second inference service should be set to 1, ..., etc. The supported service in load generator is easy to scale
For examle, firstly, click the startOnlineQuery.do?intensity=1&serviceId=0 link to generate request loads, and you will see the page in a waiting state (circle loading), then after $windowSize seconds, click the goOnlineQuery.do?serviceId=0 link to watch the real-time latency as below.
Metrics in real-time latency watch page:
- realRPS: The concurrent requests number of last second from client, RPS (request per second)
- realQPS: The response number of last second in server, QPS (query per second)
- AvgRPS: The average concurrent RPS in
$windowsizetime scale - AvgQPS: The average QPS in
$windowsizetime scale - SR: The average service rate in
$windowsizetime scale,SR=AvgQPS/AvgRPS*100% - queryTime: The 99th tail-latency of the concurrent requests per second
- Avg99th: The average
queryTimesin$windowsizetime scale
When the load generator is running, click the setIntensity.do?intensity=N&serviceId=0 link to change the RPS, please replace N to the number of concurrent requests per second you want. Finally, click stopOnlineQuery.do?serviceId=0 to stop the load testing
Type 2: Using the RMI interfaces
We also provide the Java RMI interfaces for the remote funciton calls in users' external code without clicking URL links. Using RMI, you can control the load generator and collect metric data. The RMI interface file is in LoadGen/src/main/java/scs/util/rmi/LoadInterface.java, the interface functions are shown as below:
package scs.util.rmi;
import java.rmi.Remote;
import java.rmi.RemoteException;
/**
* RMI interface class, which is used to control the load generator
* The functions can be call by remote client code
* @author Yanan Yang
* @date 2019-11-11
* @address TANKLab, TianJin University, China
*/
public interface LoadInterface extends Remote{
public float getWindowAvgPerSec99thLatency(int serviceId) throws RemoteException; //return the value of Avg99th
public float getRealPerSec99thLatency(int serviceId) throws RemoteException; //return the value of queryTime
//public float getWindowSize95thRealLatency(int serviceId) throws RemoteException; //return the value of queryTime (95th), unused
//public float getLcCurLatency999thRealLatency(int serviceId) throws RemoteException; //return the value of queryTime (99.9th), unused
public int getRealQueryIntensity(int serviceId) throws RemoteException; //return the value of realQPS
public int getRealRequestIntensity(int serviceId) throws RemoteException; //return the value of realRPS
public float getWindowAvgServiceRate(int serviceId) throws RemoteException; //return the value of SR
public void execStartHttpLoader(int serviceId) throws RemoteException; //start load generator for serviceId
public void execStopHttpLoader(int serviceId) throws RemoteException; //stop load generator for serviceId
public int setIntensity(int intensity,int serviceId) throws RemoteException; //change the RPS dynamically
}
When tomcat starts, the server side will automatically setup the RMI service using serverIp and rmiPort in LoadGen/src/main/resources/conf/sys.properties, the RMI service function is shown as below:
package scs.util.rmi;
import java.net.MalformedURLException;
import java.rmi.Naming;
import java.rmi.RemoteException;
import java.rmi.registry.LocateRegistry;
public class RmiService {
private static RmiService loadService=null;
private RmiService(){}
public synchronized static RmiService getInstance() {
if (loadService == null) {
loadService = new RmiService();
}
return loadService;
}
public void setupService(String serverIp,int rmiPort) {
try {
System.setProperty("java.rmi.server.hostname",serverIp);
LocateRegistry.createRegistry(rmiPort);
LoadInterface load = new LoadInterfaceImpl();
Naming.rebind("rmi://"+serverIp+":"+rmiPort+"/load", load);
} catch (RemoteException e) {
e.printStackTrace();
} catch (MalformedURLException e) {}
}
}
The client side need to setup the RMI connection before controling the load generator, a stand connection function is shown as below:
private static void setupRmiConnection(){
try {
LoadInterface loader=(LoadInterface) Naming.lookup("rmi://192.168.3.110:22222/load");
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (RemoteException e) {
e.printStackTrace();
} catch (NotBoundException e) {
e.printStackTrace();
}
if(loader!=null){
System.out.println(Repository.loaderRmiUrl +"connection successed");
}else{
System.out.println(Repository.loaderRmiUrl +"connection failed");
}
}
More tutorial of RMI interface can be see from here
Performance evaluation of loadGen
We evaluate the performance of load generator tool and web inference service and show the results as below:
Performance testing of load generator
To aviod the performance bottleneck of web service interfering the testing results, we set the request url in LoadGen/src/main/resources/conf/sys.properties to an empty url (this url does nothing and just returns 'helloworld')
mageClassifyBaseURL=http://192.168.3.130:31500/helloworld
Then we test the concurrent ability of load generator and collect the latency data, the client and server are deployed individually on two nodes that connected with 1Gbps WLAN
Fig.1 depicts the 99th tail-latency collected by load generator with the workloads ranges from RPS=1 to RPS=2000, the requests are sent using multi-threads in open-loop, the worst 99th tail-latency < 250ms when the RPS=2000, which shows the low queue latency in load generator. Fig.2 shows the 99th tail-latency increases linearly with the RPS, this demonstrates the load generator is well designed and has a good performance of workload scalability. Fig.3 shows the CPU usage in server end with increasing RPS, which has a same trend with the tail-latency in Fig.1. The inference service consumes < 0.5 CPU core when RPS=400, while the CPU usage no more than 2 CPU cores when RPS=2000, it demonstrates the low overhead of guicorn+flask framework
Future work
The latest released version of load generator has satisfied our experiment needs, in the future, we plan to implement these functions as below:
- Distributed load generator
- Diversified output statistics (e.g., PDF, hist graph)
Bug report & Question
We have used the load generator for a long time in our work [2,3], and fixed many bugs that have been found. If you have some new findings, please contact us via Email: ynyang@tju.edu.cn
Reference
[1] Kasture H, Sanchez D. Tailbench: a benchmark suite and evaluation methodology for latency-critical applications[C]//2016 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 2016: 1-10.
[2] Y. Yang, L. Zhao, Z. Li, L. Nie, P. Chen and K. Li. ElaX: Provisioning Resource Elastically for Containerized Online Cloud Services[C]//2019 IEEE 21st International Conference on High Performance Computing and Communications (HPCC). IEEE, 2019: 1987-1994.
[3] L. Zhao, Y. Yang, K. Zhang, etc. Rhythm: Component-distinguishable Workload
Deployment in Datacenters[C]//EuroSys 2020. ACM. Under review
基于Java的支持可变QPS的http负载生成器，提供交互界面和RMI接口的更多相关文章
- 【公开课】【阿里在线技术峰会】魏鹏：基于Java容器的多应用部署技术实践
对于公开课,可能目前用不上这些,但是往往能在以后想解决方案的时候帮助到我.以下是阿里对公开课的整理 摘要: 在首届阿里巴巴在线峰会上,阿里巴巴中间件技术部专家魏鹏为大家带来了题为<基于Java容 ...
- Spring Boot 2.2 正式发布，大幅性能提升 + Java 13 支持
之前 Spring Boot 2.2没能按时发布,是由于 Spring Framework 5.2 的发布受阻而推迟.这次随着 Spring Framework 5.2.0 成功发布之后,Spring ...
- 9个基于Java的搜索引擎框架
在这个信息相当繁杂的互联网时代,我们已经学会了如何利用搜索引擎这个强大的利器来找寻目标信息,比如你会在Google上搜索情人节如何讨女朋友欢心,你也会在百度上寻找正规的整容医疗机构(尽管有很大一部分广 ...
- 基于java平台的常用资源整理
这里整理了基于java平台的常用资源 翻译 from :akullpp | awesome-java 大家一起学习,共同进步. 如果大家觉得有用,就mark一下,赞一下,或评论一下,让更多的人知道.t ...
- 基于Java Netty框架构建高性能的部标808协议的GPS服务器
使用Java语言开发一个高质量和高性能的jt808 协议的GPS通信服务器,并不是一件简单容易的事情,开发出来一段程序和能够承受数十万台车载接入是两码事,除去开发部标808协议的固有复杂性和几个月长周 ...
- 基于Java Mina框架的部标808服务器设计和开发
在开发部标GPS平台中,部标808GPS服务器是系统的核心关键,决定了部标平台的稳定性和行那个.Linux服务器是首选,为了跨平台,开发语言选择Java自不待言. 我们为客户开发的部标服务器基于Min ...
- 这里整理了基于java平台的常用资源
这里整理了基于java平台的常用资源 翻译 from :akullpp | awesome-java 大家一起学习,共同进步. 如果大家觉得有用,就mark一下,赞一下,或评论一下,让更多的人知道.t ...
- memcached学习——常用命令+基于java客户端的3种简单实现（二）
常用命令: memcached设计的原则就是简单,所以支持的命令也不是特别多~ 1.查看memcached的状态,主要用于分析内存的使用状况.优化内存分配等 stats 查看memcached的运行状 ...
- 基于Java图片数据库Neo4j 3.0.0发布 全新的内部架构
基于Java图片数据库Neo4j 3.0.0发布 全新的内部架构 Neo4j 3.0.0 正式发布,这是 Neo4j 3.0 系列的第一个版本.此版本对内部架构进行了全新的设计;提供给开发者更强大的生 ...
随机推荐
- spring boot 配置虚拟目录
如上图,关键地方有两个: 1.下方的 web.upload-path (配置本地文件路径) 2.上方一串配置,具体代码如下: profiles: include: paperIdentify acti ...
- WPF外包团队：2019 WPF数据监控系统案例演示
项目版权均为客户所有,如有WPF项目外包欢迎联系我们. QQ:372900288 TEL:13911652504 WX:Liuxiang0884
- (CSDN迁移)js中的判空
<c:if test="${something.something == NaN}">style="display: none;"</c:if ...
- webpack打包时候去掉console.log配置
new UglifyJsPlugin({ uglifyOptions: { compress: { warnings: false, drop_console: true,//console pure ...
- sql 表的连接 inner join、full join、left join、right join、natural join
一.内连接-inner jion : SELECT * FROM table1 INNER JOIN table2 ON table1.field1 compopr table2.field2 INN ...
- PyCharm的安装方法及设置中文界面
pycharm官网下载安装包:https://www.jetbrains.com/pycharm/download/#section=windows 下载中文语言包:https://github.co ...
- 微信小程序之自定义导航栏（可实现动态添加）以及swiper（swiper-item）实现自动切换，导航标题也跟着切换
<view class="movie-container"> <!-- 导航栏 --> <view > <scroll-view scro ...
- Visual Studio特性学习
官方档案: https://docs.microsoft.com/en-us/visualstudio/get-started/visual-studio-ide?view=vs-2019
- 常见的几种异常类型 Exception
常见异常类型:Java中的异常分为两大类: 1.Checked Exception(非Runtime Exception) 2.Unchecked Exception(Runtime Exceptio ...
- Java连接数据库——最基础的方式
JAVAWEB实现增删查改(图书信息管理)之Util类 Util.java ↓ package BookSystem.Other; import java.sql.*; import java.ut ...