【原创】大叔经验分享(58)kudu写入压力大时报错
kudu写入压力大时报错
19/05/18 16:53:12 INFO AsyncKuduClient: Invalidating location fd52e4f930bc45458a8f29ed118785e3(server002:7050) for tablet 4259921cdcca4776b37771659a8cafb3: Service unavailable: Soft memory limit exceeded (at 106.05% of capacity). See https://kudu.apache.org/releases/1.7.0-cdh5.16.1/docs/troubleshooting.html
这是个INFO日志,只是在内存超过hard的80%时拒绝写入,可能请求导致部分写入超时;
参数调优(增大以下参数):
- 1 --memory_limit_hard_bytes
- 2 --maintenance_manager_num_threads (Generally, the recommended ratio of maintenance manager threads to data directories is 1:3.)
- 3 --block_cache_capacity_mb (memory_limit_hard_bytes的30%到50%)
原文:
Kudu has a hard and soft memory limit. The hard memory limit is the maximum amount a Kudu process is allowed to use, and is controlled by the --memory_limit_hard_bytes flag. The soft memory limit is a percentage of the hard memory limit, controlled by the flag memory_limit_soft_percentage and with a default value of 80%, that determines the amount of memory a process may use before it will start rejecting some write operations.
If the logs or RPC traces contain messages like
Service unavailable: Soft memory limit exceeded (at 96.35% of capacity)
then Kudu is rejecting writes due to memory backpressure. This may result in write timeouts. There are several ways to relieve the memory pressure on Kudu:
- If the host has more memory available for Kudu, increase --memory_limit_hard_bytes.
- Increase the rate at which Kudu can flush writes from memory to disk by increasing the number of disks or increasing the number of maintenance manager threads --maintenance_manager_num_threads. Generally, the recommended ratio of maintenance manager threads to data directories is 1:3.
- Reduce the volume of writes flowing to Kudu on the application side.
Finally, on versions of Kudu prior to 1.8, check the value of --block_cache_capacity_mb. This setting determines the maximum size of Kudu’s block cache. While a higher value can help with read and write performance, do not raise --block_cache_capacity_mb above the memory pressure threshold, which is --memory_pressure_percentage (default 60%) of --memory_limit_hard_bytes, as this will cause Kudu to flush aggressively even if write throughput is low. Keeping --block_cache_capacity_mb below 50% of the memory pressure threshold is recommended. With the defaults, this means --block_cache_capacity_mb should not exceed 30% of --memory_limit_hard_bytes. On Kudu 1.8 and higher, servers will refuse to start if the block cache capacity exceeds the memory pressure threshold.
调优:
https://kudu.apache.org/docs/scaling_guide.html#memory
https://kudu.apache.org/docs/scaling_guide.html#_verifying_if_a_memory_limit_is_sufficient
参考:
https://kudu.apache.org/docs/troubleshooting.html
https://kudu.apache.org/docs/troubleshooting.html#memory_limits
【原创】大叔经验分享(58)kudu写入压力大时报错的更多相关文章
- 【原创】大叔经验分享(13)spark运行报错WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
本地运行spark报错 18/12/18 12:56:55 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting ...
- 【原创】大叔经验分享(78)hive查询报错NoViableAltException
Hive或spark中执行sql字符常量包含;时会报错,比如 select instr('abc;abc', ';'); 报错 NoViableAltException(-1@[147:1: sele ...
- 【原创】经验分享:一个小小emoji尽然牵扯出来这么多东西?
前言 之前也分享过很多工作中踩坑的经验: 一个线上问题的思考:Eureka注册中心集群如何实现客户端请求负载及故障转移? [原创]经验分享:一个Content-Length引发的血案(almost.. ...
- 【原创】大叔经验分享(54)flume kudu sink运行一段时间kudu client报错
flume kudu sink运行一段时间报错: 19/05/05 10:15:56 WARN client.ConnectToCluster: Error receiving a response ...
- 【原创】大叔经验分享(63)kudu vs parquet
一 对比 存储空间对比: 查询性能对比: 二 设计方案 将数据拆分为:历史数据(hdfs+parquet+snappy)+ 近期数据(kudu),可以兼具各种优点: 1)整体低于10%的磁盘占用: 2 ...
- 【原创】大叔经验分享(53)kudu报错unable to find SASL plugin: PLAIN
kudu安装后运行不正常,master中找不到任何tserver,查看tserver日志发现有很多报错: Failed to heartbeat to master:7051: Invalid arg ...
- 【原创】大叔经验分享(62)kudu副本数量
kudu的副本数量是在表上设置,可以通过命令查看 # sudo -u kudu kudu cluster ksck $master ... Summary by table Name | RF | S ...
- 【原创】大叔经验分享(61)kudu rebalance报错
kudu rebalance命令报错 terminate called after throwing an instance of 'std::regex_error' what(): regex_e ...
- 【原创】大叔经验分享(60)hive和spark读取kudu表
从impala中创建kudu表之后,如果想从hive或spark sql直接读取,会报错: Caused by: java.lang.ClassNotFoundException: com.cloud ...
随机推荐
- python+selenium 切换至iframe
方法一: from selenium import webdriver driver = webdriver.Firefox() driver.switch_to.frame(0) # 1.用fram ...
- 2.JSON.stringify()Object
JSON.stringify() JSON 通常用于与服务端交换数据. 在向服务器发送数据时一般是字符串. 我们可以使用 JSON.stringify() 方法将 JavaScript 对象转换为字符 ...
- redis4. dict字典
基础数据结构: (注意dict是字典,dict->type是相关函数指针, dict->type->keyDup是执行该方法) 具体调用链路: 渐进式rehash: 新增/删除时: ...
- Android的内部存储
路径:/data/data/包名/ this.getCacheDir() = /data/data/com.example.qq/cache/ getFilesDir() = /data/data/c ...
- Python3+RobotFramewok RIDE环境搭建(一)
工欲善其事,必先利其器,RIDE目前可以比较稳定的支持python3了,这里简单介绍一下环境的部署,建议多看看读一读官方文档,对能力提升很有帮助,很简单的三步: (1)安装python3,不详细介绍了 ...
- R语言与概率统计(三) 多元统计分析(下)广义线性回归
广义线性回归 > life<-data.frame( + X1=c(2.5, 173, 119, 10, 502, 4, 14.4, 2, 40, 6.6, + 21.4, 2.8, 2. ...
- Windows下nginx配置多台服务器做负载均衡
Nginx (engine x) 是一个高性能的HTTP和反向代理服务,也是一个IMAP/POP3/SMTP服务. Nginx是一款轻量级的Web 服务器/反向代理服务器及电子邮件(IMAP/POP3 ...
- PHP SQL注入
开发者容易遗漏的输入点: HTTP头 X-Forwarded-For 获取用户ip User-Agent 获取浏览器 Referer 获取之 ...
- C++学习笔记-继承中的构造与析构
C++存在构造函数与析构函数,继承中也存在构造和析构函数.继承中的构造和析构函数与普通的构造析构有细微差别. 赋值兼容性原则 #include "iostream" using n ...
- A Mixed Flash Translation Layer Structure for SLC-MLC Combined Flash Memory System
http://blog.sina.com.cn/s/blog_502c8cc40100pztk.html 摘要 1.In this paper, we propose the SLC-MLC mixe ...