【原创】大数据基础之Impala(3)部分调优
1)将coordinator和executor角色分离
By default, each host in the cluster that runs the impalad daemon can act as the coordinator for an Impala query, execute the fragments of the execution plan for the query, or both. During highly concurrent workloads for large-scale queries, especially on large clusters, the dual roles can cause scalability issues:
- The extra work required for a host to act as the coordinator could interfere with its capacity to perform other work for the earlier phases of the query. For example, the coordinator can experience significant network and CPU overhead during queries containing a large number of query fragments. Each coordinator caches metadata for all table partitions and data files, which can be substantial and contend with memory needed to process joins, aggregations, and other operations performed by query executors.
- Having a large number of hosts act as coordinators can cause unnecessary network overhead, or even timeout errors, as each of those hosts communicates with the statestored daemon for metadata updates.
- The “soft limits” imposed by the admission control feature are more likely to be exceeded when there are a large number of heavily loaded hosts acting as coordinators.
2)default_pool_max_requests,默认是200,要根据自己集群的内存规模以及每个查询需要的内存进行调整;
Maximum number of concurrent outstanding requests allowed to run before incoming requests are queued. Because this limit applies cluster-wide, but each Impala node makes independent decisions to run queries immediately or queue them, it is a soft limit; the overall number of concurrent queries might be slightly higher during times of heavy load. A negative value indicates no limit. Ignored if fair_scheduler_config_path and llama_site_path are set.
3)开启kerberos之后,通过jdbc访问需要做客户端load balance,因为jdbc url里需要携带对应server的principal;
【原创】大数据基础之Impala(3)部分调优的更多相关文章
- 【原创】大数据基础之Impala(1)简介、安装、使用
impala2.12 官方:http://impala.apache.org/ 一 简介 Apache Impala is the open source, native analytic datab ...
- 大数据技术 - MapReduce的Shuffle及调优
本章内容我们学习一下 MapReduce 中的 Shuffle 过程,Shuffle 发生在 map 输出到 reduce 输入的过程,它的中文解释是 “洗牌”,顾名思义该过程涉及数据的重新分配,主要 ...
- 【原创】大数据基础之Impala(2)实现细节
一 架构 Impala is a massively-parallel query execution engine, which runs on hundreds of machines in ex ...
- 【原创】大数据基础之Zookeeper(2)源代码解析
核心枚举 public enum ServerState { LOOKING, FOLLOWING, LEADING, OBSERVING; } zookeeper服务器状态:刚启动LOOKING,f ...
- 【原创】大数据基础之Benchmark(2)TPC-DS
tpc 官方:http://www.tpc.org/ 一 简介 The TPC is a non-profit corporation founded to define transaction pr ...
- 【原创】大数据基础之词频统计Word Count
对文件进行词频统计,是一个大数据领域的hello word级别的应用,来看下实现有多简单: 1 Linux单机处理 egrep -o "\b[[:alpha:]]+\b" test ...
- 【原创】大数据基础之ElasticSearch(4)es数据导入过程
1 准备analyzer 内置analyzer 参考:https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis- ...
- 【原创】大数据基础之Hive(5)性能调优Performance Tuning
1 compress & mr hive默认的execution engine是mr hive> set hive.execution.engine;hive.execution.eng ...
- 【原创】大数据基础之ElasticSearch(5)重要配置及调优
Index Settings 重要索引配置 Index level settings can be set per-index. Settings may be: 1 static 静态索引配置 Th ...
随机推荐
- CSS 实现自动换行、强制换行、强制不换行的属性
实现效果 1.自动换行: word-wrap:break-word; word-break:normal; 2.强制换行: word-break:break-all; 按字符截断换行 /* ...
- plus webview关闭事件监听
plus.webview.currentWebview().addEventListener("close",function(){ },false);
- 2019-04-28 Mybatis generator逆向工程生成的Example代码分析
今天主要对Mybatis generator生成的DAO层等进行分析,讲解Example类的使用和扩展 1.先在数据库建表 CREATE TABLE `department` ( `fid` ) NO ...
- PowerBi利用Python Script绕过ODBC来导入MongoDB数据
- springboot全局捕获异常
捕获 捕获原理,使用AOP技术,采用异常通知. 1.捕获返回json格式 2.捕获返回页面 步骤: 1.@ControllerAdvice(pasePackage="") 注释异常 ...
- 网路知识总结(session&&Cookie&&三次握手&&请求头)
1. 请说明Session和Cookie的作用和区别 1) Cookie 存在前端 前端需要拿着cookie访问后端,Session在服务器上(文件,数据库,如Redis) 2) web访问Serve ...
- 远离go path,弃用go get,使用go mod 进行go语言的学习
标题说的是go语言的学习,因为我也没做过开发 文章要解决的仅仅是一个问题 当你使用go get 无论如何get不到所需的包的问题 第一步就是下载goland 新手极其推荐goland,因为直接使用gl ...
- 修改linux下yum镜像源为国内镜像
修改为163yum源-mirrors.163.com 1.首先备份系统自带yum源配置文件/etc/yum.repos.d/CentOS-Base.repo [root@localhost ~]# m ...
- SNOI 2019 字符串
SNOI 2019 字符串 题目 题解: 解法一: 记一个数组\(f\),\(f[i]=\min_j\ s[j]\neq s[j+1] (j\geq i)\),直接sort即可,复杂度\(O(nlog ...
- Educational Codeforces Round 63 (Rated for Div. 2)
传送门 A. Reverse a Substring 题意: 给你一串 s,让你判断能否通过反转区间[l,r]的元素,使得反转后的串的字典序小于 s: 如果能,输出 "YES",并 ...