【原创】大数据基础之Kudu（6）kudu tserver内存占用统计分析

【【原创】大数据基础之Kudu（6）kudu tserver内存占用统计分析】的更多相关文章

【原创】大数据基础之Kudu（6）kudu tserver内存占用统计分析

kudu tserver占用内存过高后会拒绝部分写请求,日志如下: 19/06/01 13:34:12 INFO AsyncKuduClient: Invalidating location 34b1c13d04664cc8bae6689d39b08b77($kudu_tserver:7050) for tablet 858c055c456549569af77d14eaf997e5: Service unavailable: Soft memory limit exceeded (at 92.3…

【原创】大数据基础之Zookeeper（2）源代码解析

核心枚举 public enum ServerState { LOOKING, FOLLOWING, LEADING, OBSERVING; } zookeeper服务器状态:刚启动LOOKING,follower是FOLLOWING,leader是LEADING,observer是OBSERVING: public enum LearnerType { PARTICIPANT, OBSERVER; } 简单来说,zookeeper启动的核心类是QuorumPeerMain,启动之后会加载配置,…

【原创】大数据基础之Kudu（1）简介、安装、使用

kudu 1.7 官方:https://kudu.apache.org/ 一简介 kudu有很多概念,有分布式文件系统(HDFS),有一致性算法(Zookeeper),有Table(Hive Table),有Tablet(Hive Table Partition),有列式存储(Parquet),有顺序和随机读取(HBase),所以看起来kudu是一个轻量级的 HDFS + Zookeeper + Hive + Parquet + HBase,除此之外,kudu还有自己的特点,快速写入+读取,使…

【原创】大数据基础之Kudu（5）kudu增加或删除目录/数据盘

kudu加减数据盘不能直接修改配置fs_data_dirs后重启,否则会报错: Check failed: _s.ok() Bad status: Already present: FS layout already exists; not overwriting existing layout: FSManager roots already exist: /data0/kudu/data 官方解释如下: When Kudu starts, it checks each configured…

【原创】大数据基础之Kudu（4）spark读写kudu

spark2.4.3+kudu1.9 1 批量读 val df = spark.read.format("kudu") .options(Map("kudu.master" -> "master:7051", "kudu.table" -> "impala::test_db.test_table")) .load df.createOrReplaceTempView("tmp_tabl…

【原创】大数据基础之Flume（2）应用之kafka-kudu

应用一:kafka数据同步到kudu 1 准备kafka topic # bin/kafka-topics.sh --zookeeper $zk:2181/kafka -create --topic test_sync --partitions 2 --replication-factor 2 WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could coll…