列式分布式数据库ClickHouse部署

简介

Clickhouse是俄罗斯yandex公司2016年开源的用于OLAP的列式数据库。

使用C++语言编写，支持SQL实时查询。

OLTP，更强调数据的完整性，行式存储；

OLAP，更强调数据处理的速度，列式存储更有优势。

适用场景

• 绝大多数请求都是读请求

• 数据量很大

• 数据经常是以大的批次进行整体更新

• 对事务的要求不是必须的，通常只要求数据的最终一致性

部署

参考链接：https://www.cnblogs.com/EminemJK/p/15138536.html

部署形态（3节点6实例）

使用批量脚本

  [root@dev-clickhouse1 ~]# cat /etc/hosts
  127.0.0.1  localhost localhost.localdomain localhost4  localhost4.localdomain4
  ::1     localhost localhost.localdomain  localhost6 localhost6.localdomain6
  10.20.210.138  dev-clickhouse1 ch1
  10.20.210.139  dev-clickhouse2 ch2
  10.20.210.140  dev-clickhouse3 ch3

  [root@dev-clickhouse1 ~]# cat bin/msh
  #!/usr/bin/bash
  hosts=(
  	"ch1"
    "ch2"
    "ch3"
  )

  for host in  "${hosts[@]}"
  do
  	echo "begin to run " $@  " on " $host
    ssh root@$host $@
  done

  [root@dev-clickhouse1 ~]# cat bin/mcp
  #!/usr/bin/bash
  hosts=(
  	"ch1"
    "ch2"
    "ch3"
  )

  for host in  "${hosts[@]}"
  do
  	echo "begin to scp " $@  " on " $host
    scp -r $1 $host:$2
  done  

环境配置

关闭selinux，关闭防火墙，配置ssh免密

版本信息

软件	版本
clickhouse	22.1.3.7
zookeeper	20.10.15
内核版本	3.10.0-1160.el7.x86_64
docker	20.10.15

容器卷挂载

主机目录	容器目录
/data/lib/clickhouse	/var/lib/clickhouse
/data/lib/zookeeper/	/
/data/lib/clickhouse-sub	/var/lib/clickhouse

Zookeeper集群部署

• 每个节点创建目录

​mkdir -p /data/lib/zookeeper/{data,datalog,logs,conf}

​chmod 777 -R /data/lib/zookeeper/

• 每个节点写入配置文件

  [root@dev-clickhouse2 ~]# cat /data/lib/zookeeper/conf/zoo.cfg
  dataDir=/data
  dataLogDir=/datalog
  tickTime=2000
  initLimit=5
  syncLimit=2
  clientPort=2181
  autopurge.snapRetainCount=3
  autopurge.purgeInterval=0
  maxClientCnxns=60
  4lw.commands.whitelist=*

  server.1=10.20.210.138:2888:3888
  server.2=10.20.210.139:2888:3888
  server.3=10.20.210.140:2888:3888

• Ch1

  [root@dev-clickhouse1 ~]# cat container/zookeeper/run
  #!/bin/bash

  cmd=(
          docker run
          --privileged=true
          -d
          -p 2181:2181
          -p 2888:2888
          -p 3888:3888
          --name zk
          --restart always
          -v /data/lib/zookeeper/data:/data
          -v /data/lib/zookeeper/datalog:/datalog
          -v /data/lib/zookeeper/logs:/logs
          -v /data/lib/zookeeper/conf:/conf
          --network host
          -e ZOO_MY_ID=1
          zookeeper
  )

  "${cmd[@]}"

  # ./run

• Ch2

  [root@dev-clickhouse2 ~]# cat container/zookeeper/run
  #!/bin/bash

  cmd=(
          docker run
          --privileged=true
          -d
          -p 2181:2181
          -p 2888:2888
          -p 3888:3888
          --name zk
          --restart always
          -v /data/lib/zookeeper/data:/data
          -v /data/lib/zookeeper/datalog:/datalog
          -v /data/lib/zookeeper/logs:/logs
          -v /data/lib/zookeeper/conf:/conf
          --network host
          -e ZOO_MY_ID=2
          zookeeper
  )

  "${cmd[@]}"

  ## ./run

• Ch3

  [root@dev-clickhouse3 ~]# cat container/zookeeper/run
  #!/bin/bash

  cmd=(
          docker run
          --privileged=true
          -d
          -p 2181:2181
          -p 2888:2888
          -p 3888:3888
          --name zk
          --restart always
          -v /data/lib/zookeeper/data:/data
          -v /data/lib/zookeeper/datalog:/datalog
          -v /data/lib/zookeeper/logs:/logs
          -v /data/lib/zookeeper/conf:/conf
          --network host
          -e ZOO_MY_ID=3
          zookeeper
  )

  "${cmd[@]}"

  ## ./run

Clickhouse集群部署

创建用户

msh useradd clickhouse

每个节点创建目录

mkdir -p /root/container/{clickhouse,clickhouse-sub}
mkdir -p /data/lib/clickhouse/{clickhouse-server,data,logs}
mkdir -p /data/lib/clickhouse-sub/{clickhouse-server,data,logs}

chown -R clickhouse: /data/lib/clickhouse/ /data/lib/clickhouse-sub/

启动脚本

Ch1

[root@dev-clickhouse1 ~]# cat /root/container/clickhouse/run
#!/bin/bash

name=ch1-shard1-main

docker stop $name
docker rm $name

cmd=(
       docker run
       --restart always
       -d
       --name $name
       --network host
       --ulimit nofile=262144:262144
       -v /data/lib/clickhouse/data/:/var/lib/clickhouse/
       -v /data/lib/clickhouse/clickhouse-server/:/etc/clickhouse-server/
       -v /data/lib/clickhouse/logs/:/var/log/clickhouse-server/
#        -p 9001:9001 -p 8124:8124 -p 9010:9010
       yandex/clickhouse-server
)

"${cmd[@]}"

[root@dev-clickhouse1 ~]# cat /root/container/clickhouse-sub/run-sub
#!/bin/bash

name=ch1-shard2-sub

docker stop $name
docker rm $name

cmd=(
       docker run
       --restart always
       -d
       --name $name
       --network host
       --ulimit nofile=262144:262144
       -v /data/lib/clickhouse-sub/data/:/var/lib/clickhouse/
       -v /data/lib/clickhouse-sub/clickhouse-server/:/etc/clickhouse-server/
       -v /data/lib/clickhouse-sub/logs/:/var/log/clickhouse-server/
#        -p 9001:9001 -p 8124:8124 -p 9010:9010
       yandex/clickhouse-server
)

"${cmd[@]}"

Ch2

[root@dev-clickhouse1 ~]# cat /root/container/clickhouse/run
#!/bin/bash

name=ch1-shard2-main

docker stop $name
docker rm $name

cmd=(
        docker run
        --restart always
        -d
        --name $name
        --network host
        --ulimit nofile=262144:262144
        -v /data/lib/clickhouse/data/:/var/lib/clickhouse/
        -v /data/lib/clickhouse/clickhouse-server/:/etc/clickhouse-server/
        -v /data/lib/clickhouse/logs/:/var/log/clickhouse-server/
#        -p 9001:9001 -p 8124:8124 -p 9010:9010
        yandex/clickhouse-server
)

"${cmd[@]}"

[root@dev-clickhouse1 ~]# cat /root/container/clickhouse-sub/run-sub
#!/bin/bash

name=ch1-shard3-sub

docker stop $name
docker rm $name

cmd=(
        docker run
        --restart always
        -d
        --name $name
        --network host
        --ulimit nofile=262144:262144
        -v /data/lib/clickhouse-sub/data/:/var/lib/clickhouse/
        -v /data/lib/clickhouse-sub/clickhouse-server/:/etc/clickhouse-server/
        -v /data/lib/clickhouse-sub/logs/:/var/log/clickhouse-server/
#        -p 9001:9001 -p 8124:8124 -p 9010:9010
        yandex/clickhouse-server
)

"${cmd[@]}"

Ch3

[root@dev-clickhouse1 ~]# cat /root/container/clickhouse/run
#!/bin/bash

name=ch1-shard3-main

docker stop $name
docker rm $name

cmd=(
        docker run
        --restart always
        -d
        --name $name
        --network host
        --ulimit nofile=262144:262144
        -v /data/lib/clickhouse/data/:/var/lib/clickhouse/
        -v /data/lib/clickhouse/clickhouse-server/:/etc/clickhouse-server/
        -v /data/lib/clickhouse/logs/:/var/log/clickhouse-server/
#        -p 9001:9001 -p 8124:8124 -p 9010:9010
        yandex/clickhouse-server
)

"${cmd[@]}"

[root@dev-clickhouse1 ~]# cat /root/container/clickhouse-sub/run-sub
#!/bin/bash

name=ch1-shard1-sub

docker stop $name
docker rm $name

cmd=(
        docker run
        --restart always
        -d
        --name $name
        --network host
        --ulimit nofile=262144:262144
        -v /data/lib/clickhouse-sub/data/:/var/lib/clickhouse/
        -v /data/lib/clickhouse-sub/clickhouse-server/:/etc/clickhouse-server/
        -v /data/lib/clickhouse-sub/logs/:/var/log/clickhouse-server/
#        -p 9001:9001 -p 8124:8124 -p 9010:9010
        yandex/clickhouse-server
)

"${cmd[@]}"

配置文件

各实例公共部分（集群3分片2副本配置）

<listen_host>0.0.0.0</listen_host>

    <remote_servers>
        <xxx_cluster>
            <shard>
                <replica>
                    <host>ch1</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>ch3</host>
                    <port>9001</port>
                </replica>
            </shard>
            <shard>
                <replica>
                    <host>ch2</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>ch1</host>
                    <port>9001</port>
                </replica>
            </shard>
            <shard>
                <replica>
                    <host>ch3</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>ch2</host>
                    <port>9001</port>
                </replica>
            </shard>
        </xxx_cluster>
</remote_servers>

各实例独立部分

• ch1-shard1-main

    <zookeeper>
        <node>
            <host>ch1</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch2</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch3</host>
            <port>2181</port>
        </node>
</zookeeper>

    <macros>
        <shard>01</shard>
        <replica>cluster01-01-1</replica>
    </macros>

• ch2-shard2-main

    <zookeeper>
        <node>
            <host>ch2</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch1</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch3</host>
            <port>2181</port>
        </node>
</zookeeper>

    <macros>
        <shard>02</shard>
        <replica>cluster01-02-1</replica>
    </macros>

• ch3-shard3-main

    <zookeeper>
        <node>
            <host>ch3</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch2</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch1</host>
            <port>2181</port>
        </node>
</zookeeper>

    <macros>
        <shard>03</shard>
        <replica>cluster01-03-1</replica>
    </macros>

• ch3-shard1-sub

    <zookeeper>
        <node>
            <host>ch1</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch2</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch3</host>
            <port>2181</port>
        </node>
</zookeeper>

    <macros>
        <shard>02</shard>
        <replica>cluster01-02-2</replica>
    </macros>

• ch1-shard2-sub

    <zookeeper>
        <node>
            <host>ch2</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch1</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch3</host>
            <port>2181</port>
        </node>
</zookeeper>

    <macros>
        <shard>03</shard>
        <replica>cluster01-03-2</replica>
    </macros>

• ch2-shard3-sub

    <zookeeper>
        <node>
            <host>ch3</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch2</host>
            <port>2181</port>
        </node>
        <node>
            <host>ch1</host>
            <port>2181</port>
        </node>
</zookeeper>

    <macros>
        <shard>01</shard>
        <replica>cluster01-01-2</replica>
    </macros>

集群机制

clickhouse单机的性能足够强，集群机制相对简单。

数据副本（高可用）

官网链接：数据副本 视频链接

• 只有 MergeTree 系列里的表可支持副本

• 副本是表级别的，不是整个服务器级的。所以，服务器里可以同时有复制表和非复制表。

• 副本不依赖分片。每个分片有它自己的独立副本。

• 对于 INSERT 和 ALTER 语句操作数据的会在压缩的情况下被复制（更多信息，看 ALTER ）。

• 而 CREATE，DROP，ATTACH，DETACH 和 RENAME 语句只会在单个服务器上执行，不会被复制。


单节点写入成功，即返回成功；通过zookeeper-cluster去把数据复制到其他节点。

创建复制表

复制表

在表引擎名称上加上 **Replicated** 前缀。例如：**ReplicatedMergeTree**。

分片集群（横向扩展）

分布式表

副本虽然能够提高数据的可用性，降低丢失风险，但是每台服务器实际上必须容纳全量

数据，对数据的横向扩容没有解决。

要解决数据水平切分的问题，需要引入分片的概念。通过分片把一份完整的数据进行切

分，不同的分片分布到不同的节点上，再通过 Distributed 表引擎把数据拼接起来一同使用。

基础概念

什么是列式数据库

列式数据库独立存储每一列的数据。这允许仅从磁盘读取用于任何给定查询的列的数据。

代价是影响整行的操作成比例地变得更加昂贵。

列式数据库的同义词是面向列的数据库管理系统。

列式数据库的主要优势是：

• 仅使用众多列中的少数列的查询。

• 针对大量数据聚合查询。

• 逐列数据压缩，压缩比极高。

为什么叫ClickHouse

它是“点击流”和“数据仓库”的组合。它来自 Yandex.Metrica 的原始用例，ClickHouse 应该记录来自互联网各地的人们的所有点击记录，它仍然可以完成这项工作。

为什么不使用 MapReduce 之类的东西？

由于它们的高延迟，这些系统不适合在线查询。换句话说，它们不能用作 Web 界面的后端。

面向列的存储

源数据通常包含数百甚至数千列，而报表只能使用其中的几个。系统需要避免读取不必要的列，否则将浪费最昂贵的磁盘读取操作。

索引

ClickHouse 将数据结构保存在内存中，不仅允许读取已使用的列，还允许读取这些列的必要行范围。

数据压缩

将同一列的不同值存储在一起通常会导致更好的压缩率（与面向行的系统相比），因为在实际数据列中，相邻行通常具有相同或不同的值。除了通用压缩之外，ClickHouse 还支持可以使数据更加紧凑的专用编解码器。

矢量化查询执行

ClickHouse 不仅在列中存储数据，而且在列中处理数据。它可以提高 CPU 缓存利用率并允许使用SIMD CPU 指令。

可扩展性

ClickHouse 可以利用所有可用的 CPU 内核和磁盘来执行单个查询。不仅在单个服务器上，而且在集群的所有 CPU 内核和磁盘上也是如此。

高吞吐写入能力

ClickHouse采用类LSM Tree的结构，数据写入后定期在后台Compaction。通过类LSM tree

的结构，ClickHouse 在数据导入时全部是顺序 append 写，写入后数据段不可更改，在后台

compaction 时也是多个段 merge sort 后顺序写回磁盘。顺序写的特性，充分利用了磁盘的吞

吐能力，即便在 HDD 上也有着优异的写入性能。

官方公开 benchmark 测试显示能够达到 50MB-200MB/s 的写入吞吐能力，按照每行

100Byte 估算，大约相当于 50W-200W 条/s 的写入速度。

数据分区与线程级并行

ClickHouse 将数据划分为多个 partition，每个 partition 再进一步划分为多个 index

granularity(索引粒度)，然后通过多个CPU核心分别处理其中的一部分来实现并行数据处理。

在这种设计下，单条 Query 就能利用整机所有 CPU。极致的并行处理能力，极大的降低了查

询延时。

所以，ClickHouse 即使对于大量数据的查询也能够化整为零平行处理。但是有一个弊端

就是对于单条查询使用多 cpu，就不利于同时并发多条查询。所以对于高 qps 的查询业务，

ClickHouse 并不是强项。

核心概念

数据库引擎

https://clickhouse.com/docs/zh/engines/database-engines/

默认情况下，ClickHouse使用Atomic数据库引擎。

数据类型

https://clickhouse.com/docs/zh/sql-reference/data-types/

• 整形

• 布尔型

• 浮点型

• Decimal型

• 字符串型

• 枚举型

• 时间类型

• 数组

表引擎

ClickHouse 和 MySQL 类似，把表级的存储引擎插件化，根据表的不同需求可以设定不同

的存储引擎。目前包括合并树、日志、接口和其他四大类 20 多种引擎。

https://clickhouse.com/docs/zh/engines/table-engines/

表引擎（即表的类型）决定了：

• 数据的存储方式和位置，写到哪里以及从哪里读取数据

• 支持哪些查询以及如何支持。

• 并发数据访问。

• 索引的使用（如果存在）。

• 是否可以执行多线程请求。

• 数据复制参数。

表引擎类型

• MergeTree（合并树）

适用于高负载任务的最通用和功能最强大的表引擎。这些引擎的共同特点是可以快速插入数据并进行后续的后台数据处理。 MergeTree系列引擎支持数据复制（使用Replicated* 的引擎版本），分区和一些其他引擎不支持的其他功能。

• 日志

具有最小功能的轻量级引擎。当您需要快速写入许多小表（最多约100万行）并在以后整体读取它们时，该类型的引擎是最有效的。

• 集成引擎

用于与其他的数据存储与处理系统集成的引擎。

• 其他用于特定功能的

MergeTree引擎

Clickhouse 中最强大的表引擎当属 MergeTree （合并树）引擎及该系列（*MergeTree）中的其他引擎。

MergeTree 系列的引擎被设计用于插入极大量的数据到一张表当中。数据可以以数据片段的形式一个接着一个的快速写入，数据片段在后台按照一定的规则进行合并。相比在插入时不断修改（重写）已存储的数据，这种策略会高效很多。

建表语句与字段解释详见链接

特点

• 存储的数据按主键排序

• 如果指定了 分区键 的话，可以使用分区

在相同数据集和相同结果集的情况下 ClickHouse 中某些带分区的操作会比普通操作更快。查询中指定了分区键时 ClickHouse 会自动截取分区数据。这也有效增加了查询性能。比如，指定时日期为分区键时，不同日期的数据会保存的不同的文件中。

• 支持数据副本

ReplicatedMergeTree 系列的表提供了数据副本功能。

• 支持数据采样

指定对某个字段进行采用‘SAMPLE BY ’，在表申明时指定。

指定采样数据‘SAMPLE 0.1’，在select语句中指定。

!!! note "注意" 合并 引擎并不属于 *MergeTree 系列。

数据之间以逗号分隔。

数据存储

路径	意义
/var/lib/clickhouse	默认存储路径
data/	数据存储路径
metadata/	元数据存储路径
metadata/default/	default 库的元数据目录
data/default/t_stock/	default 库t_stock表 的数据目录

表由按主键排序的数据片段（DATA PART）组成。

当数据被插入到表中时，会创建多个数据片段并按主键的字典序排序。例如，主键是 (CounterID, Date) 时，片段中数据首先按 CounterID 排序，具有相同 CounterID 的部分按 Date 排序。

不同分区的数据会被分成不同的片段，ClickHouse 在后台合并数据片段以便更高效存储。不同分区的数据片段不会进行合并。合并机制并不保证具有相同主键的行全都合并到同一个数据片段中。

文件名	作用
checksums.txt	检查分区数据总数
columns.txt:	可查看该分区的数据列信息，包括名称和类型
count.txt	该分区文件夹下的数据条数
data.bin	每一列具体的属性，当然是压缩后的
data.mrk	索引的偏移量
minmax_date.idx	最小最大数据的记录
partition.dat	分区信息
Primary.idx	主键

操作

基本操作

clickhouse-client 连接数据库

--host, -h -– 服务端的host名称, 默认是localhost。您可以选择使用host名称或者IPv4或IPv6地址。

--port – 连接的端口，默认值：9000。注意HTTP接口以及TCP原生接口使用的是不同端口。

查看集群信息

SELECT * FROM system.clusters

查看集群macros 变量

SELECT * from system.macros ;

角色和用户管理

创建角色、授予权限

# 管理员
CREATE role admin on cluster xxx_cluster ;
GRANT ALL ON *.* TO admin WITH GRANT OPTION  on cluster xxx_cluster ;

# 只读
CREATE ROLE ro on cluster xxx_cluster ;
GRANT SELECT ON *.* TO ro on cluster xxx_cluster ;

# 读写
CREATE ROLE rw on cluster xxx_cluster ;
GRANT SELECT, INSERT, ALTER UPDATE, ALTER DELETE ON *.* TO rw on cluster xxx_cluster ;

创建用户、指定角色

CREATE user user host any IDENTIFIED WITH sha256_password BY 'passwd' on cluster xxx_cluster ;

grant ro to user on cluster xxx_cluster ;

修改用户密码、权限

ALTER user xxx IDENTIFIED WITH sha256_password BY 'xxx' on cluster xxx_cluster;

删除用户、角色

DROP USER [IF EXISTS] name [,...] [ON CLUSTER cluster_name]

DROP ROLE [IF EXISTS] name [,...] [ON CLUSTER cluster_name]

查看授权信息

**SHOW** GRANTS ；

修改授权

GRANT [ON CLUSTER cluster_name] privilege[(column_name [,...])] [,...] ON {db.table|db.*|*.*|table|*} TO {user | role | CURRENT_USER} [,...] [WITH GRANT OPTION] [WITH REPLACE OPTION]

• privilege — 权限类型
• role — 用户角色
• user — 用户账号

WITH GRANT OPTION 授予 user 或 role执行 GRANT 操作的权限，用户可将在自身权限范围内的权限进行授权 ；

WITH REPLACE OPTION 以当前sql里的新权限替代掉 user 或 role的旧权限，如果没有该选项则是追加授权。

SQL

https://clickhouse.com/docs/zh/sql-reference/statements/

数据查询

https://www.bilibili.com/video/BV1xg411w7AP?p=9&vd_source=af2ea4a917aaa7140d659b5b32e97e86

查询优化

https://www.bilibili.com/video/BV1xg411w7AP?p=13&vd_source=af2ea4a917aaa7140d659b5b32e97e86

创建分布式表

创建库表，三台主机一起创建

create database test;

use test;

CREATE TABLE logs (cur Date, size Int32, message String) ENGINE = MergeTree(cur, message, 8192);

CREATE TABLE logs_dist AS logs ENGINE = Distributed(xxx_clusters,test,logs,rand());

在第一个节点插入数据，若干次

insert into logs_dist values(now(), 1, '1');

查看结果

略。

创建集群复制表

• 测试语句

CREATE database test3 ON CLUSTER xxx_cluster;

create table test3.T03 ON CLUSTER xxx_cluster
(
    ts  DateTime,
    uid String,
    biz String
)
    engine = ReplicatedMergeTree('/clickhouse/tables/{shard}/T03', '{replica}')
        PARTITION BY toYYYYMMDD(ts)
        ORDER BY ts
        SETTINGS index_granularity = 8192
;

insert into test3.T03 values
('2020-06-01 12:00:00','100','aa'),
('2020-06-01 12:00:00','101','bb');

insert into test3.T03 values
('2020-06-01 12:00:00','102','cc'),
('2020-06-01 12:00:00','103','dd');

insert into test3.T03 values
('2020-06-01 12:00:00','104','aa'),
('2020-06-01 12:00:00','105','bb');

insert into test3.T03 values
('2020-06-01 12:00:00','106','aa'),
('2020-06-01 12:00:00','107','bb');

insert into test3.T03 values
('2020-06-01 12:00:00','108','aa'),
('2020-06-01 12:00:00','109','bb');

SELECT * from test3.T03 ; 

• 测试结果

测试通过。

ON CLUSTER 语法

• 查看集群信息

SELECT * FROM system.clusters

• 执行集群创建数据库命令

create database test2 on cluster xxx_cluster ;

• 确认结果
  
  测试通过。

​

常见问题

参考链接

数据一致性问题

尽量避免使用分布式表。

在业务空闲的时候手工发起计划外的合并。

从业务流程上去保证一致性。

多副本表，尽量固定写入的节点

如果一个节点数据丢失，只需要将副本节点的 data/ metadata/ 目录拷过来覆盖。

Zookeeper数据丢失导致副本表无法启动

附录

阿里云提供的CH常见问题及解决

• 启动日志

root@test-server /etc/clickhouse-server# sudo -u clickhouse clickhouse-server --config-file=/etc/clickhouse-server/config.xml
Processing configuration file '/etc/clickhouse-server/config.xml'.
Couldn't save preprocessed config to /var/lib/clickhouse/preprocessed_configs/config.xml: Access to file denied: /var/lib/clickhouse/preprocessed_configs/config.xml
Logging trace to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Logging trace to console
2022.06.17 04:37:07.960312 [ 11720 ] {} <Information> SentryWriter: Sending crash reports is disabled
2022.06.17 04:37:08.003464 [ 11720 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2022.06.17 04:37:08.415141 [ 11720 ] {} <Information> : Starting ClickHouse 22.2.2.1 with revision 54459, build id: 5F3D9E4F48D4CC47, PID 11720
2022.06.17 04:37:08.415518 [ 11720 ] {} <Information> Application: starting up
2022.06.17 04:37:08.415622 [ 11720 ] {} <Information> Application: OS name: Linux, version: 3.10.0-1160.66.1.el7.x86_64, architecture: x86_64
2022.06.17 04:37:08.460887 [ 11720 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/thermal
2022.06.17 04:37:08.461152 [ 11720 ] {} <Trace> AsynchronousMetrics: Scanning /sys/block
2022.06.17 04:37:08.462070 [ 11720 ] {} <Trace> AsynchronousMetrics: Scanning /sys/devices/system/edac
2022.06.17 04:37:08.462255 [ 11720 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/hwmon
2022.06.17 04:37:09.276347 [ 11720 ] {} <Warning> Application: Calculated checksum of the binary: 03A21B8EF25D04A4DCD8C0FCA8310FDA. There is no information about the reference checksum.
2022.06.17 04:37:09.276564 [ 11720 ] {} <Information> Application: It looks like the process has no CAP_IPC_LOCK capability, binary mlock will be disabled. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_ipc_lock=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
2022.06.17 04:37:09.278976 [ 11720 ] {} <Debug> Application: rlimit on number of file descriptors is 262144
2022.06.17 04:37:09.279089 [ 11720 ] {} <Debug> Application: Initializing DateLUT.
2022.06.17 04:37:09.279128 [ 11720 ] {} <Trace> Application: Initialized DateLUT with time zone 'America/New_York'.
2022.06.17 04:37:09.279201 [ 11720 ] {} <Debug> Application: Setting up /var/lib/clickhouse/tmp/ to store temporary data in it
2022.06.17 04:37:09.296585 [ 11720 ] {} <Debug> Application: Configuration parameter 'interserver_http_host' doesn't exist or exists and empty. Will use 'test-server' as replica host.
2022.06.17 04:37:09.296651 [ 11720 ] {} <Debug> Application: Initiailizing interserver credentials.
2022.06.17 04:37:09.296915 [ 11720 ] {} <Information> SensitiveDataMaskerConfigRead: 1 query masking rules loaded.
2022.06.17 04:37:09.301771 [ 11720 ] {} <Debug> ConfigReloader: Loading config '/etc/clickhouse-server/config.xml'
Processing configuration file '/etc/clickhouse-server/config.xml'.
Couldn't save preprocessed config to /var/lib/clickhouse/preprocessed_configs/config.xml: Access to file denied: /var/lib/clickhouse/preprocessed_configs/config.xml
2022.06.17 04:37:09.303671 [ 11720 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/config.xml', performing update on configuration
2022.06.17 04:37:09.305894 [ 11720 ] {} <Information> Application: Setting max_server_memory_usage was set to 8.65 GiB (9.61 GiB available * 0.90 max_server_memory_usage_to_ram_ratio)
2022.06.17 04:37:09.310332 [ 11720 ] {} <Error> CertificateReloader: Cannot obtain modification time for certificate file /etc/clickhouse-server/server.crt, skipping update. errno: 2, strerror: No such file or directory
2022.06.17 04:37:09.310427 [ 11720 ] {} <Error> CertificateReloader: Cannot obtain modification time for key file /etc/clickhouse-server/server.key, skipping update. errno: 2, strerror: No such file or directory
2022.06.17 04:37:09.310503 [ 11720 ] {} <Debug> CertificateReloader: Initializing certificate reloader.
2022.06.17 04:37:09.311737 [ 11720 ] {} <Error> CertificateReloader: Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-server/server.key: error:02000002:system library:OPENSSL_internal:No such file or directory (version 22.2.2.1)
2022.06.17 04:37:09.311810 [ 11720 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/config.xml', performed update on configuration
2022.06.17 04:37:09.313739 [ 11720 ] {} <Debug> ConfigReloader: Loading config '/etc/clickhouse-server/users.xml'
Processing configuration file '/etc/clickhouse-server/users.xml'.
Couldn't save preprocessed config to /var/lib/clickhouse/preprocessed_configs/users.xml: Access to file denied: /var/lib/clickhouse/preprocessed_configs/users.xml
2022.06.17 04:37:09.314591 [ 11720 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/users.xml', performing update on configuration
2022.06.17 04:37:09.315865 [ 11720 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/users.xml', performed update on configuration
2022.06.17 04:37:09.316186 [ 11720 ] {} <Debug> Access(user directories): Added users.xml access storage 'users.xml', path: /etc/clickhouse-server/users.xml
2022.06.17 04:37:09.316586 [ 11720 ] {} <Warning> Access(local directory): File /var/lib/clickhouse/access/users.list doesn't exist
2022.06.17 04:37:09.316633 [ 11720 ] {} <Warning> Access(local directory): Recovering lists in directory /var/lib/clickhouse/access/
2022.06.17 04:37:09.317520 [ 11720 ] {} <Debug> Access(user directories): Added local directory access storage 'local directory', path: /var/lib/clickhouse/access/
2022.06.17 04:37:09.317586 [ 11720 ] {} <Information> Application: Uncompressed cache size was lowered to 4.80 GiB because the system has low amount of memory
2022.06.17 04:37:09.320784 [ 11720 ] {} <Information> Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32
2022.06.17 04:37:09.325819 [ 11720 ] {} <Information> Context: Initialized background executor for move operations with num_threads=8, num_tasks=8
2022.06.17 04:37:09.327569 [ 11720 ] {} <Information> Context: Initialized background executor for fetches with num_threads=8, num_tasks=8
2022.06.17 04:37:09.330919 [ 11720 ] {} <Information> Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8
2022.06.17 04:37:09.331521 [ 11720 ] {} <Information> Application: Mark cache size was lowered to 4.80 GiB because the system has low amount of memory
2022.06.17 04:37:09.332031 [ 11720 ] {} <Information> Application: Loading user defined objects from /var/lib/clickhouse/
2022.06.17 04:37:09.332399 [ 11720 ] {} <Debug> UserDefinedSQLObjectsLoader: loading user defined objects
2022.06.17 04:37:09.332616 [ 11720 ] {} <Debug> Application: Loaded user defined objects
2022.06.17 04:37:09.332659 [ 11720 ] {} <Information> Application: Loading metadata from /var/lib/clickhouse/
2022.06.17 04:37:09.338597 [ 11720 ] {} <Information> DatabaseAtomic (system): Metadata processed, database system has 0 tables and 0 dictionaries in total.
2022.06.17 04:37:09.338652 [ 11720 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 0.000178821 sec
2022.06.17 04:37:09.338693 [ 11720 ] {} <Information> TablesLoader: Loading 0 tables with 0 dependency level
2022.06.17 04:37:09.342130 [ 11720 ] {} <Debug> SystemLog: Not creating system.text_log since corresponding section 'text_log' is missing from config
2022.06.17 04:37:09.345511 [ 11720 ] {} <Debug> SystemLog: Not creating system.zookeeper_log since corresponding section 'zookeeper_log' is missing from config
2022.06.17 04:37:09.367914 [ 11720 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal.
2022.06.17 04:37:09.369863 [ 11720 ] {} <Information> DatabaseAtomic (default): Metadata processed, database default has 0 tables and 0 dictionaries in total.
2022.06.17 04:37:09.369918 [ 11720 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 0.000111427 sec
2022.06.17 04:37:09.369950 [ 11720 ] {} <Information> TablesLoader: Loading 0 tables with 0 dependency level
2022.06.17 04:37:09.369979 [ 11720 ] {} <Information> DatabaseAtomic (default): Starting up tables.
2022.06.17 04:37:09.370009 [ 11720 ] {} <Information> DatabaseAtomic (system): Starting up tables.
2022.06.17 04:37:09.373931 [ 11720 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads
2022.06.17 04:37:09.405561 [ 11720 ] {} <Debug> Application: Loaded metadata.
2022.06.17 04:37:09.405830 [ 11720 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2022.06.17 04:37:09.406520 [ 11720 ] {} <Information> Application: It looks like this system does not have procfs mounted at /proc location, neither clickhouse-server process has CAP_NET_ADMIN capability. 'taskstats' performance statistics will be disabled. It could happen due to incorrect ClickHouse package installation. You can try to resolve the problem manually with 'sudo setcap cap_net_admin=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems. It also doesn't work if you run clickhouse-server inside network namespace as it happens in some containers.
2022.06.17 04:37:09.406624 [ 11720 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_priority' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
2022.06.17 04:37:09.425247 [ 11720 ] {} <Trace> MySQLHandlerFactory: Failed to create SSL context. SSL will be disabled. Error: Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-server/server.key: error:02000002:system library:OPENSSL_internal:No such file or directory (version 22.2.2.1)
2022.06.17 04:37:09.425626 [ 11720 ] {} <Trace> MySQLHandlerFactory: Failed to read RSA key pair from server certificate. Error: Code: 76. DB::Exception: Cannot open certificate file: /etc/clickhouse-server/server.crt. (CANNOT_OPEN_FILE) (version 22.2.2.1)
2022.06.17 04:37:09.425670 [ 11720 ] {} <Trace> MySQLHandlerFactory: Generating new RSA key pair.
2022.06.17 04:37:09.509628 [ 11720 ] {} <Trace> MySQLHandlerFactory: Failed to create SSL context. SSL will be disabled. Error: Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-server/server.key: error:02000002:system library:OPENSSL_internal:No such file or directory (version 22.2.2.1)
2022.06.17 04:37:09.509888 [ 11720 ] {} <Trace> MySQLHandlerFactory: Failed to read RSA key pair from server certificate. Error: Code: 76. DB::Exception: Cannot open certificate file: /etc/clickhouse-server/server.crt. (CANNOT_OPEN_FILE) (version 22.2.2.1)
2022.06.17 04:37:09.509922 [ 11720 ] {} <Trace> MySQLHandlerFactory: Generating new RSA key pair.
2022.06.17 04:37:09.679469 [ 11720 ] {} <Error> CertificateReloader: Cannot obtain modification time for certificate file /etc/clickhouse-server/server.crt, skipping update. errno: 2, strerror: No such file or directory
2022.06.17 04:37:09.679546 [ 11720 ] {} <Error> CertificateReloader: Cannot obtain modification time for key file /etc/clickhouse-server/server.key, skipping update. errno: 2, strerror: No such file or directory
2022.06.17 04:37:09.679587 [ 11720 ] {} <Debug> CertificateReloader: Initializing certificate reloader.
2022.06.17 04:37:09.679924 [ 11720 ] {} <Error> CertificateReloader: Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-server/server.key: error:02000002:system library:OPENSSL_internal:No such file or directory (version 22.2.2.1)
2022.06.17 04:37:09.680028 [ 11720 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 11.24 MiB, peak 15.83 MiB, will set to 472.38 MiB (RSS), difference: 461.14 MiB
2022.06.17 04:37:09.682820 [ 11720 ] {} <Information> DNSCacheUpdater: Update period 15 seconds
2022.06.17 04:37:09.682955 [ 11720 ] {} <Information> Application: Available RAM: 9.61 GiB; physical cores: 6; logical cores: 6.
2022.06.17 04:37:09.683073 [ 11783 ] {} <Debug> DNSResolver: Updating DNS cache
2022.06.17 04:37:09.683827 [ 11783 ] {} <Debug> DNSResolver: Updated DNS cache
2022.06.17 04:37:09.683972 [ 11720 ] {} <Information> Application: Listening for http://[::1]:8123
2022.06.17 04:37:09.684252 [ 11720 ] {} <Information> Application: Listening for native protocol (tcp): [::1]:9000
2022.06.17 04:37:09.684513 [ 11720 ] {} <Information> Application: Listening for replica communication (interserver): http://[::1]:9009
2022.06.17 04:37:09.684721 [ 11720 ] {} <Information> Application: Listening for MySQL compatibility protocol: [::1]:9004
2022.06.17 04:37:09.684985 [ 11720 ] {} <Information> Application: Listening for PostgreSQL compatibility protocol: [::1]:9005
2022.06.17 04:37:09.685303 [ 11720 ] {} <Information> Application: Listening for http://127.0.0.1:8123
2022.06.17 04:37:09.685569 [ 11720 ] {} <Information> Application: Listening for native protocol (tcp): 127.0.0.1:9000
2022.06.17 04:37:09.685759 [ 11720 ] {} <Information> Application: Listening for replica communication (interserver): http://127.0.0.1:9009
2022.06.17 04:37:09.685983 [ 11720 ] {} <Information> Application: Listening for MySQL compatibility protocol: 127.0.0.1:9004
2022.06.17 04:37:09.686250 [ 11720 ] {} <Information> Application: Listening for PostgreSQL compatibility protocol: 127.0.0.1:9005
2022.06.17 04:37:09.686311 [ 11720 ] {} <Information> Application: Ready for connections.