Flink 配置文件 对于管理员来说,差不多经常调整的就只有 conf 下的flink-conf.yaml : 经过初步的调整,大约有以下模块的参数(未优化)

Licensed to the Apache Software Foundation (ASF) under one

or more contributor license agreements. See the NOTICE file

distributed with this work for additional information

regarding copyright ownership. The ASF licenses this file

to you under the Apache License, Version 2.0 (the

"License"); you may not use this file except in compliance

with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

==============================================================================

Jobmanager && TaskManager 的基本参数

==============================================================================

jobmanager.rpc.address: flink-master

jobmanager.rpc.port: 6123
jobmanager.heap.size: 32768m
jobstore.cache-size: 52428800
taskmanager.heap.size: 92160m
taskmanager.numberOfTaskSlots: 30
parallelism.default: 3
io.tmp.dirs: /data1/data/flink/local
taskmanager.network.numberOfBuffers: 4096
taskmanager.memory.fraction: 0.7
taskmanager.memory.preallocate: false
taskmanager.debug.memory.startLogThread: 8
taskmanager.debug.memory.logIntervalMs: 3600000
blob.fetch.retries: 50
blob.fetch.num-concurrent: 200
blob.fetch.backlog: 500
task.cancellation-interval: 30000
taskmanager.exit-on-fatal-akka-error: false
taskkmanager.log.path:

==============================================================================

High Availability && and checkpointing

=============================================================================

high-availability: ZOOKEEPER
high-availability.cluster-id: /default_flink_dc
high-availability.zookeeper.quorum: VECS01455:2181,VECS01456:2181,VECS01457:2181,VECS01458:2181,VECS01459:2181
high-availability.storageDir: hdfs://flashHadoop/flink/recovery
high-availability.zookeeper.path.root: /flink
high-availability.zookeeper.path.checkpoint-counter: /checkpoint-counter
high-availability.zookeeper.path.checkpoints: /checkpoints
high-availability.zookeeper.path.jobgraphs: /jobgraphs
high-availability.zookeeper.path.latch: /leaderlatch
high-availability.zookeeper.path.leader: /leader
high-availability.zookeeper.path.mesos-workers: /mesos-workers
high-availability.zookeeper.path.running-registry: /running_job_registry/
high-availability.zookeeper.client.connection-timeout: 18000
high-availability.zookeeper.client.max-retry-attempts: 10
high-availability.zookeeper.client.retry-wait: 5000
high-availability.zookeeper.client.session-timeout: 60000

==============================================================================

HDFS and Fault tolerance and checkpointing

==============================================================================

fs.default-scheme: hdfs://flashHadoop/flink/data
fs.output.always-create-directory: false
fs.overwrite-files: false

state.backend: filesystem
state.backend.async: true
state.backend.fs.memory-threshold: 4096
state.backend.incremental: false
state.backend.local-recovery: false
state.checkpoints.dir: hdfs://flashHadoop/flink/checkpoints_data
state.checkpoints.num-retained: 1
state.savepoints.dir: hdfs://flashHadoop/flink/savepoints_data

==============================================================================

Web Frontend

==============================================================================

web.address: 0.0.0.0
web.access-control-allow-origin: "*"
web.backpressure.cleanup-interval: 600000
web.backpressure.delay-between-samples: 50
web.backpressure.num-samples: 100
web.backpressure.refresh-interval: 60000
web.checkpoints.history: 20
web.history: 20
web.log.path: /data1/data/flink/web/log
web.refresh-interval: 3000
web.submit.enable: true
web.timeout: 10000
web.tmpdir: /data1/data/flink/web/tmp

==============================================================================

Flink Cluster Security Configuration

==============================================================================

test is not set Security

==============================================================================

HistoryServer Config

==============================================================================

historyserver.web.address: 0.0.0.0
historyserver.web.port: 8082
historyserver.archive.fs.dir: hdfs://flashHadoop/flink/completed_jobs/
historyserver.archive.fs.refresh-interval: 2000
historyserver.web.tmpdir: /data1/data/flink/tmp

==============================================================================

==============================================================================

Environment 角色环境 申明

==============================================================================

env.java.opts: -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -XX:+G1SummarizeConcMark -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX:+ParallelRefProcEnabled -XX:ParallelGCThreads=24 -XX:ConcGCThreads=16 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1MixedGCLiveThresholdPercent=85

env.java.opts.historyserver: -Xmx16g -Xms16g -Xmn4g -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -XX:+G1SummarizeConcMark -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:ParallelGCThreads=24 -XX:ConcGCThreads=16 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1MixedGCLiveThresholdPercent=85 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/data1/data/flink/gc/gc-historyserver-hostname.log

env.java.opts.jobmanager: -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -XX:+G1SummarizeConcMark -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX:+ParallelRefProcEnabled -XX:ParallelGCThreads=24 -XX:ConcGCThreads=16 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1MixedGCLiveThresholdPercent=85 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/data1/data/flink/gc/gc-jobmanager-hostname.log

env.java.opts.taskmanager: -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -XX:+G1SummarizeConcMark -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX:+ParallelRefProcEnabled -XX:ParallelGCThreads=24 -XX:ConcGCThreads=16 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1MixedGCLiveThresholdPercent=85 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/data1/data/flink/gc/gc-taskmanager-hostname.log

env.log.dir: /data1/data/flink/log

env.log.max: 1

flink-conf.yaml的更多相关文章

  1. flink部署操作-flink standalone集群安装部署

    flink集群安装部署 standalone集群模式 必须依赖 必须的软件 JAVA_HOME配置 flink安装 配置flink 启动flink 添加Jobmanager/taskmanager 实 ...

  2. flink Standalone Cluster

    Requirements Software Requirements Flink runs on all UNIX-like environments, e.g. Linux, Mac OS X, a ...

  3. Flink集群部署

    部署方式 一般来讲有三种方式: Local Standalone Flink On Yarn/Mesos/K8s… 单机模式 参考上一篇Flink从入门到放弃(入门篇2)-本地环境搭建&构建第 ...

  4. Flink(二)CentOS7.5搭建Flink1.6.1分布式集群

    一. Flink的下载 安装包下载地址:http://flink.apache.org/downloads.html  ,选择对应Hadoop的Flink版本下载 [admin@node21 soft ...

  5. Flink的安装配置

    一. Flink的下载 安装包下载地址:http://flink.apache.org/downloads.html  ,选择对应Hadoop的Flink版本下载 [admin@node21 soft ...

  6. 一张图轻松掌握 Flink on YARN 应用启动全流程(上)

    Flink 支持 Standalone 独立部署和 YARN.Kubernetes.Mesos 等集群部署模式,其中 YARN 集群部署模式在国内的应用越来越广泛.Flink 社区将推出 Flink ...

  7. kerberos系列之flink认证配置

    大数据安全系列的其它文章 https://www.cnblogs.com/bainianminguo/p/12548076.html-----------安装kerberos https://www. ...

  8. 分享一个Flink checkpoint失败的问题和解决办法

    本文来自: PerfMa技术社区 PerfMa(笨马网络)官网 接触Flink一段时间了,遇到了一些问题,其中有一个checkpoint失败导致作业重启的问题,遇到了很多次,重启之后一般也能恢复正常, ...

  9. flink集群模式安装配置

    一.手动下载安装包 wget http://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27 ...

  10. Linux使用docker安装flink

    配置文件不是原始的 修改过,可以根据自己的配置文件来 jobmanager配置 flink-conf.yaml ############################################ ...

随机推荐

  1. hbase snapshot 表备份/恢复

    snapshot其实就是一组metadata信息的集合,它可以让管理员将表恢复到以前的一个状态.snapshot并不是一份拷贝,它只是一个文件名的列表,并不拷贝数据.一个全的snapshot恢复以为着 ...

  2. 从0开始的Python学习019更多的Python内容2

    书接上文,接演Python全传 话说学了这么多Python的基础知识,也该写一点让别人看不懂的代码了. lambda lambda表达式,是一个方法的简化形似,它没有自己的代码块,它后面的语句就是它的 ...

  3. Linux学习历程——Centos 7 chmod命令

    一.命令介绍 chmod 命令,是Linux管理员最常用的命令之一,用于修改文件或目录的访问权限. Linux系统中,每一个文件都有文件所有者和所属群组,并且规定文件的所有者,所属群组,以及其他人队问 ...

  4. Gitlab源码库里代码提交后,如何触发jenkins自动构建?

    版本库里代码提交后,如何触发jenkins自动构建?这是一个面试题,感觉自己回答的并不好,因为并没有用过这个功能,之前公司实际项目用的是svn版本管理,一般都用立刻构建,和定时任务构建(不管代码是否有 ...

  5. ES6的Map如何遍历

    projectMap.forEach(function (value, key, map) { //value和key就是map的key,value,map是map本身 });

  6. 蚂蚁 uva 10881

    思路: 一,两个点相撞然后,相互弹开.这样其实和两个点穿过去差不多, 但是对于单个点而言,这样肯定不行. 二,当你发现了不管什么情况,这些点的相对位置是不变的, 比如 1, 4, 3, 2 不管怎么移 ...

  7. [认证授权] 4.OIDC(OpenId Connect)身份认证(核心部分)

    1 什么是OIDC? 看一下官方的介绍(http://openid.net/connect/): OpenID Connect 1.0 is a simple identity layer on to ...

  8. 使用FreeHttp强制登出微信公众号登陆状态(实现~原理)

    概述 我们使用的部分网站设计成一旦登录即不允许用户手动退出,现实场景中是没有问题的 但如果是在测试或调试过程中就会有强制登出的需求 如果当前使用的是PC浏览器,您或许可以通过调试模式清除保持登录信息的 ...

  9. maven导入外部jar包的方法

    maven的导入外部jar包的方法(前提还有外部jar包) 1.问题: 在maven添加依赖包时,添加不成功,可以选择通过外部命令导入到仓库里. 2.命令:(在Dos命令行) mvn install: ...

  10. 解决hash冲突的方法

    复制粘贴于:https://www.cnblogs.com/wuchaodzxx/p/7396599.html#H1_2 开放地址法(线性探测法.二次探测.伪随机探测) 再哈希法 链地址法 建立公共溢 ...