Splunk Enterprise architecture and processes

This topic discusses the internal architecture and processes of Splunk Enterprise at a high level. If you're looking for information about third-party components used in Splunk Enterprise, see the credits section in the Release notes.

Splunk Enterprise Processes

A Splunk Enterprise server installs a process on your host, splunkd.

splunkd is a distributed C/C++ server that accesses, processes and indexes streaming IT data. It also handles search requests. splunkd processes and indexes your data by streaming it through a series of pipelines, each made up of a series of processors.

  • Pipelines are single threads inside the splunkd process, each configured with a single snippet of XML.
  • Processors are individual, reusable C or C++ functions that act on the stream of IT data that passes through a pipeline. Pipelines can pass data to one another through queues.

Architecture diagram

注意:负载均衡,副本!

Splunk Architecture

A Bit About Architecture

Splunk is a high performance, scalable software server written in C/C++ and Python. It indexes and searches logs and other IT data in real time. Splunk works with data generated by any application, server or device. The Splunk Developer API is accessible via REST, SOAP or the command line. After downloading, installing and starting Splunk, you'll find two Splunk Server processes running on your host, splunkd and splunkweb.

    • splunkd is a distributed C/C++ server that accesses, processes and indexes streaming IT data and also handles search requests. splunkd processes and indexes your data by streaming it through a series of pipelines, each made up of a series of processors. Pipelines are single threads inside the splunkd process, each configured with a single snippet of XML. Processors are individual, reusable C/C++ or Python functions that act on the stream of IT data passing through a pipeline. Pipelines can pass data to one another via queues. splunkd supports a command line interface for searching and viewing results.
  • splunkweb is a Python-based application server providing the Splunk Web user interface. It allows users to search and navigate IT data stored by Splunk servers and to manage your Splunk deployment through the browser interface. splunkweb communicates with your web browser via REST and communicates with splunkd via SOAP.

    • Splunk's Data Store manages the original raw data in compressed format as well as the indexes into the data. Data can be deleted or archived based on retention period or maximum data store size.
    • Splunk Servers can communicate with one another via Splunk-2-Splunk, a TCP-based protocol, to forward data from one server to another and to distribute searches across multiple servers.
    • Bundles are files that contain configuration settings including, user accounts, Splunks, Live Splunks, Data Inputs and Processing Properties to easily create specific Splunk environments.
  • Modules are files that add new functionality to Splunk by adding to or modifying existing processors and pipelines.

About forwarding and receiving

You can forward data from one Splunk instance to another Splunk server or even to a non-Splunk system. The Splunk instance that performs theforwarding is typically a smaller footprint version of Splunk, called a forwarder.

A Splunk instance that receives data from one or more forwarders is called a receiver. The receiver is usually a Splunk indexer, but can also be another forwarder, as described here.

This diagram shows three forwarders sending data to a single Splunk receiver (an indexer), which then indexes the data and makes it available for searching:

Splunk Enterprise architecture——转发器本质上是日志收集client附加负载均衡,indexer是分布式索引,外加一个集中式管理协调的中心节点的更多相关文章

  1. Jexus是一款Linux平台上的高性能WEB服务器和负载均衡网关

    什么是Jexus Jexus是一款Linux平台上的高性能WEB服务器和负载均衡网关,以支持ASP.NET.ASP.NET CORE.PHP为特色,同时具备反向代理.入侵检测等重要功能.可以这样说,J ...

  2. 动态DNS——本质上是IP变化,将任意变换的IP地址绑定给一个固定的二级域名。不管这个线路的IP地址怎样变化,因特网用户还是可以使用这个固定的域名 这样看的话,p2p可以用哇

    动态域名是因应网络远程访问的需要而产生的一项应用技术.因为没有固定IP,只能运用二级域名来应对经常变化的IP,动态域名的由来因此而产生. 它当前主要应用在:路由器.网络摄像机.带网络监控的硬盘录像机. ...

  3. Linux下Rsyslog日志远程集中式管理

    Rsyslog简介 Rsyslog的全称是 rocket-fast system for log,它提供了高性能,高安全功能和模块化设计.rsyslog能够接受从各种各样的来源,将其输入,输出的结果到 ...

  4. Dubbo入门到精通学习笔记(二十):MyCat在MySQL主从复制的基础上实现读写分离、MyCat 集群部署(HAProxy + MyCat)、MyCat 高可用负载均衡集群Keepalived

    文章目录 MyCat在MySQL主从复制的基础上实现读写分离 一.环境 二.依赖课程 三.MyCat 介绍 ( MyCat 官网:http://mycat.org.cn/ ) 四.MyCat 的安装 ...

  5. Centos7搭建集中式日志系统

    在CentOS7中,Rsyslong是一个集中式的日志收集系统,可以运行在TCP或者UDP的514端口上.   目录 开始之前 配置接收日志的主机 配置发送日志的主机 日志回滚 附件:创建日志接收模板 ...

  6. flume集群日志收集

    一.Flume简介 Flume是一个分布式的.高可用的海量日志收集.聚合和传输日志收集系统,支持在日志系统中定制各类数据发送方(如:Kafka,HDFS等),便于收集数据.其核心为agent,agen ...

  7. ELK日志收集平台部署

    需求背景 由于公司的后台服务有三台,每当后台服务运行异常,需要看日志排查错误的时候,都必须开启3个ssh窗口进行查看,研发们觉得很不方便,于是便有了统一日志收集与查看的需求. 这里,我用ELK集群,通 ...

  8. 理解OpenShift(6):集中式日志处理

    理解OpenShift(1):网络之 Router 和 Route 理解OpenShift(2):网络之 DNS(域名服务) 理解OpenShift(3):网络之 SDN 理解OpenShift(4) ...

  9. SpringBoot+kafka+ELK分布式日志收集

    一.背景 随着业务复杂度的提升以及微服务的兴起,传统单一项目会被按照业务规则进行垂直拆分,另外为了防止单点故障我们也会将重要的服务模块进行集群部署,通过负载均衡进行服务的调用.那么随着节点的增多,各个 ...

随机推荐

  1. P3627 [APIO2009]抢掠计划

    P3627 [APIO2009]抢掠计划 Tarjan缩点+最短(最长)路 显然的缩点...... 在缩点时,顺便维护每个强连通分量的总权值 缩完点按照惯例建个新图 然后跑一遍spfa最长路,枚举每个 ...

  2. 编写第一个微信小程序界面

    编写第一个微信小程序界面 不忘初心,方得始终:初心易得,始终难守. 传统的 web 结构 小程序文件目录结构 小程序页面层级结构 编写第一个小程序 1. 创建小程序目录结构 2. 编写代码 welco ...

  3. PyCharm/IDEA 使用技巧总结

    基本概念 IDEA 没有类似 Eclipse 的工作空间的概念(workspace),最大单元就是 Project.这里可以把 Project 理解为 Eclipse 中的 workspace.Mod ...

  4. POJ 2425 A Chess Game(有向图SG函数)题解

    题意:给一个有向图,然后个m颗石头放在图上的几个点上,每次只能移动一步,如果不能移动者败 思路:dfs打表sg函数,然后求异或和 代码: #include<queue> #include& ...

  5. 【修改帐号信息】Eclipse中修改SVN用户名和密码方法

    由于在svn 的界面中并没有为我们提供直接更换用户名密码的地方,所以一旦我们需要更换用户名的就需要自己想一些办法. 解决方案: 在Eclipse 使用SVN 的过程中大多数人往往习惯把访问SVN 的用 ...

  6. 51nod 1137 矩阵乘法

    基本的矩阵乘法 中间for(int j=0;i<n;i++)  //这里写错了   应该是j<n 晚上果然  效率不行 等会早点儿睡 //矩阵乘法 就是 两个矩阵 第一个矩阵的列 等与 第 ...

  7. xshell5 Linux 上传下载文件

    1,先登录身份验证和文件传输ZMODEM 选择自动激活. 2,rpm -qa | grep lrzsz 利用此命令查看是否安装了lrzsz . 如果没有任何反应则是没有安装 若没有安装 yum ins ...

  8. Web前端代码规范

    新增:http://materliu.github.io/code-guide/#project-naming HTML 原则1.规范 .保证您的代码规范,保证结构表现行为相互分离.2.简洁.保证代码 ...

  9. UVa 10905 孩子们的游戏

    https://vjudge.net/problem/UVA-10905 题意: 给定n个正整数,把它们连接成一个最大的整数. 思路: 实在是没想到直接用string来排序了. #include< ...

  10. ubuntu 14.04 添加、删除用户,修改用户名称,修改主机名

    ubuntu添加用户 sudo adduser test ubuntu删除用户 sudo userdel test cd /home/ sudo rm -r test