[elk]elk的诸多beats&&kibana插件

elk的诸多beats

参考: https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html

jmxproxybeat

参考: https://github.com/radoondas/jmxproxybeat

Metricbeat

正确姿势启动metricbeat

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - filesystem
    - memory
    - network
    - process
  enabled: true
  period: 10s
  processes: ['.*']
  cpu_ticks: false

output.elasticsearch:
    hosts: ["http://192.168.x.x:9200"]

setup.template.name: "metricbeat"
setup.template.fields: "fields.yml"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1

setup.kibana.host: "192.168.x.x:5601"
setup.dashboards.enabled: true

./metricbeat -e -c metricbeat.yml -d "publish"

它有这些指标模块

This section contains detailed information about the metric collecting modules contained in Metricbeat. Each module contains one or multiple metricsets. More details about each module can be found under the links below.

Aerospike
Apache
Ceph
Couchbase
Docker
Dropwizard
Elasticsearch
Golang
HAProxy
HTTP
Jolokia
Kafka
Kibana
Kubernetes
Memcached
MongoDB
MySQL
Nginx
PHP-FPM
PostgreSQL
Prometheus
RabbitMQ
Redis
System
vSphere
Windows
ZooKeeper

amazonbeat

Reads data from a specified Amazon product.

apachebeat

Reads status from Apache HTTPD server-status.

apexbeat

Extracts configurable contextual data and metrics from Java applications via the APEX toolkit.

burrowbeat

Monitors Kafka consumer lag using Burrow.

cassandrabeat

Uses Cassandra’s nodetool cfstats utility to monitor Cassandra database nodes and lag.

cloudflarebeat

Indexes log entries from the Cloudflare Enterprise Log Share API.

cloudfrontbeat

Reads log events from Amazon Web Services CloudFront.

cloudtrailbeat

Reads events from Amazon Web Services' CloudTrail.

cloudwatchmetricbeat

A beat for Amazon Web Services' CloudWatch Metrics.

cloudwatchlogsbeat

Reads log events from Amazon Web Services' CloudWatch Logs.

collectbeat

Adds discovery on top of Filebeat and Metricbeat in environments like Kubernetes.

connbeat

Exposes metadata about TCP connections.

consulbeat

Reads services health checks from consul and pushes them to Elastic.

dockbeat

Reads Docker container statistics and indexes them in Elasticsearch.

elasticbeat

Reads status from an Elasticsearch cluster and indexes them in Elasticsearch.

etcdbeat

Reads stats from the Etcd v2 API and indexes them into Elasticsearch.

execbeat

Periodically executes shell commands and sends the standard output and standard error to Logstash or Elasticsearch.

factbeat

Collects facts from Facter.

flowbeat

Collects, parses, and indexes sflow samples.

gabeat

Collects data from Google Analytics Realtime API.

githubbeat

Easily monitors GitHub repository activity.

gpfsbeat

Collects GPFS metric and quota information.

hsbeat

Reads all performance counters in Java HotSpot VM.

httpbeat

Polls multiple HTTP(S) endpoints and sends the data to Logstash or Elasticsearch. Supports all HTTP methods and proxies.

hwsensorsbeat

Reads sensors information from OpenBSD.

icingabeat

Icingabeat ships events and states from Icinga 2 to Elasticsearch or Logstash.

iobeat

Reads IO stats from /proc/diskstats on Linux.

jmxproxybeat

Reads Tomcat JMX metrics exposed over JMX Proxy Servlet to HTTP.

journalbeat

Used for log shipping from systemd/journald based Linux systems.

kafkabeat

Reads data from Kafka topics.

krakenbeat

Collect information on each transaction on the Kraken crypto platform.

lmsensorsbeat

Collects data from lm-sensors (such as CPU temperatures, fan speeds, and voltages from i2c and smbus).

logstashbeat

Collects data from Logstash monitoring API (v5 onwards) and indexes them in Elasticsearch.

mcqbeat

Reads the status of queues from memcacheq.

mongobeat

Monitors MongoDB instances and can be configured to send multiple document types to Elasticsearch.

mqttbeat

Add messages from mqtt topics to Elasticsearch.

mysqlbeat

Run any query on MySQL and send results to Elasticsearch.

nagioscheckbeat

For Nagios checks and performance data.

nginxbeat

Reads status from Nginx.

nginxupstreambeat

Reads upstream status from nginx upstream module.

nvidiagpubeat

Uses nvidia-smi to grab metrics of NVIDIA GPUs.

openconfigbeat

Streams data from OpenConfig-enabled network devices

packagebeat

Collects information about system packages from package managers.

phpfpmbeat

Reads status from PHP-FPM.

pingbeat

Sends ICMP pings to a list of targets and stores the round trip time (RTT) in Elasticsearch.

prombeat

Indexes Prometheus metrics.

prometheusbeat

Send Prometheus metrics to Elasticsearch via the remote write feature.

protologbeat

Accepts structured and unstructured logs via UDP or TCP. Can also be used to receive syslog messages or GELF formatted messages. (To be used as a successor to udplogbeat)

redditbeat

Collects new Reddit Submissions of one or multiple Subreddits.

redisbeat

Used for Redis monitoring.

retsbeat

Collects counts of RETS resource/class records from Multiple Listing Service (MLS) servers.

rsbeat

Ships redis slow logs to elasticsearch and anlyze by Kibana.

saltbeat

Reads events from salt master event bus.

springbeat

Collects health and metrics data from Spring Boot applications running with the actuator module.

twitterbeat

Reads tweets for specified screen names.

udpbeat

Ships structured logs via UDP.

udplogbeat

Accept events via local UDP socket (in plain-text or JSON with ability to enforce schemas). Can also be used for applications only supporting syslog logging.

unifiedbeat

Reads records from Unified2 binary files generated by network intrusion detection software and indexes the records in Elasticsearch.

uwsgibeat

Reads stats from uWSGI.

varnishlogbeat

Reads log data from a Varnish instance and ships it to Elasticsearch.

varnishstatbeat

Reads stats data from a Varnish instance and ships it to Elasticsearch.

wmibeat

Uses WMI to grab your favorite, configurable Windows metrics.

kibana插件

kibana_shard_allocation

参考: https://github.com/asileon/kibana_shard_allocation

mathlion

Mathlion is an advanced math plugin for Kibana's Timelion

参考: https://github.com/fermiumlabs/mathlion

.es(*).math("a=source")  //the variable "a" now contains the elasticsearch query.
.nop().math("a")  //this row now equals the former elasticsearch query

.es(*).math("source") //return the .es(*) query
.es(*).math("source+5") // add 5 to the .es(*) query

.nop().math("a=a+2 ; a=a+3 ")  //adds 5 to a
.nop().math("a=a+2 ; a=a+3 ; a ")  //adds 5 to a and displays a+5

.es(*).math("a=source")  //this query is invisible and does not generate an axis
.es(*).math("a=source; a")  //this query does

.nop.math("sqrt(3^2 + 4^2)") //returns 5

//Calculate power comsumption based on measured current and stimated voltage (in Europe)
.nop().math("electricPower(v,i)=(v*i)")
.es(metric=avg:current).math(machineCurrent=source)
.nop().math("elascPower(230,machineCurrent)")

//plot the horizontal statistical mean and variance
.es(*).math("me=mean(source); va=var(source)")
.value(1).math(me*source)
.value(1).math("(me+sqrt(va))*source")
.value(1).math("(me-sqrt(va))*source")