logstash

logstash作为数据搜集器，主要分为三个部分：input->filter->output  作为pipeline的形式进行处理,支持复杂的操作，如发邮件等

input配置数据的输入和简单的数据转换

filter配置数据的提取，一般使用grok

output配置数据的输出和简单的数据转换

运行：logstash  -f /etc/logstash.conf

-f  指定配置文件

-e  只在控制台运行

具体的配置见官网

https://www.elastic.co/products/logstash

Centralize, Transform & Stash Your Data

input

Plugin	Description	Github repository
beats	Receives events from the Elastic Beats framework	logstash-input-beats
couchdb_changes	Streams events from CouchDB’s _changes URI	logstash-input-couchdb_changes
elasticsearch	Reads query results from an Elasticsearch cluster	logstash-input-elasticsearch
file	Streams events from files	logstash-input-file
gelf	Reads GELF-format messages from Graylog2 as events	logstash-input-gelf
generator	Generates random log events for test purposes	logstash-input-generator
graphite	Reads metrics from the graphite tool	logstash-input-graphite
heartbeat	Generates heartbeat events for testing	logstash-input-heartbeat
http	Receives events over HTTP or HTTPS	logstash-input-http
http_poller	Decodes the output of an HTTP API into events	logstash-input-http_poller
jdbc	Creates events from JDBC data	logstash-input-jdbc
kafka	Reads events from a Kafka topic	logstash-input-kafka
log4j	Reads events over a TCP socket from a Log4jSocketAppender object	logstash-input-log4j
lumberjack	Receives events using the Lumberjack protocl	logstash-input-lumberjack
rabbitmq	Pulls events from a RabbitMQ exchange	logstash-input-rabbitmq
redis	Reads events from a Redis instance	logstash-input-redis
s3	Streams events from files in a S3 bucket	logstash-input-s3
sqs	Pulls events from an Amazon Web Services Simple Queue Service queue	logstash-input-sqs
stdin	Reads events from standard input	logstash-input-stdin
syslog	Reads syslog messages as events	logstash-input-syslog
tcp	Reads events from a TCP socket	logstash-input-tcp
twitter	Reads events from the Twitter Streaming API	logstash-input-twitter
udp	Reads events over UDP	logstash-input-udp

Community supported plugins

These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.

Plugin	Description	Github repository
cloudwatch	Pulls events from the Amazon Web Services CloudWatch API	logstash-input-cloudwatch
drupal_dblog	Retrieves watchdog log events from Drupal installations with DBLog enabled	logstash-input-drupal_dblog
eventlog	Pulls events from the Windows Event Log	logstash-input-eventlog
exec	Captures the output of a shell command as an event	logstash-input-exec
ganglia	Reads Ganglia packets over UDP	logstash-input-ganglia
gemfire	Pushes events to a GemFire region	logstash-input-gemfire
github	Reads events from a GitHub webhook	logstash-input-github
heroku	Streams events from the logs of a Heroku app	logstash-input-heroku
imap	Reads mail from an IMAP server	logstash-input-imap
irc	Reads events from an IRC server	logstash-input-irc
jmx	Retrieves metrics from remote Java applications over JMX	logstash-input-jmx
kinesis	Receives events through an AWS Kinesis stream	logstash-input-kinesis
meetup	Captures the output of command line tools as an event	logstash-input-meetup
pipe	Streams events from a long-running command pipe	logstash-input-pipe
puppet_facter	Receives facts from a Puppet server	logstash-input-puppet_facter
rackspace	Receives events from a Rackspace Cloud Queue service	logstash-input-rackspace
relp	Receives RELP events over a TCP socket	logstash-input-relp
rss	Captures the output of command line tools as an event	logstash-input-rss
salesforce	Creates events based on a Salesforce SOQL query	logstash-input-salesforce
snmptrap	Creates events based on SNMP trap messages	logstash-input-snmptrap
sqlite	Creates events based on rows in an SQLite database	logstash-input-sqlite
stomp	Creates events received with the STOMP protocol	logstash-input-stomp
unix	Reads events over a UNIX socket	logstash-input-unix
varnishlog	Reads from the varnish cache shared memory log	logstash-input-varnishlog
websocket	Reads events from a websocket	logstash-input-websocket
wmi	Creates events based on the results of a WMI query	logstash-input-wmi
xmpp	Receives events over the XMPP/Jabber protocol	logstash-input-xmpp
zenoss	Reads Zenoss events from the fanout exchange	logstash-input-zenoss
zeromq	Reads events from a ZeroMQ SUB socket	logstash-input-zeromq

filter

Plugin	Description	Github repository
aggregate	Aggregates information from several events originating with a single task	logstash-filter-aggregate
anonymize	Replaces field values with a consistent hash	logstash-filter-anonymize
csv	Parses comma-separated value data into individual fields	logstash-filter-csv
date	Parses dates from fields to use as the Logstash timestamp for an event	logstash-filter-date
de_dot	Computationally expensive filter that removes dots from a field name	logstash-filter-de_dot
dissect	Extracts unstructured event data into fields using delimiters	logstash-filter-dissect
dns	Performs a standard or reverse DNS lookup	logstash-filter-dns
drop	Drops all events	logstash-filter-drop
fingerprint	Fingerprints fields by replacing values with a consistent hash	logstash-filter-fingerprint
geoip	Adds geographical information about an IP address	logstash-filter-geoip
grok	Parses unstructured event data into fields	logstash-filter-grok
json	Parses JSON events	logstash-filter-json
kv	Parses key-value pairs	logstash-filter-kv
multiline	Merges multiple lines into a single event	logstash-filter-multiline
mutate	Performs mutations on fields	logstash-filter-mutate
ruby	Executes arbitrary Ruby code	logstash-filter-ruby
sleep	Sleeps for a specified time span	logstash-filter-sleep
split	Splits multi-line messages into distinct events	logstash-filter-split
syslog_pri	Parses the PRI (priority) field of a syslog message	logstash-filter-syslog_pri
throttle	Throttles the number of events	logstash-filter-throttle
translate	Replaces field contents based on a hash or YAML file	logstash-filter-translate
urldecode	Decodes URL-encoded fields	logstash-filter-urldecode
useragent	Parses user agent strings into fields	logstash-filter-useragent
uuid	Adds a UUID to events	logstash-filter-uuid
xml	Parses XML into fields	logstash-filter-xml

Community supported plugins

These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.

Plugin	Description	Github repository
alter	Performs general alterations to fields that the mutate filter does not handle	logstash-filter-alter
cidr	Checks IP addresses against a list of network blocks	logstash-filter-cidr
cipher	Applies or removes a cipher to an event	logstash-filter-cipher
clone	Duplicates events	logstash-filter-clone
collate	Collates events by time or count	logstash-filter-collate
elapsed	Calculates the elapsed time between a pair of events	logstash-filter-elapsed
elasticsearch	Copies fields from previous log events in Elasticsearch to current events	logstash-filter-elasticsearch
environment	Stores environment variables as metadata sub-fields	logstash-filter-environment
extractnumbers	Extracts numbers from a string	logstash-filter-extractnumbers
i18n	Removes special characters from a field	logstash-filter-i18n
json_encode	Serializes a field to JSON	logstash-filter-json_encode
metaevent	Adds arbitrary fields to an event	logstash-filter-metaevent
metricize	Takes complex events containing a number of metrics and splits these up into multiple events, each holding a single metric	logstash-filter-metricize
metrics	Aggregates metrics	logstash-filter-metrics
oui	Parse OUI data from MAC addresses	logstash-filter-oui
prune	Prunes event data based on a list of fields to blacklist or whitelist	logstash-filter-prune
punct	Strips all non-punctuation content from a field	logstash-filter-punct
range	Checks that specified fields stay within given size or length limits	logstash-filter-range
tld	Replaces the contents of the default message field with whatever you specify in the configuration	logstash-filter-tld
yaml	Takes an existing field that contains YAML and expands it into an actual data structure within the Logstash event	logstash-filter-yaml
zeromq	Sends an event to ZeroMQ	logstash-filter-zeromq

output

Elastic supported plugins

These plugins are maintained and supported by Elastic.

Plugin	Description	Github repository
csv	Writes events to disk in a delimited format	logstash-output-csv
elasticsearch	Stores logs in Elasticsearch	logstash-output-elasticsearch
email	Sends email to a specified address when output is received	logstash-output-email
file	Writes events to files on disk	logstash-output-file
graphite	Writes metrics to Graphite	logstash-output-graphite
http	Sends events to a generic HTTP or HTTPS endpoint	logstash-output-http
kafka	Writes events to a Kafka topic	logstash-output-kafka
lumberjack	Sends events using the lumberjack protocol	logstash-output-lumberjack
rabbitmq	Pushes events to a RabbitMQ exchange	logstash-output-rabbitmq
redis	Sends events to a Redis queue using the RPUSHcommand	logstash-output-redis
s3	Sends Logstash events to the Amazon Simple Storage Service	logstash-output-s3
stdout	Prints events to the standard output	logstash-output-stdout
tcp	Writes events over a TCP socket	logstash-output-tcp
udp	Sends events over UDP	logstash-output-udp

Community supported plugins

These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.

Plugin	Description	Github repository
boundary	Sends annotations to Boundary based on Logstash events	logstash-output-boundary
circonus	Sends annotations to Circonus based on Logstash events	logstash-output-circonus
cloudwatch	Aggregates and sends metric data to AWS CloudWatch	logstash-output-cloudwatch
datadog	Sends events to DataDogHQ based on Logstash events	logstash-output-datadog
datadog_metrics	Sends metrics to DataDogHQ based on Logstash events	logstash-output-datadog_metrics
elasticsearch_java	Stores logs in Elasticsearch using the node andtransport protocols	logstash-output-elasticsearch_java
exec	Runs a command for a matching event	logstash-output-exec
ganglia	Writes metrics to Ganglia’s gmond	logstash-output-ganglia
gelf	Generates GELF formatted output for Graylog2	logstash-output-gelf
google_bigquery	Writes events to Google BigQuery	logstash-output-google_bigquery
google_cloud_storage	Writes events to Google Cloud Storage	logstash-output-google_cloud_storage
graphtastic	Sends metric data on Windows	logstash-output-graphtastic
hipchat	Writes events to HipChat	logstash-output-hipchat
influxdb	Writes metrics to InfluxDB	logstash-output-influxdb
irc	Writes events to IRC	logstash-output-irc
jira	Writes strutured JSON events to JIRA	logstash-output-jira
juggernaut	Pushes messages to the Juggernaut websockets server	logstash-output-juggernaut
librato	Sends metrics, annotations, and alerts to Librato based on Logstash events	logstash-output-librato
loggly	Ships logs to Loggly	logstash-output-loggly
metriccatcher	Writes metrics to MetricCatcher	logstash-output-metriccatcher
mongodb	Writes events to MongoDB	logstash-output-mongodb
nagios	Sends passive check results to Nagios	logstash-output-nagios
nagios_nsca	Sends passive check results to Nagios using the NSCA protocol	logstash-output-nagios_nsca
newrelic	Sends logstash events to New Relic Insights as custom events	logstash-output-newrelic
opentsdb	Writes metrics to OpenTSDB	logstash-output-opentsdb
pagerduty	Sends notifications based on preconfigured services and escalation policies	logstash-output-pagerduty
pipe	Pipes events to another program’s standard input	logstash-output-pipe
rackspace	Sends events to a Rackspace Cloud Queue service	logstash-output-rackspace
redmine	Creates tickets using the Redmine API	logstash-output-redmine
riak	Writes events to the Riak distributed key/value store	logstash-output-riak
riemann	Sends metrics to Riemann	logstash-output-riemann
sns	Sends events to Amazon’s Simple Notification Service	logstash-output-sns
solr_http	Stores and indexes logs in Solr	logstash-output-solr_http
sqs	Pushes events to an Amazon Web Services Simple Queue Serice queue	logstash-output-sqs
statsd	Sends metrics using the statsd network daemon	logstash-output-statsd
stomp	Writes events using the STOMP protocol	logstash-output-stomp
syslog	Sends events to a syslog server	logstash-output-syslog
webhdfs	Sends Logstash events to HDFS using the webhdfsREST API	logstash-output-webhdfs
websocket	Publishes messages to a websocket	logstash-output-websocket
xmpp	Posts events over XMPP	logstash-output-xmpp
zabbix	Sends events to a Zabbix server	logstash-output-zabbix
zeromq	Writes events to a ZeroMQ PUB socket	logstash-output-zeromq