如何设置Samza的metrics

参考这个里边对API的调用

http://samza.incubator.apache.org/learn/documentation/0.7.0/container/metrics.html

参考这里列出的可用配置

http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/configuration-table.html

在正常的配置之外，需要额外添加的配置。添加这些配置以后，就可以在StreamTask里使用metrics的API来提供metrics信息

 

1. 需要哪些配置？
   1. 设置用哪种reporter factory，Samza自带有Kafka和JMX两种
   2. 设置reporter使用的stream,包括设置这个stream的serde
   3. 注册这个reporter

#设置用来做为输出流的Stream

#设置samza-metrics这个stream的system

streams.samza-metrics.system= kafka 

#设置samza-metrics这个stream的名字,即在kafka里对应的topic的名字

streams.samza-metrics.stream= samza-metrics

---

#配置reporter

#设置reporter factory这里使用的MetricsSnapshotReporterFactory,把JSON格式的metrics发给Kafka。这个class要是Factory的名字，samza当前的document里的配置是错的

metrics.reporter.samza-metrics.class= org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory

 

#samza-metrics这个reporter10s发送一次metrics

metrics.reporter.samza-metrics.window.ms= 10000

 

#定义samza-metrics这个reporter对应的stream。此reporter使用kafka这个system下的samza-metrics这个stream。注意一定要使用

#system.stream这种写法，即写成kafka.samza-metrics

#这个配置是必须的，但是Samza的config文档里没有列出

metrics.reporter.samza-metrics.stream= kafka.samza-metrics

 

#注册samza-metrics这个reporter

metrics.reporters= samza-metrics

---

#配置serde

# Serializers 定义可以使用的Serde factory的名字

serializers.registry.json.class= org.apache.samza.serializers.JsonSerdeFactory

serializers.registry.metrics.class= org.apache.samza.serializers.MetricsSnapshotSerdeFactory

 

#定义samza-metrics这个stream，用来做消息serde的Serde factory为metrics。由于这个stream输出的是metrics的信息，因此必须使用这个serde factory

systems.kafka.streams.samza-metrics.samza.msg.serde= metrics

 

 

最后会有三种metrics。StramTask的、SystemProducer的、SystemConsumer的

{"metrics":{"org.apache.samza.container.TaskInstanceMetrics":{"process-calls":28575767,"messages-sent":0,"commit-calls":12,"window-skipped":30204500,"kafka-pizza-offset":"36950462","commit-skipped":30204488,"send-skipped":30204500,"window-calls":0,"send-calls":0},"class hs.samza.simple.SimpleKafkaTask":{"messageCount":28500000}},"header":{"reset-time":1398672217312,"job-id":"1","time":1398672937854,"host":"hadoop-node-1","container-name":"samza-container-2","source":"Partition-2","job-name":"my-samza-test","samza-version":"0.0.1","version":"0.0.1"}}

{"metrics":{"org.apache.samza.system.kafka.KafkaSystemProducerMetrics":{"kafka-producer-sends":38,"kafka-partition-2-producer-buffer-size":0,"kafka-flushes":38,"kafka-metricssnapshotreporterfactory-producer-buffer-size":0,"kafka-samza-container-2-producer-buffer-size":0,"kafka-producer-reconnects":0,"kafka-flush-sizes":38}},"header":{"reset-time":1398672217312,"job-id":"1","time":1398672937855,"host":"hadoop-node-1","container-name":"samza-container-2","source":"MetricsSnapshotReporterFactory","job-name":"my-samza-test","samza-version":"0.0.1","version":"0.0.1"}}

{"metrics":{"org.apache.samza.system.SystemConsumersMetrics":{"blocking-poll-timeout":10,"kafka-messages-per-poll":15667419,"chose-object":29071417,"kafka-ssp-fetches-per-poll":1501156,"max-buffered-messages-per-stream-partition":1000,"ssps-needed-by-chooser":1,"kafka-pizza-messages-chosen":29071417,"unprocessed-messages":0,"chose-null":1483650,"kafka-polls":15667420,"poll-timeout":10},"org.apache.samza.metrics.JvmMetrics":{"threads-runnable":5,"mem-heap-committed-mb":313.5625,"threads-new":0,"mem-non-heap-committed-mb":24.75,"mem-heap-used-mb":93.080475,"mem-non-heap-used-mb":24.63375,"threads-terminated":0,"ps marksweep-gc-time-millis":41,"ps scavenge-gc-count":380,"ps scavenge-gc-time-millis":16289,"gc-time-millis":16330,"threads-blocked":0,"threads-timed-waiting":6,"ps marksweep-gc-count":1,"threads-waiting":4,"gc-count":381},"org.apache.samza.container.SamzaContainerMetrics":{"process-null-envelopes":1483649,"process-envelopes":29071417,"process-calls":30555067,"commit-calls":30555066,"window-calls":30555066,"send-calls":30555066},"org.apache.samza.system.chooser.RoundRobinChooserMetrics":{"buffered-messages":0},"org.apache.samza.system.kafka.KafkaSystemConsumerMetrics":{"kafka-pizza-3-offset-change":37437828,"poll-count":15667420,"kafka-10.5.132.122-9092-topic-partitions":1,"no-more-messages-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":false,"kafka-10.5.132.122-9092-messages-read":36263,"blocking-poll-count-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":0,"kafka-pizza-3-bytes-read":1512471142,"kafka-pizza-3-messages-read":29071417,"kafka-10.5.132.122-9092-skipped-fetch-requests":8,"blocking-poll-timeout-count-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":728147,"kafka-pizza-3-messages-behind-high-watermark":0,"buffered-message-count-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":0,"kafka-10.5.132.122-9092-bytes-read":1512471142,"kafka-10.5.132.122-9092-reconnects":0},"org.apache.samza.system.kafka.KafkaSystemProducerMetrics":{"kafka-producer-sends":0,"kafka-flushes":14,"kafka-partition-3-producer-buffer-size":0,"kafka-producer-reconnects":0,"kafka-flush-sizes":0},"org.apache.samza.system.SystemProducersMetrics":{"partition-3-sends":0,"partition-3-flushes":14,"flushes":14,"sends":0}},"header":{"reset-time":1398672219649,"job-id":"1","time":1398673001854,"host":"hd-e.cdh","container-name":"samza-container-3","source":"samza-container-3","job-name":"my-samza-test","samza-version":"0.0.1","version":"0.0.1"}}

看着头都大了。应该还是JMX那种reporter看着舒服点。以后再试吧……