Flink - metrics

 

Metrics是以MetricsGroup来组织的

MetricGroup

MetricGroup

这就是个metric容器，里面可以放subGroup，或者各种metric

所以主要的接口就是注册，

/**
 * A MetricGroup is a named container for {@link Metric Metrics} and further metric subgroups.
 *
 * <p>Instances of this class can be used to register new metrics with Flink and to create a nested
 * hierarchy based on the group names.
 *
 * <p>A MetricGroup is uniquely identified by it's place in the hierarchy and name.
 */
public interface MetricGroup {
    <C extends Counter> C counter(int name, C counter);
    <T, G extends Gauge<T>> G gauge(int name, G gauge);
    <H extends Histogram> H histogram(String name, H histogram);
    MetricGroup addGroup(String name);
}

 

AbstractMetricGroup

关键是实现MetricGroup，逻辑很简单，在注册或close的时候都需要加锁互斥

/**
 * Abstract {@link MetricGroup} that contains key functionality for adding metrics and groups.
 *
 */

public abstract class AbstractMetricGroup implements MetricGroup {

    /** The registry that this metrics group belongs to */
    protected final MetricRegistry registry;

    /** All metrics that are directly contained in this group */
    private final Map<String, Metric> metrics = new HashMap<>();

    /** All metric subgroups of this group */
    private final Map<String, AbstractMetricGroup> groups = new HashMap<>();

    /** The metrics scope represented by this group.
     *  For example ["host-7", "taskmanager-2", "window_word_count", "my-mapper" ]. */
    private final String[] scopeComponents;  //命名空间

    /** The metrics scope represented by this group, as a concatenated string, lazily computed.
     * For example: "host-7.taskmanager-2.window_word_count.my-mapper" */
    private String scopeString;

    @Override
    public <C extends Counter> C counter(String name, C counter) {
        addMetric(name, counter);
        return counter;
    }

    /**
     * Adds the given metric to the group and registers it at the registry, if the group
     * is not yet closed, and if no metric with the same name has been registered before.
     *
     * @param name the name to register the metric under
     * @param metric the metric to register
     */
    protected void addMetric(String name, Metric metric) {
        // add the metric only if the group is still open
        synchronized (this) { //加锁
            if (!closed) {
                // immediately put without a 'contains' check to optimize the common case (no collition)
                // collisions are resolved later
                Metric prior = metrics.put(name, metric);

                // check for collisions with other metric names
                if (prior == null) {
                    // no other metric with this name yet

                    registry.register(metric, name, this);
                }
                else {
                    // we had a collision. put back the original value
                    metrics.put(name, prior);

                }
            }
        }
    }
}

 

MetricReporter

采集好的Metrics需要用reporter才能发送出去，

/**
 * Reporters are used to export {@link Metric Metrics} to an external backend.
 *
 * <p>Reporters are instantiated via reflection and must be public, non-abstract, and have a
 * public no-argument constructor.
 */
public interface MetricReporter {

    // ------------------------------------------------------------------------
    //  life cycle
    // ------------------------------------------------------------------------

    /**
     * Configures this reporter. Since reporters are instantiated generically and hence parameter-less,
     * this method is the place where the reporters set their basic fields based on configuration values.
     *
     * <p>This method is always called first on a newly instantiated reporter.
     *
     * @param config The configuration with all parameters.
     */
    void open(MetricConfig config);

    /**
     * Closes this reporter. Should be used to close channels, streams and release resources.
     */
    void close();

    void notifyOfAddedMetric(Metric metric, String metricName, MetricGroup group);
    void notifyOfRemovedMetric(Metric metric, String metricName, MetricGroup group);
}

 

AbstractReporter实现MetricReport接口，

/**
 * Base interface for custom metric reporters.
 */
public abstract class AbstractReporter implements MetricReporter, CharacterFilter {
    protected final Logger log = LoggerFactory.getLogger(getClass());

    protected final Map<Gauge<?>, String> gauges = new HashMap<>();
    protected final Map<Counter, String> counters = new HashMap<>();
    protected final Map<Histogram, String> histograms = new HashMap<>();

    @Override
    public void notifyOfAddedMetric(Metric metric, String metricName, MetricGroup group) {
        final String name = group.getMetricIdentifier(metricName, this); //group只是用来获取metrics完整的name

        synchronized (this) {
            if (metric instanceof Counter) {
                counters.put((Counter) metric, name);
            } else if (metric instanceof Gauge) {
                gauges.put((Gauge<?>) metric, name);
            } else if (metric instanceof Histogram) {
                histograms.put((Histogram) metric, name);
            } else {
                log.warn("Cannot add unknown metric type {}. This indicates that the reporter " +
                    "does not support this metric type.", metric.getClass().getName());
            }
        }
    }

    @Override
    public void notifyOfRemovedMetric(Metric metric, String metricName, MetricGroup group) {
        synchronized (this) {
            if (metric instanceof Counter) {
                counters.remove(metric);
            } else if (metric instanceof Gauge) {
                gauges.remove(metric);
            } else if (metric instanceof Histogram) {
                histograms.remove(metric);
            } else {
                log.warn("Cannot remove unknown metric type {}. This indicates that the reporter " +
                    "does not support this metric type.", metric.getClass().getName());
            }
        }
    }
}

 

MetricRegistry

MetricRegistry用于连接MetricGroups和MetricReporters，

会把需要report的metric加到MetricReporters，并启动定时的report线程

/**
 * A MetricRegistry keeps track of all registered {@link Metric Metrics}. It serves as the
 * connection between {@link MetricGroup MetricGroups} and {@link MetricReporter MetricReporters}.
 */
public class MetricRegistry {

    private List<MetricReporter> reporters;
    private ScheduledExecutorService executor;

    private final ScopeFormats scopeFormats;

    private final char delimiter;

    /**
     * Creates a new MetricRegistry and starts the configured reporter.
     */
    public MetricRegistry(Configuration config) {
        // first parse the scope formats, these are needed for all reporters
        ScopeFormats scopeFormats;
        try {
            scopeFormats = createScopeConfig(config);  //从配置中读到scope的格式，即监控数据的namespace的格式是什么
        }
        catch (Exception e) {
            LOG.warn("Failed to parse scope format, using default scope formats", e);
            scopeFormats = new ScopeFormats();
        }
        this.scopeFormats = scopeFormats;

        char delim;
        try {
            delim = config.getString(ConfigConstants.METRICS_SCOPE_DELIMITER, ".").charAt(0);  //从配置里面读出分隔符
        } catch (Exception e) {
            LOG.warn("Failed to parse delimiter, using default delimiter.", e);
            delim = '.';
        }
        this.delimiter = delim;

        // second, instantiate any custom configured reporters
        this.reporters = new ArrayList<>();

        final String definedReporters = config.getString(ConfigConstants.METRICS_REPORTERS_LIST, null);  //读出配置的Reporters

        if (definedReporters == null) {
            // no reporters defined
            // by default, don't report anything
            LOG.info("No metrics reporter configured, no metrics will be exposed/reported.");
            this.executor = null;
        } else {
            // we have some reporters so
            String[] namedReporters = definedReporters.split("\\s*,\\s*");
            for (String namedReporter : namedReporters) {  //对于配置的每个reporter

                DelegatingConfiguration reporterConfig = new DelegatingConfiguration(config, ConfigConstants.METRICS_REPORTER_PREFIX + namedReporter + ".");
                final String className = reporterConfig.getString(ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX, null);  //reporter class名配置

                try {
                    String configuredPeriod = reporterConfig.getString(ConfigConstants.METRICS_REPORTER_INTERVAL_SUFFIX, null); //report interval配置
                    TimeUnit timeunit = TimeUnit.SECONDS;
                    long period = 10;

                    if (configuredPeriod != null) {
                        try {
                            String[] interval = configuredPeriod.split(" ");
                            period = Long.parseLong(interval[0]);
                            timeunit = TimeUnit.valueOf(interval[1]);
                        }
                        catch (Exception e) {
                            LOG.error("Cannot parse report interval from config: " + configuredPeriod +
                                    " - please use values like '10 SECONDS' or '500 MILLISECONDS'. " +
                                    "Using default reporting interval.");
                        }
                    }

                    Class<?> reporterClass = Class.forName(className);
                    MetricReporter reporterInstance = (MetricReporter) reporterClass.newInstance(); //实例化reporter

                    MetricConfig metricConfig = new MetricConfig();
                    reporterConfig.addAllToProperties(metricConfig);
                    reporterInstance.open(metricConfig);  //open reporter

                    if (reporterInstance instanceof Scheduled) {
                        if (this.executor == null) {
                            executor = Executors.newSingleThreadScheduledExecutor(); //创建Executor
                        }
                        LOG.info("Periodically reporting metrics in intervals of {} {} for reporter {} of type {}.", period, timeunit.name(), namedReporter, className);

                        executor.scheduleWithFixedDelay(
                                new ReporterTask((Scheduled) reporterInstance), period, period, timeunit); //Scheduled report
                    }
                    reporters.add(reporterInstance); //加入reporters列表
                }
                catch (Throwable t) {
                    shutdownExecutor();
                    LOG.error("Could not instantiate metrics reporter" + namedReporter + ". Metrics might not be exposed/reported.", t);
                }
            }
        }
    }

    // ------------------------------------------------------------------------
    //  Metrics (de)registration
    // ------------------------------------------------------------------------

    /**
     * Registers a new {@link Metric} with this registry.
     *
     * @param metric      the metric that was added
     * @param metricName  the name of the metric
     * @param group       the group that contains the metric
     */
    public void register(Metric metric, String metricName, MetricGroup group) { //在AbstractMetricGroup.addMetric中被调用，metric被加到group的同时也会加到reporter中

        try {
            if (reporters != null) {
                for (MetricReporter reporter : reporters) {
                    if (reporter != null) {
                        reporter.notifyOfAddedMetric(metric, metricName, group); //把metric加到每个reporters上面
                    }
                }
            }
        } catch (Exception e) {
            LOG.error("Error while registering metric.", e);
        }
    }

    /**
     * Un-registers the given {@link org.apache.flink.metrics.Metric} with this registry.
     *
     * @param metric      the metric that should be removed
     * @param metricName  the name of the metric
     * @param group       the group that contains the metric
     */
    public void unregister(Metric metric, String metricName, MetricGroup group) {
        try {
            if (reporters != null) {
                for (MetricReporter reporter : reporters) {
                    if (reporter != null) {
                        reporter.notifyOfRemovedMetric(metric, metricName, group);
                    }
                }
            }
        } catch (Exception e) {
            LOG.error("Error while registering metric.", e);
        }
    }

    // ------------------------------------------------------------------------

    /**
     * This task is explicitly a static class, so that it does not hold any references to the enclosing
     * MetricsRegistry instance.
     *
     * This is a subtle difference, but very important: With this static class, the enclosing class instance
     * may become garbage-collectible, whereas with an anonymous inner class, the timer thread
     * (which is a GC root) will hold a reference via the timer task and its enclosing instance pointer.
     * Making the MetricsRegistry garbage collectible makes the java.util.Timer garbage collectible,
     * which acts as a fail-safe to stop the timer thread and prevents resource leaks.
     */
    private static final class ReporterTask extends TimerTask {

        private final Scheduled reporter;

        private ReporterTask(Scheduled reporter) {
            this.reporter = reporter;
        }

        @Override
        public void run() {
            try {
                reporter.report();  //Task的核心就是调用reporter.report
            } catch (Throwable t) {
                LOG.warn("Error while reporting metrics", t);
            }
        }
    }
}

 

TaskManager

在TaskManager中，

associateWithJobManager

metricsRegistry = new FlinkMetricRegistry(config.configuration)

taskManagerMetricGroup =
  new TaskManagerMetricGroup(metricsRegistry, this.runtimeInfo.getHostname, id.toString)

TaskManager.instantiateStatusMetrics(taskManagerMetricGroup)

创建metricsRegistry 和TaskManagerMetricGroup

可以看到instantiateStatusMetrics，只是注册各种taskManager的status metrics，

private def instantiateStatusMetrics(taskManagerMetricGroup: MetricGroup) : Unit = {
    val jvm = taskManagerMetricGroup
      .addGroup("Status")
      .addGroup("JVM")

    instantiateClassLoaderMetrics(jvm.addGroup("ClassLoader"))
    instantiateGarbageCollectorMetrics(jvm.addGroup("GarbageCollector"))
    instantiateMemoryMetrics(jvm.addGroup("Memory"))
    instantiateThreadMetrics(jvm.addGroup("Threads"))
    instantiateCPUMetrics(jvm.addGroup("CPU"))
  }

  private def instantiateClassLoaderMetrics(metrics: MetricGroup) {
    val mxBean = ManagementFactory.getClassLoadingMXBean //从ManagementFactory可以取出表示JVM指标的MXBean

    metrics.gauge[Long, FlinkGauge[Long]]("ClassesLoaded", new FlinkGauge[Long] {
      override def getValue: Long = mxBean.getTotalLoadedClassCount
    })
    metrics.gauge[Long, FlinkGauge[Long]]("ClassesUnloaded", new FlinkGauge[Long] {
      override def getValue: Long = mxBean.getUnloadedClassCount
    })
  }

 

在submitTask的时候，

submitTask

  val taskMetricGroup = taskManagerMetricGroup.addTaskForJob(tdd)

  val task = new Task(
    tdd,
    memoryManager,
    ioManager,
    network,
    bcVarManager,
    selfGateway,
    jobManagerGateway,
    config.timeout,
    libCache,
    fileCache,
    runtimeInfo,
    taskMetricGroup)

看到会为每个task，创建taskMetricGroup

并在创建Task对象的时候传入该对象，

Environment env = new RuntimeEnvironment(jobId, vertexId, executionId,
        executionConfig, taskInfo, jobConfiguration, taskConfiguration,
        userCodeClassLoader, memoryManager, ioManager,
        broadcastVariableManager, accumulatorRegistry,
        splitProvider, distributedCacheEntries,
        writers, inputGates, jobManager, taskManagerConfig, metrics, this);

// let the task code create its readers and writers
invokable.setEnvironment(env);

在Task中， 关键的就是把这个taskMetricGroup，加入RuntimeEnvironment，这样在实际逻辑中，就可以通过RuntimeEnvironment获取到metrics

而StreamTask就是一种Invokable，接口定义如下

public abstract class AbstractInvokable {

    /** The environment assigned to this invokable. */
    private Environment environment;

    /**
     * Starts the execution.
     *
     * <p>Must be overwritten by the concrete task implementation. This method
     * is called by the task manager when the actual execution of the task
     * starts.
     *
     * <p>All resources should be cleaned up when the method returns. Make sure
     * to guard the code with <code>try-finally</code> blocks where necessary.
     *
     * @throws Exception
     *         Tasks may forward their exceptions for the TaskManager to handle through failure/recovery.
     */
    public abstract void invoke() throws Exception;

    /**
     * Sets the environment of this task.
     *
     * @param environment
     *        the environment of this task
     */
    public final void setEnvironment(Environment environment) {
        this.environment = environment;
    }

    /**
     * Returns the environment of this task.
     *
     * @return The environment of this task.
     */
    public Environment getEnvironment() {
        return this.environment;
    }
}

 

所以在StreamTask里面可以这样使用metrics，

getEnvironment().getMetricGroup().gauge("lastCheckpointSize", new Gauge<Long>() {
    @Override
    public Long getValue() {
        return StreamTask.this.lastCheckpointSize;
    }
});