Support custom metrics rules for PrometheusSink

[thinker0]

#3445

Support custom metrics rules for PrometheusSink

AS-IS

• Prometheus Metrics (apache-storm-kafka-client Spot)

heron_instance_bytes_received_1
heron_client_stmgr_10_hello_messages_to_stmgrs
heron_kafkaoffset_ad_dmp_dpa_event_integration_partition_100_earliesttimeoffset
heron_kafkaoffset_ad_dmp_dpa_event_integration_partition_100_latestcompletedoffset
heron_kafkaoffset_ad_dmp_dpa_event_integration_partition_100_latestemittedoffset
heron_kafkaoffset_ad_dmp_dpa_event_integration_partition_100_latesttimeoffset
heron_kafkaoffset_ad_dmp_dpa_event_integration_partition_100_recordsinpartition
heron_kafkaoffset_ad_dmp_dpa_event_integration_partition_100_spoutlag

TO-BE

Add metics_sink.yaml

### Config for prometheus-sink
prometheus-sink:
  class: "org.apache.heron.metricsmgr.sink.PrometheusSink"
  # port: 8080 # The port on which to run (either port or port-file are mandatory)
  path: /metrics # The path on which to publish the metrics (mandatory)
  port-file: metrics.port
  flat-metrics: true # By default the web-sink will publish a flat "name -> value" json map
  include-topology-name: true # Include topology name in metric name (default false)
  metrics-cache-max-size: 1000000 # Max number of metrics cached and published (default 1000000)
  metrics-cache-ttl-sec: 600 # Time in seconds after which a metric that was collected will stopped being published (default 600)
  rules:
    # kafkaOffset of KafkaSpout 'kafkaOffset/topicName/totalSpoutLag'
    - pattern: kafkaOffset/(.+)/(.+)
      name: kafka_offset_$2
      attrNameSnakeCase: true
      type: COUNTER
      labels:
        topic: "$1"
    # kafkaOffset of KafkaSpout 'kafkaOffset/topicName/partition_2/spoutLag'
    - pattern: kafkaOffset/(.+)/partition_([0-9]+)/(.+)
      name: kafka_offset_$3
      attrNameSnakeCase: true
      type: COUNTER
      labels:
        topic: "$1"
        partition: "$2"

to

heron_instance_bytes_received{metric_instance_id="1"}
heron_client_stmgr_hello_messages_to_stmgrs{metric_instance_id="stmgr-10"}
heron_kafka_offset_earliest_time_offset{topic="ad_dmp_dpa_event_integration",partition="100"}
heron_kafka_offset_latest_completed_offset{topic="ad_dmp_dpa_event_integration",partition="100"}
heron_kafka_offset_latest_emitted_offset{topic="ad_dmp_dpa_event_integration",partition="100"}
heron_kafka_offset_latest_time_offset{topic="ad_dmp_dpa_event_integration",partition="100"}
heron_kafka_offset_records_in_partition{topic="ad_dmp_dpa_event_integration",partition="100"}
heron_kafka_offset_spout_lag{topic="ad_dmp_dpa_event_integration",partition="100"}

[thinker0]

@nicknezis how to retest this please

[joshfischer1108]

@thinker0 I don't seem to have the permissions to restart the build anymore. Not sure what's going on with Travis. @nwangtw could you try to restart the build?

[joshfischer1108]

@thinker0 I was able to restart the job for you.