Elasticsearch metrics integration

[steven-sheehy]

DevOps team uses Elasticsearch and Kibana to store and visualize metrics and logs. Currently the main node generates a CSV file with metrics which is scraped by Filebeat and stored in Elastic. On mirror node, we can do better by using Micrometer and its Elastic integration to push to Elastic on an interval. Micrometer also gives us a lot of built-in metrics by default (JVM, file, memory, logging, JMX, etc.) and we can add our custom ones on top.

[steven-sheehy]

Custom Metrics:

Download Rate

• Description: The duration in seconds it took to download the file from S3
• Name: hedera.mirror.download.request
• Tags: realm=0, shard=0, node=3, method=GET,status=200, action=(list|signature|signed), type=(balance|event|record)
• Type: Timer
• Unit: ms

Download Size

• Description: The size of the file in bytes that was downloaded from S3
• Name: hedera.mirror.download.response
• Tags: realm=0, shard=0, node=3, status=200, action=(list|signature|signed), type=(balance|event|record)
• Type: Distribution Summary
• Unit: bytes

Signature Validation

• Description: The number of signatures verified from a particular node
• Name: hedera.mirror.signature.verification
• Tags: realm=0, shard=0, node=3, status=(DOWNLOADED|PARSED|VERIFIED|CONSENSUS_REACHED), type=(balance|event|record)
• Type: Counter

Stream Verification

• Description: The duration in seconds it took to verify consensus and hash chain of a stream file
• Name: hedera.mirror.stream.verification
• Tags: success=(true|false), type=(balance|event|record)
• Type: Timer
• Unit: ms

File Parse Rate

• Description: The duration in ms it took to parse the file and store it in the database
• Name: hedera.mirror.parse.duration
• Tags: success=(true|false), type=(balance|event|record), version=(v1|v2)
• Type: Timer
• Unit: ms

Balance Insert Rate

• Description: The amount of time it took to process an account balance
• Name: hedera.mirror.balance.duration
• Tags: success=(true|false)
• Type: Timer
• Unit: ms

Event Insert Rate

• Description: The amount of time it took to process an event
• Name: hedera.mirror.event.duration
• Tags: success=(true|false), version=(v1|v2)
• Type: Timer
• Unit: ms

Event Latency

• Description: The difference in ms between the time the event was created and the mirror node processed the event
• Name: hedera.mirror.event.latency
• Tags: version=(v1|v2)
• Type: Timer
• Unit: ms

Event Size

• Description: The size of the event in bytes
• Name: hedera.mirror.event.size
• Tags: ``
• Type: Distribution Summary
• Unit: bytes

Transaction Latency

• Description: The difference in ms between the time consensus was achieved and the mirror node processed the transaction
• Name: hedera.mirror.transaction.latency
• Tags: type=(cryptotransfer|...)
• Type: Timer
• Unit: ms

Transaction Size

• Description: The size of the transaction in bytes
• Name: hedera.mirror.transaction.size
• Tags: type=(cryptotransfer|...)
• Type: Distribution Summary
• Unit: bytes

Transaction Consensus Time

• Description: The amount of time it took to achieve consensus by mainnet
• Name: hedera.transaction.duration
• Tags: type=(cryptotransfer|...)
• Type: Timer
• Unit: ms

Note: Overall latency can't be calculated without events since it should be record.consensusTimestamp - submitEvent.timestamp

[steven-sheehy]

Fixed by #753. Remaining tasks covered by #759