Developing metrics for Nano Representatives

[mistakia]

status: Draft
title: Developing Metrics for Nano Representatives
author: trashman
date created: 2024-03-26

Motivation

To help the average Nano holder select a representative, optimizing for Network performance, reliability, self-sovereignty, and censorship resistance.

Design Principles

• easy/quick to understand by the average Nano holder
• use percentiles scaled from 0-100, with 100 being "good" and 0 being "bad"
• prefer objectivity, avoid subjectivity and preferences of the designers except for in cases where there may be unanimous agreement

Specification

Observations will be aggregated from multiple non-voting nodes.

Defining & measuring performance

A performant representative helps blocks reach quorum (confirmation) quickly and consistently. I propose the following metrics to capture speed and consistency

• Vote Latency : A measure of how quickly a representative votes on a block compared to the first vote observed. The percentile rank of the median vote latency for a representative over a period of time will be used with a lower vote latency being considered "good". Vote latency will be calculated for normal and final votes.
• Slow Final Vote : This tracks how often votes arrive late, after a block has reached quorum. The percentile rank of the percentage of slow votes will be calculated over a period of time with a low percentage being considered "good".
• Fast Vote : A fast vote is a vote received before a round of voting has reached quorum. The percentile rank of the percentage of confirmation votes will be calculated over a period of time with a high percentage being considered "good". Fast votes will be calculated for normal and final votes.
• Participation : Participation is defined as the percentage of blocks a vote has been received for over a period of time. The percentile rank of the percentage will be used with a higher percentage being considered "good". Participation will be calculated for normal and final votes.

These metrics will serve as the basis for calculating composite metrics that will then be used to calculate a representative's score. Further research and observation will be required to develop the composite metrics. Potential composite metrics:

• Percentage of periods with a sub-25 percentile vote latency
• Percentage of periods with a sub-25 percentile slow vote
• Percentage of periods with a sub-25 percentile confirmation vote
• Percentage of periods with a sub-25 percentile participation

Defining bad, abusive, and misconfigured behavior, along with censorship

The following events will be considered bad behavior. Reps observed committing these behaviors should be immutably added to the bad behavior list along with the messages observed

• observing conflicting final votes, two different votes for the same election root
• observing conflicting normal votes, two different votes for the same election root and same timestamp
• receiving a conflicting block
• receiving a block belonging to the burn account

Reps observed to be engaging in censorship should be added to the censorship list, along with information about which blocks they are censoring, until the behavior ceases.

• representative will not respond to vote requests for a given block

The following events will be considered abuse or misconfiguration. Reps observed committing these behaviors should be added to the misconfigured reps list until the behavior ceases.

• observing normal or final votes from multiple IPs over a period of time
• receiving invalid signatures (for a block, vote, handshake)
• receiving malformed message headers or bodies
• receiving a fork for a confirmed block
• receiving a block with an invalid balance, height, or work field
• receiving a high rate of unchecked blocks (missing previous or source)
• receiving a high rate of confirmed blocks

Defining & measuring uptime

Uptime will be measured over 1-minute and 5-minute resolutions with a representative being considered online if it generates a single vote (normal or final) for a block first observed in the period being monitored.

Defining & measuring reputation, commitment, and trustworthiness

Developing objective measures of reputation and trustworthiness are challenging, thus I have chosen to focus on commitment and attentiveness. A committed and attentive representative has been running for longer and applies non-contentious updates promptly.

• Time since first observed : The percentile rank of the UNIX timestamp of the first vote observed, with a lower timestamp being "good".
• Slow Upgrades : A slow upgrade is observed when a node does not apply a minor or major upgrade within 7 days of the version achieving a quorum (67%) of voting weight. A low number of slow upgrades will be considered "good".
• Upgrade Latency : Upgrade latency will be measured as the difference in time between a node applying an upgrade and when the version was first applied by a principal representative. The percentile rank of the median upgrade latency will be used with a lower upgrade latency being considered "good".

Defining & measuring censorship, liveliness, and conflict resolution risks of voting weight concentration

Voting weight concentration increases the risk of censorship, liveliness, and conflict resolution issues. The key thresholds for voting weight concentration are 34% (censorship & liveliness) and 67% (conflict resolution) of the online weight or trended weight, whichever is greater. Being that 34% is less than 67% and liveliness is a more prominent risk that threshold should be the focus when considering how to develop a score for voting weight concentration. With that in mind, a potential voting weight score function could be a sigmoid curve scaled from 100 to 0 for voting weight values from 0 to 5%.

Voting Weight Score Formula

where x is the voting weight percentage without decimal conversion (e.g. 1% = 1.0)

$$ 100 - \frac{100}{1 + e^{-2.5x + 0.6e^2}} $$

0.1% = 98 (x = 0.1)
0.5% = 96 (x = 0.5)
1.0% = 87 (x = 1.0)
1.5% = 66 (x = 1.5)
2.5% = 14 (x = 2.5)
4.0% = 0.3 (x = 4.0)

Defining the "Outdated" List

Representatives that are running an older version than the current version that has quorum will be added to the "Outdated" list. Representatives will remain on this list until they update to a version that has quorum.

Tasks

• update database schema
• implement a light node for message monitoring
• store normal vote and final vote timestamps
• create a script to calculate vote latency, slow votes, confirmation votes, and participation
• create a script to calculate slow upgrades
• create a script to calculate upgrade latency

[kopeboy]

Wow, thank you for this great work.

I noticed that the Voting Weight Score Formula above is not producing the results you put underneath, for example:

• x = 5% leads to 100 - ( 100 / (1 + e^(-2.5 * 0.05 + 0.6 * e^2))) = 98.6724015564
• x = 1% => 98.7972196872
• x = 0.1% => 98.8236652695

And I have a question: how to measure latency of votes without considering the internet connection latency between the node and you, the observer? Shouldn't all representative nodes measure the performance metrics of each other?

[mistakia]

Ah the discrepancy is between how the percentage is represented (i.e. 1.0 vs 0.1 for 1%). I was not converting from percentage to decimal. I've updated the post so it's more clear (good catch). Perhaps the formula should be adjusted to be more correct so that X could be a "percentage".

And I have a question: how to measure latency of votes without considering the internet connection latency between the node and you, the observer?

The suggested approach for measuring "vote latency" is not directly measuring internet connection latency but rather latency as a whole. It's an attempt at an encompassing measure of not just messaging latency but processing/computing latency as well. Simple put, it's how quickly did you get me your vote relative to the fastest voter (first voter).

I think measuring messaging latency directly is useful and something we should be doing so I've opened this issue to get that underway — #120

Shouldn't all representative nodes measure the performance metrics of each other?

There would have to be a justification for that overhead for voting nodes specifcally. There hasn't been a need for that thus far. The only thing that comes to mind is it could be relevant to "vote spacing". Though I don't think something like the suggested consensus improvement draft by colin would have a need for it.

To sum up, my thinking is that voting nodes (representatives) should be optimized/focused on voting performance. Thus any other functions like network monitoring should be done by other nodes with extended functionality.