ethstats plugin for node status reporting

[veox]

Background

ethstats is an Ethereum-specific, opt-in node status reporting service.

The original JavaScript implementation was developed by Marian Oancea (@cubedro).

The server component is called eth-netstats; it can be seen on ethstats.net (ETH main-net), ropsten-stats.parity.io (Ropsten test-net), and others, including private/development chains via puppeth.

The stand-alone client component is called eth-net-intelligence-api; this can be used with any node implementing a certain sub-set of Ethereum's JSON-RPC queries required by e-n-i-a.

(An alternative approach - taken, for example, by geth, - is implementing a client as a sub-service in the node itself.)

Client-server communication happens over WebSockets.

This issue is about implementing the functionality in trinity, preferably as a plug-in.

Details

e-n-i-a is by now mostly unmaintained; it also expects features that trinity does not currently have, like a JSON-RPC-over-HTTP interface, as well as certain JSON-RPC queries (e.g. subscriptions to
"new block" events).

Instead of trying to "fix" that, it "would be nice" to have this as a plug-in. This eliminates the need for JSON-RPC-over-HTTP (which is itself well-suited to be a plug-in), and allows using internal APIs (event
subscriptions, timers, etc.).

The plug-in APIs in trinity are unstable; this work will help produce a "demo" of what it takes to write a trinity plug-in, as well as shape the APIs themselves.

Inputs

These are listed "for information purposes"; refer to the existing implementations for actual ones.

Note that some are or will be implemented as part of a different plug-in!

(Likely obtained using internal APIs; falling back to JSON-RPC request over IPC socket could be possible, but that would probably show an inconsistency in internal APIs...)

• Operator-specified (yet human-readable ;)) node name;
• operator-specified "contact details";
• node software id/version;
• OS id/version;
• Python impl-n id/version;
• whether active/syncing/mining;
• PoW hash rate (actually mostly useless due to prevalence of grinder proxies);
• chain head:
  • block number/hash/difficulty;
  • transaction count;
  • ommer (uncle) count;
• recommended gas price;
• peer count;
• transaction count in pending transaction pool;
• uptime.

Output

JSON messages over a WebSocket, to an operator-specified remote server.

Since ethstats is a "push" scheme, a "shared secret" (essentially a passphrase, also operator-specified) must be used.

Notes

I highly recommend reading @shazow's notes on the ethstats protocol (linked below, in section References -> Server) to get in touch with its quirks.

Personally, I found geth's client code slightly more readable than e-n-i-a, but I'm very biased.

Also, to clarify: I'm not exactly thrilled by the ethstats protocol, but it's already got infrastructure around it to be immediately usable.

References

Client

• geth's ethstats package, enabled with the --ethstats CLI arg, in Go;
• original eth-net-intelligence-api stand-alone implementation, as a PM2 node application, in JS.

Server

• @shazow's vipnode/ethstats re-implementation of the collection server, in Go;
• original eth-netstats server implementation, in JS.

Possibly related issues/PRs

These are for further exploration of the topic. Status in parenthesis, at time of posting.

• PR Convert "fix-unclean-shutdown" to plugin #1182 (merged) - Convert "fix-unclean-shutdown" to plugin
• Issue Plugin architecture #979 (open) - Plugin architecture
• Issue Convert plugin API to be purely event based #1090 (open) - Convert plugin API to be purely event based
• Issue Create guide on how to write plugins #1103 (open) - Create guide on how to write plugins

[pipermerriam]

cc @owocki

This is a non trivial task but well enough suited to be bountied. Whoever takes this on should expect:

• To have some support from my team, primarily myself and @cburgdorf
• To do a reasonable amount of just figuring it out
• To spend roughly a week working on this intermittently (not a full time week of work, but rather a few rounds of development, basic review and feedback)

And similarly, this is not a bounty well suited for a junior or beginner level contributor. Well suited to someone who is confident in their software architecture and development ability.

[owocki]

@veox @pipermerriam thanks for the thoughtful spec and expectation-setting

i will tee this off with a 500 DAI bounty, and i will set it to an approval-required bounty which is a new type of bounty that allows bounty posters to screen who is going to be working on the issue.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 500.0 DAI (500.0 USD @ $1.0/DAI) attached to it.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $20,714.98 more funded OSS Work available on the Gitcoin Issue Explorer

[shazow]

@veox

Also, to clarify: I'm not exactly thrilled by the ethstats protocol, but it's already got infrastructure around it to be immediately usable.

I did a survey of the ecosystem a couple months back and I was surprised at how little infrastructure was actually available around the protocol. All I found was some dashboards, the original NodeJS implementation and the half-hearted geth implementation. I didn't get the impression that any of these components were still actively developed.

The biggest component that does exist and we can benefit from is the popular https://ethstats.net/ collector.

I feel it's still early enough to consider proposing something new, if it fits better. Related notes to my thoughts on this are here: https://github.com/vipnode/ethstats#appendix

[cburgdorf]

That's a great issue to help drive our plugin architecture! As @pipermerriam already said, I'd be happy to work with whoever wants to take this bounty. It doesn't seem to be something that can be finished easily just yet as we need to make changes to our architecture to enable plugins like this. So figuring out the right path forward will be a big part of the mission.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work has been started.

These users each claimed they can complete the work by 10 months, 2 weeks from now.
Please review their action plans below:

1) evgeniuz has been approved to start work.

Figuring something out is my favorite part of development. Unfortunately, I have no prior experience in the internal works of py-evm, but would like to learn that as well, so if this issue is not time critical I would like to work on it. Will start by checking out existing implementations, making some rough plug-in with minimal functionality (at least some stats) and then continue by implementing other stats.

Learn more on the Gitcoin Issue Details page.

[owocki]

@pipermerriam let me know if you'd like to work with the above applicant

[owocki]

@evgeniuz has been auto approved to start work... i'd still feel better about an explicit approval though.

[pipermerriam]

Taking a look, been away for the weekend. Taking a look at @evgeniuz I'm not seeing the depth of contribution I'd like to give me confidence. That isn't an outright "no" but I'm hesitant. Maybe we can start and see where things are at a week from now?

[gitcoinbot]

@evgeniuz Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• warning (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[vs77bb]

Hi @evgeniuz could you let us know about your progress here? We'd like to see where you've been able to get here in week one, as this is a pretty advanced task. Hope you're doing well!

[gitcoinbot]

@evgeniuz Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• warning (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@evgeniuz Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• warning (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[owocki]

@evgenius looks like theres a lot of comments on the PR from piper => #1216

[evgenius]

@owocki no way, don't bring me into this :-D

[cubedro]

Guys, we're working on a new version of ethstats. we have a rock-solid server now with more features like node and events persistence in Redis or Cassandra, authentication, node log history. we also developed a new client https://www.npmjs.com/package/ethstats-cli

We did breaking changes but we're working on making the server backwards compatible with the old version (and geth implementation). In the upcoming weeks we'll open source everything.

@baxy is the lead dev on this project. Please let us know how we can help. We're more than happy to provide early access to our new version.

[evgeniuz]

@cubedro Thanks for the heads up. Do you plan to have some specification of the API available (i. e. list of API calls, responses, available fields)?

I've glanced at ethstats-cli source and it looks that flow of API is mostly same, so it shouldn't be too difficult to upgrade to new protocol version.

[cubedro]

@evgeniuz we will have a documentation for the API calls, responses etc. but we don't at the moment. We'll put something together in the upcoming weeks.
We tried to keep the flow roughly the same so there shouldn't be too many changes needed to upgrade.

[veox]

@cubedro You've misspelled @evgeniuz again. ;) Fixed.

[gitcoinbot]

@evgeniuz Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• warning (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

@evgeniuz due to inactivity, we have escalated this issue to Gitcoin's moderation team. Let us know if you believe this has been done in error!

• warning (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@evgeniuz Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• warning (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[pipermerriam]

I'm @evgenius there's PR feedback waiting in #1216 after which it'll be up to @cburgdorf to give final sign-off.

[carver]

Well hopefully evgenius has muted this thread by now 😆 .

Just making sure that @evgeniuz gets pinged about ^.

[veox]

I'll try and test the code worked on in PR #1216 (and then squash-and-merged in PR #1336) at the earliest opportunity.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work for 500.0 DAI (500.0 USD @ $1.0/DAI) has been submitted by:

1. @evgeniuz
2. @evgeniuz

@owocki please take a look at the submitted work:

• PR by @evgeniuz
• PR by @evgeniuz

---

• Learn more on the Gitcoin Issue Details page
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $23,183.14 more funded OSS Work available on the Gitcoin Issue Explorer

[owocki]

@carver let me know if/when good to payout.

[pipermerriam]

@cburgdorf can be the one to authorize payment for this one.

[cburgdorf]

I looked through the issue description (both here and on gitcoin) and couldn't find a clear definition of done. (No finger pointing, I totally missed to recognize that when I read the issue initially. I just think in the future that would be helpful to have)

I'd be happy to see @evgeniuz add some tests and docs (obviously all other improvements are welcome to) but that shouldn't hold us back from paying out this bounty.

So yes @owocki, let's pay this out.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

The funding of 500.0 DAI (500.0 USD @ $1.0/DAI) attached to this issue has been approved & issued to @evgeniuz.

• Learn more on the Gitcoin Issue Details page
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $28,361.67 more funded OSS Work available on the Gitcoin Issue Explorer