Easier Onboarding and Maintenance

[afmsavage]

I want to start a discussion around different ways to ease the onboarding of new nodes as well as making it easier to maintain a node after it has started running. There are some data points about to be exposed from the node which will help in monitoring. Some ideas that have been tossed out are:

• Marketplace node image on difference providers (AWS, GCP, DigitalOcean) allowing for 1 click deploy
• Automation script to ask user questions for different required variables for config creation and container spinup
• Monitoring dashboard/sidecar to be included with the node. Could use a Docker Compose file to create both in conjunction

Any discussion or additional ideas around these items is appreciated. We would like to make running a mainnet node as painless and possible as it can be. Real money after all!

[ElderOrb]

Absolutely! Onboarding simplicity and health monitoring would not only help node operators, but eliminate questions like 'is my node ok' allowing team to concentrate on the project. Speaking about dashboard - at the moment its blocked by the lack of RPC, so lets hope this is something which will appear before mainnet.

Also I would add builds simplification to the list, especially for native builds. Single 'make && make install' or 'build.sh' would be really helpful. Maybe it would even allow to reduce requirements for CPU / memory and use the cheapest of vps.

[ElderOrb]

And one more item worth discussion... Taken into account vps typically doesn't provide enough hdd space for full geth and infura is not that cheap - would be great to find some alternatives like:

• investigate whether lite eth node works
• investigate possibility to introduce proxy node (on vps?) which would forward requests to free infura or some other node (this could also reduce risk of slashing due to node issues)
• collect list of eth providers (if any exist besides infura)

[mhluongo]

at the moment its blocked by the lack of RPC, so lets hope this is something which will appear before mainnet.

In progress 😀

Who's our ultimate audience here? Are we looking for a "node in a box" where the team and/or community actually maintain a full from-scratch script, or just the pieces to allow other people to? One of the reasons I haven't advocated for a 1-liner setup is that it usually requires bad trust assumptions.

If we can get to the point where people can choose, I'd love it. You guys want to describe a setup experience you'd like to see as a new staker? Is it more like

1. a better build process

git clone https://github.com/keep-network/keep-ecdsa.git
make ...
# more stuff from scratch

2. pre-built releases

wget https://keep.network/releases/ecdsa/1.0
wget https://keep.network/releases/core/1.0
# stuff to push binaries to VPS, etc

or are you thinking to go as far as

3. a 1-liner?

curl https://keep.network/releases/latests/setup.sh | bash
./magic-keep-setup

[ElderOrb]

at the moment its blocked by the lack of RPC, so lets hope this is something which will appear before mainnet.

In progress 😀

Cool, how could I miss it!

Who's our ultimate audience here? Are we looking for a "node in a box" where the team and/or community actually maintain a full from-scratch script, or just the pieces to allow other people to? One of the reasons I haven't advocated for a 1-liner setup is that it usually requires bad trust assumptions.

If we can get to the point where people can choose, I'd love it. You guys want to describe a setup experience you'd like to see as a new staker? Is it more like

1. a better build process

git clone https://github.com/keep-network/keep-ecdsa.git
make ...

This is my personal preference with './configure' between 'git clone' and 'make' (which would install all the prerequisites and ask for things like keystore path, eth rpc and so on). But mostly because for me its simpler to just ssh to vps and do things above than to investigate how to push things to vps using some non-standard solutions :). ... which is funny taking into account here https://github.com/ElderOrb/keep.network-build-scripts I'm trying to implement ugly hybrid of solution 3 and 1 :)

more stuff from scratch

1. pre-built releases

wget https://keep.network/releases/ecdsa/1.0
wget https://keep.network/releases/core/1.0
stuff to push binaries to VPS, etc

This should make most of non-technical users happy but might take significantly more efforts because each vps has some peculiarities.

or are you thinking to go as far as

1. a 1-liner?

curl https://keep.network/releases/latests/setup.sh | bash
./magic-keep-setup

magic-keep-setup allowing to select between docker & native, configure and push to vps, deploy health monitoring and mining bitcoins in parallel sounds like great idea for ICO. :) Too much efforts imo..

Independently on the approach, actual peers list and contracts could be pulled by 'configure' script from git and written into config file. Especially peers list which causes a lot of questions in discord.

[afmsavage]

@mhluongo I get your point of avoiding a scenario where trust is needed to ensure a script, binary, or whatever is secure. What about utilizing IPFS plus a .eth domain in some way. The script or whatever can be hosted there, while a link for people to download whatever can be provided to the IPFS hash of that. Services such as https://eth.link/ make it so the user doesn't even need to be running an IPFS compatible browser or local node.

Once the nodes data points are ready as well, a simple dashboard can be hosted in IPFS that would allow for the boostrap peerlists to be advertised to users. If you want decentralization, we can do decentralization!

[experiencedft]

I have created simple scripts that I believe will significantly simplify onboarding for ECDSA v1.0.0 testnet nodes connected to the blockchain via Infura. This should be easy to update for future versions as well as mainnet, or for people who want to connect to the Ethereum blockchain via something else than Infura (e.g. Cloudflare for mainnet).

Here is what it looks like from the user perspective for the setup of the node.

The script automates all steps usually found in tutorials. Since it is already a well configured file, there is no risk of copy pasting issues with line ending formatting and such.

Parts of the scripts are taken from @afmsavage's guide here.

In concrete terms, the UX would look like this:

1. Create a MEW wallet with keystore option and rename the file to keep_wallet.json

2. Import wallet to Metamask

3. Create an Infura project

4. Get KEEP tokens grant, delegate tokens, authorize contracts, and bond some ETH via dashboard

5. Download the setup_testnet_ecdsa_1-0-0.sh and run_testnet_ecdsa_1-0-0.sh scripts from a trusted, verifiable source (official GitHub repo for example).

6. Move the scripts in the home directory.

7. Run source ./setup_testnet_ecdsa_1-0-0.sh and follow the instructions (just copy pasting Infura ID and other stuff). This creates the desired directories structure, exports environment variables, adds them to the .bashrc so that the user needs not export them again upon restart, and configures the config.toml file.

8. Run run_testnet_ecdsa_1-0-0.sh.

And this is it.

Note that this doesn't require any knowledge of command line instructions and can be easily used to onboard newbies. Furthermore, it gives insight into what is going on under the hood with command line outputs to ease newcomers into learning the stuff they will need to maintain future mainnet nodes.

Attached are the two aforementioned scripts. I have QA tested this from scratch locally.

I believe this is the simplest way to setup a node pending the development of executables.

Note: This also simplifies issues related to compatibility between different tutorials in terms of variable naming and such, and clearly flags the files as setup_testnet_ecdsa_v1-0-0.sh or setup_mainnet_beacon_v1-2-0rc.sh etc for easier troubleshooting.

onboarding_scripts.zip

[afmsavage]

Awesome work @jeremyid /experience. You should create a github repo with these files in place, and your walkthrough as the README.md. I could help you with working with git on Discord if you would like so that others can submit PRs for any bugs or feedback. It also allows you to distribute the scripts in a more trustful manner than sending someone a .zip file since the source code is available for review inside of the repo. @me when you want to go through this.

[experiencedft]

You should create a github repo with these files in place, and your walkthrough as the README.md.

I will look into this on Monday when I get back to my workstation. I thought I would get some feedback here before creating a repo. Thank you for your comment!

[afmsavage]

You should create a github repo with these files in place, and your walkthrough as the README.md.

I will look into this on Monday when I get back to my workstation. I thought I would get some feedback here before creating a repo. Thank you for your comment!

You want the feedback on your repo, not in a comment on a different project. PRs are the way of the future! I took the weekend away from KEEP really, but am starting to think back about easier onboarding items that can be done. I am going to schedule a Jitsu call for sometime this coming week for all stakers to come and get help or give help.

[experiencedft]

Published: https://github.com/jeremyid/easy-keep-nodes

[Eastban120]

Hi guys,
I've got involved in Keep project some days after stakedrop in June. I installed my nodes and I’m currently working on the translation into spanish (https://gist.github.com/Estebank97/e9a5246c264a69cc30b19ab7d0ac7e44) of the most complete documentation - with the exception of the whitepaper- I found, which is https://keep-network.gitbook.io/staking-documentation/keep-network-staking/keep-network-101.
I’ve read all the stuff you wrote here, which I couldn’t agree more. @jeremyid Experience’ scripts are awesome. As I understand it it’s utterly important to get as many nodes as possible up and running. And running smoothly without downtimes. With upgrades done rather fast when needed.
So, I guess the first two issues originally mentioned by Antonio @afmsavage are taken care of. There are more and more good installation guides out there.

But I’m worried with the third issue.
When the time comes where we start putting some serious stake in Keep Network we should make sure every node operator (with or without devops experience) is comfortable with node monitoring, downtime issues and alerts handling.

As a month old node operator myself there are some common issues which IMHO should be better addressed by the whole community:

• what to look at in the logs (which warnings are important, which one are not)
• how to make a complete monitoring dashboard (I’ve installed Grafana monitoring as per @mutedtommy’s guide, which is great but there is no ECDSA specific log info there)
• how to alert node operator of failures automatically (I've installed telegram bot from Grafana, but it is sufficient ?)
• since Edcsa nodes will have variable ETH/KEEP ($$$) stake and Beacon nodes will have an important stake (100k KEEP - $$$$) involved, may be there should be some redundancy configurations available. Or some “easy” container shut down, switch over, turn on path prepared.

The core Keep Network are their nodes, so these nodes should be as safe as possible. If we manage to keep downtime issues minimal, Slashing should only occur if there is misbehaving, and our Keep Network will be really healthy.

[afmsavage]

I wanted to share my monitoring setup here in case anyone comes across this. I think its still a good topic to keep up for now, especially if we can generate some conversations around new node runners coming on board for mainnet. I plan to do a better writeup eventually but for now this should get people on their way.

Monitoring Setup
https://gist.github.com/afmsavage/7c8a9ccf085bedbc0a2880472a9ef984