[Infrastructure] LocalNet on Kubernetes #354

okdas · 2022-11-14T19:30:29Z

Description

Implements LocalNet that is powered by Kubernetes.

Issue

~~pocket-operator issue: pokt-network/pocket-operator#7~~ not using operator currently, just utilizing Kubernetes directly.

Type of change

Please mark the relevant option(s):

List of changes

Introducing new dependencies to run localnet developer environment

Tilt
Kubernetes - the workloads controlled by Tilt and pocket-operator run on local kuberenetes. Potential choices are:
- Docker Desktop
- Rancher Desktop
- other options: https://docs.tilt.dev/choosing_clusters.html

Testing

tilt up should bring the whole stack online.
LocalNet w/ all of the steps outlined in the README

Required Checklist

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have tested my changes using the available tooling
I have updated the corresponding CHANGELOG

If Applicable Checklist

I have updated the corresponding README(s); local and/or global
I have added tests that prove my fix is effective or that my feature works
I have added, or updated, mermaid.js diagrams in the corresponding README(s)
I have added, or updated, documentation and mermaid.js diagrams in shared/docs/* if I updated shared/*README(s)

ManInWeb3

lgtm
Great job

Olshansk · 2022-11-22T21:52:10Z

lgtm Great job

Before we merge this in, @okdas can you show (gif, video, demo, instructions) of it working?

Also, please make sure to follow the LocalNet test steps to make sure it works.

okdas · 2022-11-23T22:56:37Z

@Olshansk of course, I'm adding the instructions! :)

Olshansk · 2022-12-06T23:57:45Z

@Olshansk of course, I'm adding the instructions! :)

@okdas Lmk when this is ready for a spin.

okdas · 2022-12-07T23:35:09Z

@Olshansk of course, I'm adding the instructions! :)

@okdas Lmk when this is ready for a spin.

Still in progress, but if you want to check it out locally and provide feedback/see if you catch any issues please make sure to follow the doc and also utilize the dk-localnet-pocket-operator branch from pocket-operator (the second repo is needed - more info in docs).

…ndor/mock/etc changes

okdas · 2022-12-10T00:55:20Z

docs/development/localnet-with-kubernetes.md

+
+### How to change configuration files
+
+LocalNet resources are managed by pocket-operator, so if you'd like to change a config file (e.g. genesis file), you can do it by modifying the corresponding CRD in `pocket-operator` repo.


I think I should expand on that, but this should probably go into the pocket-operator documentation and linked here.

Olshansk · 2023-02-03T03:36:01Z

f75fde2

I really like the until nc -z trick. Brilliant.

I am thinking about a way to stake and unstake validators when they come online/go offline during the scale-up scale-down operations. My idea is to write a simple program that interacts with client-go to react to k8s events and stake/unstake accordingly (obviously I'd have to change the genesis so that the validators have POKT to stake and also provide access to the keys of the validators to this program, parsing the private-keys.yaml I guess).

@okdas My understanding is this:

We couldn't get pocket-operator working on LocalNet (for now)
We'll use pocket-operator in DevNet
The current configuration scales the number of validators (e.g. services) but doesn't actually trigger the stake/unstake command based on k8s events

The flow @deblasis described might literally let us scale (and discover) validators in a local environment next week, so I'd support going in that approach unless I'm missing something?

Olshansk

A small set of minor NITs and comments and we're good to go!

The only major point I wanted to discuss is why we shouldn't move forward with the approach outlined by Alessandro.

build/localnet/README.md

Olshansk · 2023-02-02T22:20:04Z

Tiltfile

+])
+
+# # Exposes postgres port to 5432 on the host machine.
+# k8s_resource(new_name='postgres',


Yup, just need to update the comment.

Olshansk · 2023-02-02T22:20:56Z

build/localnet/cli-client.yaml

@@ -0,0 +1,71 @@
+apiVersion: apps/v1


@deblasis put this together. See field #6 in P2PConfig. Because of the prefix we use with viper + the change in case POCKET_P2P_IS_CLIENT_ONLY maps to is_client_only in the proto.

message P2PConfig { string private_key = 1; uint32 consensus_port = 2; bool use_rain_tree = 3; conn.ConnectionType connection_type = 4; uint64 max_mempool_count = 5; // this is used to limit the number of nonces that can be stored in the mempool, after which a FIFO mechanism is used to remove the oldest nonces and make space for the new ones bool is_client_only = 6; }

build/localnet/README.md

Olshansk · 2023-02-03T04:01:26Z

build/localnet/Tiltfile

@@ -3,25 +3,29 @@ load("ext://helm_resource", "helm_resource", "helm_repo")
 load("ext://namespace", "namespace_create")
 load("ext://restart_process", "docker_build_with_restart")

+tiltfile_dir = os.path.dirname(config.main_dir)


Thank you for doing this! I think it's going to make the codebase much easier to understand

build/localnet/Tiltfile

Olshansk · 2023-02-03T04:06:26Z

build/localnet/templates/v1-validator-template.yaml.tpl

@@ -69,6 +71,15 @@ spec:
              subPath: genesis.json
            - name: validator-storage
              mountPath: /validator-storage
+      initContainers:


I like this :)

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

okdas · 2023-02-03T22:17:44Z

The only major point I wanted to discuss is why we shouldn't move forward with the approach outlined by Alessandro.

We totally should! I love what @deblasis did!

I think I've addressed all of your suggestions, let me resolve conflicts and it'll be ready for another round.

Olshansk · 2023-02-03T22:56:25Z

build/localnet/README.md

+Addresses begin with `YYYXX` number, where `YYY` is a number of an actor and `XX` is [a type of actor](../../shared/core/types/proto/actor.proto#L7). For example:
+*  `420043b854e78f2d5f03895bba9ef16972913320` is a validator #420.
+*  `66603bc4082281b7de23001ffd237da62c66a839` is a fisherperson #666.
+*  `0010297b55fc9278e4be4f1bcfe52bf9bd0443f8` is a service node (a servicer) #001.


Just FYI: We're going to rename all Service Node like instances to Servicer soon

Olshansk

Just one minor suggestion, but otherwise, LGTM!

build/localnet/README.md

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

okdas added 2 commits November 11, 2022 17:04

tilt + PO + localnet

f7a82ba

--wip-- [skip ci]

62683a5

okdas added the infra Core infrastructure - not protocol related label Nov 14, 2022

okdas added this to the M2: Pocket DoS (Devnet of Servicers) milestone Nov 14, 2022

okdas requested a review from a team as a code owner November 14, 2022 19:30

okdas self-assigned this Nov 14, 2022

okdas requested a review from Gustavobelfort November 14, 2022 20:03

okdas added 4 commits November 14, 2022 16:59

--wip-- [skip ci]

67132eb

observability stack

764c722

--wip-- [skip ci]

c2c0fd5

grafana is auto provisioned with observability stack

2960669

ManInWeb3 approved these changes Nov 22, 2022

View reviewed changes

--wip-- [skip ci]

1455a40

Olshansk added the do not merge Prevent merging even with sufficient approvals label Nov 29, 2022

Olshansk marked this pull request as draft November 29, 2022 16:45

stabilizing tilt

ca9ebeb

okdas added 2 commits December 7, 2022 12:04

--wip-- [skip ci]

1fd8bb1

more changes to grafana, tilt and docs

048947c

more docs

2fb9f04

okdas mentioned this pull request Dec 10, 2022

Tilt for pocket operator pokt-network/pocket-operator#11

Merged

okdas marked this pull request as ready for review December 10, 2022 00:37

okdas changed the title ~~[WIP] LocalNet controlled by pocket-operator~~ LocalNet controlled by pocket-operator Dec 10, 2022

okdas and others added 3 commits December 9, 2022 16:44

let tilt ignore the same stuff as git, so it wont get triggered by ve…

e66bd74

…ndor/mock/etc changes

note about changing pocket-operator code

cc95782

Merge branch 'main' into dk-localnet-pocket-operator

5198b1d

okdas commented Dec 10, 2022

View reviewed changes

okdas added 2 commits February 2, 2023 17:25

update links

54a7629

include demo

7ec2217

okdas requested a review from Olshansk February 3, 2023 01:46

Olshansk requested changes Feb 3, 2023

View reviewed changes

okdas and others added 10 commits February 3, 2023 12:35

Update build/localnet/README.md

33a45cf

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

Update build/localnet/README.md

469e67a

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

Update build/localnet/README.md

a8df9d9

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

Update build/localnet/README.md

5a5c042

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

Update build/localnet/Tiltfile

62561c6

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

Update build/localnet/README.md

46d3f74

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

Update build/localnet/README.md

2135d26

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

Update build/localnet/README.md

fe98bce

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

Update build/localnet/Tiltfile

1e59e9c

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

clarify postgres ports

1c17e68

okdas and others added 2 commits February 3, 2023 14:19

Merge branch 'main' into dk-localnet-pocket-operator

f3d083b

fix changelogs

8e864f3

okdas requested a review from Olshansk February 3, 2023 22:36

oops, this got lost!

93851be

Olshansk reviewed Feb 3, 2023

View reviewed changes

Olshansk approved these changes Feb 3, 2023

View reviewed changes

build/localnet/README.md Outdated Show resolved Hide resolved

deblasis mentioned this pull request Feb 6, 2023

[Infra] K8S Cluster manager #490

Closed

14 tasks

okdas and others added 3 commits February 6, 2023 10:28

Update build/localnet/README.md

3347fbe

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>

Merge branch 'main' into dk-localnet-pocket-operator

b8ba1c9

change date on changelog

5a1325b

okdas removed the do not merge Prevent merging even with sufficient approvals label Feb 6, 2023

redundant changelog

da4a402

okdas merged commit 6728d6f into main Feb 6, 2023

okdas deleted the dk-localnet-pocket-operator branch February 6, 2023 19:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Infrastructure] LocalNet on Kubernetes #354

[Infrastructure] LocalNet on Kubernetes #354

okdas commented Nov 14, 2022 •

edited

Loading

ManInWeb3 left a comment

Olshansk commented Nov 22, 2022

okdas commented Nov 23, 2022

Olshansk commented Dec 6, 2022

okdas commented Dec 7, 2022

okdas Dec 10, 2022

Olshansk commented Feb 3, 2023

Olshansk left a comment

Olshansk Feb 2, 2023

Olshansk Feb 2, 2023

Olshansk Feb 3, 2023

Olshansk Feb 3, 2023

okdas commented Feb 3, 2023

Olshansk Feb 3, 2023

Olshansk left a comment


		### How to change configuration files

		LocalNet resources are managed by pocket-operator, so if you'd like to change a config file (e.g. genesis file), you can do it by modifying the corresponding CRD in `pocket-operator` repo.

[Infrastructure] LocalNet on Kubernetes #354

[Infrastructure] LocalNet on Kubernetes #354

Conversation

okdas commented Nov 14, 2022 • edited Loading

Description

Issue

Type of change

List of changes

Testing

Required Checklist

If Applicable Checklist

ManInWeb3 left a comment

Choose a reason for hiding this comment

Olshansk commented Nov 22, 2022

okdas commented Nov 23, 2022

Olshansk commented Dec 6, 2022

okdas commented Dec 7, 2022

okdas Dec 10, 2022

Choose a reason for hiding this comment

Olshansk commented Feb 3, 2023

Olshansk left a comment

Choose a reason for hiding this comment

Olshansk Feb 2, 2023

Choose a reason for hiding this comment

Olshansk Feb 2, 2023

Choose a reason for hiding this comment

Olshansk Feb 3, 2023

Choose a reason for hiding this comment

Olshansk Feb 3, 2023

Choose a reason for hiding this comment

okdas commented Feb 3, 2023

Olshansk Feb 3, 2023

Choose a reason for hiding this comment

Olshansk left a comment

Choose a reason for hiding this comment

okdas commented Nov 14, 2022 •

edited

Loading