Skip to content
This repository has been archived by the owner on Jan 11, 2024. It is now read-only.

IPC-91: Test environment (Part 2) #161

Merged
merged 33 commits into from
Apr 27, 2023
Merged

IPC-91: Test environment (Part 2) #161

merged 33 commits into from
Apr 27, 2023

Conversation

aakoshh
Copy link
Contributor

@aakoshh aakoshh commented Apr 14, 2023 •
edited
Loading

Part of consensus-shipyard/ipc#443

Continuing down the README from where #157 left off.

New commands in this PR:

  • make topologies/<name>.sh to turn topologies/<name>.yaml into a runnable script
  • make setup/<name> to compile and run a topology script
  • make wallet creates a new wallet
  • make subnet creates a new subnet
  • make wallet/fund sends tokens to the wallet
  • make subnet/fund sends tokens to the subnet, executed on the parent node
  • make subnet/join makes the validator join the subnet , executed on the parent node

The only sane way to run these is e.g. make setup/simple.

Notes

It's not expected to start a validator before joining a subnet, yet it's done to simplify starting the containers with docker-compose. The validator will crash the first time it's started, but it's given a kick again in the post-up step, after joining; at that point it runs.

@aakoshh aakoshh marked this pull request as ready for review April 17, 2023 17:47
@aakoshh
Copy link
Contributor Author

aakoshh commented Apr 21, 2023 •
edited
Loading

Useful tip from Will:

❯ docker exec -it ipc-node-0-daemon eudico wallet list
Address                                    Balance                      Nonce  Default  
t1ah7phdlf5eg2nf25trxvvddzzde7w6ehtrfz2yi  1 FIL                        0               
t1cp4q4lqsdhob23ysywffg2tvbmar5cshia4rweq  4998.998985909661230076 FIL  2      X

@aakoshh aakoshh force-pushed the ipc-91-test-env-multi branch 6 times, most recently from 9003b2c to 26e508d Compare April 25, 2023 14:11
@aakoshh
Copy link
Contributor Author

aakoshh commented Apr 26, 2023

Here's an example log from setting up the simple topology: setup-log.txt

It actually fails at the end while setting up the sub-subnet with actor not found. I don't know why this error happens, but it's very frequent with the sub-subnet. cc @adlrocha

@adlrocha
Copy link
Contributor

It actually fails at the end while setting up the sub-subnet with actor not found. I don't know why this error happens, but it's very frequent with the sub-subnet. cc @adlrocha

This is because whatever address you are using for some operation doesn't have enough funds and consequently its account actors hasn't been yet initialized in the chain (thus the error)

@adlrocha
Copy link
Contributor

It's not expected to start a validator before joining a subnet, yet it's done to simplify starting the containers with docker-compose. The validator will crash the first time it's started, but it's given a kick again in the post-up step, after joining; at that point it runs.

We are improving this after this PR, and instead of crashing the validator polls to see if it can start mining on a subnet or not. consensus-shipyard/lotus#166

adlrocha
adlrocha reviewed Apr 26, 2023
View reviewed changes
Copy link
Contributor

@adlrocha adlrocha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢 This a perfect first iteration of what we wanted for the integration test. I think it can really reduce the debugging time. Let's get this merged :)

src/cli/commands/mod.rs
e
)
}
r.with_context(|| format!("error processing command {:?}", args.command))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really helpful, thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like I said the other change I would make (one that would probably break the scripts) is to log to stderr (which the agent does at the moment already) and use simple println to output results. That would make it easier to automate.

Base automatically changed from ipc-91-test-env to main April 26, 2023 11:46
@aakoshh
IPC-91: Copy genesis files. Use the one with funds in root
f4a2327
@aakoshh
IPC-91: Allow adding subnet variables to the topology
26cd1d8
@aakoshh
IPC-91: Move wallet out of genesis directory
e39ace8
@aakoshh
IPC-91: Various fixes
ad96d88
@aakoshh
IPC-91: Rename to topologies
a39ce92
@aakoshh
IPC-91: Join subnet
a7c160a
@aakoshh
IPC-91: Trying to fund the subnet
0892bd6
@aakoshh
IPC-91: Add explicit step tombstones so funding only happens once
506818c
@aakoshh
IPC-91: Add explicit step tombstone to the agent as well.
8843423
@aakoshh
IPC-91: Join first, then fund
260d178
@aakoshh
IPC-91: Add env var to construct agent URL to the node
257f201
@aakoshh
IPC-91: Fix CLI to exit with error code
46accf2
@aakoshh
IPC-91: Fix equality check
3b0e6f8
@aakoshh
IPC-91: Get rid of the wallet address from the validator net addr.
9498802
@aakoshh
IPC-91: Fix address in root
66d7232
@aakoshh
IPC-91: Give the validator another chance to start
a23d8f2
@aakoshh
IPC-91: Remove any newlines after docker capture
70b2110
@aakoshh
IPC-91: Fix shadowing of env vars
644eff4
@aakoshh
IPC-91: Ephasize topologies
d951b8e
@aakoshh
Copy link
Contributor Author

aakoshh commented Apr 26, 2023

Let's get this merged :)

Thanks! Just needs an approval.

@aakoshh aakoshh merged commit 0dc3e7e into main Apr 27, 2023
@aakoshh aakoshh deleted the ipc-91-test-env-multi branch April 27, 2023 07:28
cryptoAtwill added a commit that referenced this pull request May 5, 2023
@cryptoAtwill @jsoares
@aakoshh @adlrocha
migrate bottom up checkpoint (#182)
79b2169 
* Fix typo

* IPC-91: Test env (Part 1) (#157)

* IPC-91: Fix infra directory mkdir. Exit on fail.

* IPC-91: Update init config command

* IPC-91: Makefile to build the agent and eudico

* IPC-91: Allow empty subnets list

* IPC-91: Create agent docker-compose, up and down

* IPC-91: make clean

* IPC-91: Trying to get a eudico node going

* IPC-91: Separate daemon and validator

* IPC-91: Docs and clean

* IPC-91: Fix API address for the validator

* IPC-91: Remove volumes

* IPC-91: Copy the infra scripts that needed modification.

* IPC-91: Connect script. Fix clean

* IPC-91: Try reload the agent config

* IPC-91: Fix host to accept connections

* IPC-91: Sleep after wait. Fix finding files and config when no subnets yet

* IPC-91: Tab to spaces

* IPC-91: Spaces to tabs

* IPC-91: Config not needed to call reload

* IPC-91: Test environment (Part 2) (#161)

* IPC-91: make wallet

* IPC-91: Create a wallet for a child node

* IPC-91: Keep just one subnet-validator script

* IPC-91: Create new subnet

* IPC-91: Refer to the wallet key rather than copy.

* IPC-91: Start agent by number

* IPC-91: Create topology

* IPC-91: Fixes, stop on error

* IPC-91: Fix target in setup

* IPC-91: Remove agent NR from node env

* IPC-91: Comments and docs

* IPC-91: Simpler subnet to demonstrate the error

* IPC-91: Trying to fund the wallet created

* IPC-91: Copy genesis files. Use the one with funds in root

* IPC-91: Allow adding subnet variables to the topology

* IPC-91: Move wallet out of genesis directory

* IPC-91: Various fixes

* IPC-91: Rename to topologies

* IPC-91: Join subnet

* IPC-91: Trying to fund the subnet

* IPC-91: Add explicit step tombstones so funding only happens once

* IPC-91: Add explicit step tombstone to the agent as well.

* IPC-91: Join first, then fund

* IPC-91: Add env var to construct agent URL to the node

* IPC-91: Fix CLI to exit with error code

* IPC-91: Fix equality check

* IPC-91: Get rid of the wallet address from the validator net addr.

* IPC-91: Fix address in root

* IPC-91: Give the validator another chance to start

* IPC-91: Remove any newlines after docker capture

* IPC-91: Fix shadowing of env vars

* IPC-91: Ephasize topologies

* IPC-91: Remove leftover script

* migrate bottom up checkpoint

* add get validators and submit

* Update src/lotus/client.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Update src/lotus/mod.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* make obtain validator common

* handle error responses

* abstract common methods

* more logs and concurrent futures

* add pre submission checks:

* make lint

* remove unused method

* Migrate top down (#184)

* wip

* merge with bottom up

* merge with bottom up

* Update src/manager/checkpoint/manager/topdown.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Update src/manager/checkpoint/manager/topdown.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Update src/manager/checkpoint/manager/topdown.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Integrate new checkpoint (#185)

* integrate new checkpoint system

* format code

* add error logging

* add more logging

* add epoch of fund and release

* add more logging

* update info

* update bottom up

* update comment

* update log

* update logging

* remove subsystem file

---------

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* remove default gateway address const (#188)

---------

Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com>
Co-authored-by: Akosh Farkash <aakoshh@gmail.com>
Co-authored-by: adlrocha <adlrocha@tutamail.com>
adlrocha added a commit that referenced this pull request May 18, 2023
@cryptoAtwill @jsoares
@aakoshh @adlrocha
Refactor checkpoint submission (#175)
255cd80 
* refactored code

* add sleep

* update sleep sec

* reorg code

* simplify

* migrate bottom up checkpoint (#182)

* Fix typo

* IPC-91: Test env (Part 1) (#157)

* IPC-91: Fix infra directory mkdir. Exit on fail.

* IPC-91: Update init config command

* IPC-91: Makefile to build the agent and eudico

* IPC-91: Allow empty subnets list

* IPC-91: Create agent docker-compose, up and down

* IPC-91: make clean

* IPC-91: Trying to get a eudico node going

* IPC-91: Separate daemon and validator

* IPC-91: Docs and clean

* IPC-91: Fix API address for the validator

* IPC-91: Remove volumes

* IPC-91: Copy the infra scripts that needed modification.

* IPC-91: Connect script. Fix clean

* IPC-91: Try reload the agent config

* IPC-91: Fix host to accept connections

* IPC-91: Sleep after wait. Fix finding files and config when no subnets yet

* IPC-91: Tab to spaces

* IPC-91: Spaces to tabs

* IPC-91: Config not needed to call reload

* IPC-91: Test environment (Part 2) (#161)

* IPC-91: make wallet

* IPC-91: Create a wallet for a child node

* IPC-91: Keep just one subnet-validator script

* IPC-91: Create new subnet

* IPC-91: Refer to the wallet key rather than copy.

* IPC-91: Start agent by number

* IPC-91: Create topology

* IPC-91: Fixes, stop on error

* IPC-91: Fix target in setup

* IPC-91: Remove agent NR from node env

* IPC-91: Comments and docs

* IPC-91: Simpler subnet to demonstrate the error

* IPC-91: Trying to fund the wallet created

* IPC-91: Copy genesis files. Use the one with funds in root

* IPC-91: Allow adding subnet variables to the topology

* IPC-91: Move wallet out of genesis directory

* IPC-91: Various fixes

* IPC-91: Rename to topologies

* IPC-91: Join subnet

* IPC-91: Trying to fund the subnet

* IPC-91: Add explicit step tombstones so funding only happens once

* IPC-91: Add explicit step tombstone to the agent as well.

* IPC-91: Join first, then fund

* IPC-91: Add env var to construct agent URL to the node

* IPC-91: Fix CLI to exit with error code

* IPC-91: Fix equality check

* IPC-91: Get rid of the wallet address from the validator net addr.

* IPC-91: Fix address in root

* IPC-91: Give the validator another chance to start

* IPC-91: Remove any newlines after docker capture

* IPC-91: Fix shadowing of env vars

* IPC-91: Ephasize topologies

* IPC-91: Remove leftover script

* migrate bottom up checkpoint

* add get validators and submit

* Update src/lotus/client.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Update src/lotus/mod.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* make obtain validator common

* handle error responses

* abstract common methods

* more logs and concurrent futures

* add pre submission checks:

* make lint

* remove unused method

* Migrate top down (#184)

* wip

* merge with bottom up

* merge with bottom up

* Update src/manager/checkpoint/manager/topdown.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Update src/manager/checkpoint/manager/topdown.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Update src/manager/checkpoint/manager/topdown.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Integrate new checkpoint (#185)

* integrate new checkpoint system

* format code

* add error logging

* add more logging

* add epoch of fund and release

* add more logging

* update info

* update bottom up

* update comment

* update log

* update logging

* remove subsystem file

---------

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* remove default gateway address const (#188)

---------

Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com>
Co-authored-by: Akosh Farkash <aakoshh@gmail.com>
Co-authored-by: adlrocha <adlrocha@tutamail.com>

* update comments

* Bottom up checkpoint proof (#199)

* add bottom up proof

* format proof

* wrap inner proof

* update comment

* Update src/manager/checkpoint/proof/mod.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* update proof state

* Update src/manager/checkpoint/proof/v1.rs

---------

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* update mem push method

* update response

* share key store

* add subnet to lotus client

* fix docs

---------

Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com>
Co-authored-by: Akosh Farkash <aakoshh@gmail.com>
Co-authored-by: adlrocha <adlrocha@tutamail.com>
adlrocha added a commit that referenced this pull request May 22, 2023
@cryptoAtwill @jsoares
@aakoshh @adlrocha
Fix checkpoint config load (#205)
17a6445 
* refactored code

* add sleep

* update sleep sec

* reorg code

* simplify

* migrate bottom up checkpoint (#182)

* Fix typo

* IPC-91: Test env (Part 1) (#157)

* IPC-91: Fix infra directory mkdir. Exit on fail.

* IPC-91: Update init config command

* IPC-91: Makefile to build the agent and eudico

* IPC-91: Allow empty subnets list

* IPC-91: Create agent docker-compose, up and down

* IPC-91: make clean

* IPC-91: Trying to get a eudico node going

* IPC-91: Separate daemon and validator

* IPC-91: Docs and clean

* IPC-91: Fix API address for the validator

* IPC-91: Remove volumes

* IPC-91: Copy the infra scripts that needed modification.

* IPC-91: Connect script. Fix clean

* IPC-91: Try reload the agent config

* IPC-91: Fix host to accept connections

* IPC-91: Sleep after wait. Fix finding files and config when no subnets yet

* IPC-91: Tab to spaces

* IPC-91: Spaces to tabs

* IPC-91: Config not needed to call reload

* IPC-91: Test environment (Part 2) (#161)

* IPC-91: make wallet

* IPC-91: Create a wallet for a child node

* IPC-91: Keep just one subnet-validator script

* IPC-91: Create new subnet

* IPC-91: Refer to the wallet key rather than copy.

* IPC-91: Start agent by number

* IPC-91: Create topology

* IPC-91: Fixes, stop on error

* IPC-91: Fix target in setup

* IPC-91: Remove agent NR from node env

* IPC-91: Comments and docs

* IPC-91: Simpler subnet to demonstrate the error

* IPC-91: Trying to fund the wallet created

* IPC-91: Copy genesis files. Use the one with funds in root

* IPC-91: Allow adding subnet variables to the topology

* IPC-91: Move wallet out of genesis directory

* IPC-91: Various fixes

* IPC-91: Rename to topologies

* IPC-91: Join subnet

* IPC-91: Trying to fund the subnet

* IPC-91: Add explicit step tombstones so funding only happens once

* IPC-91: Add explicit step tombstone to the agent as well.

* IPC-91: Join first, then fund

* IPC-91: Add env var to construct agent URL to the node

* IPC-91: Fix CLI to exit with error code

* IPC-91: Fix equality check

* IPC-91: Get rid of the wallet address from the validator net addr.

* IPC-91: Fix address in root

* IPC-91: Give the validator another chance to start

* IPC-91: Remove any newlines after docker capture

* IPC-91: Fix shadowing of env vars

* IPC-91: Ephasize topologies

* IPC-91: Remove leftover script

* migrate bottom up checkpoint

* add get validators and submit

* Update src/lotus/client.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Update src/lotus/mod.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* make obtain validator common

* handle error responses

* abstract common methods

* more logs and concurrent futures

* add pre submission checks:

* make lint

* remove unused method

* Migrate top down (#184)

* wip

* merge with bottom up

* merge with bottom up

* Update src/manager/checkpoint/manager/topdown.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Update src/manager/checkpoint/manager/topdown.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Update src/manager/checkpoint/manager/topdown.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* Integrate new checkpoint (#185)

* integrate new checkpoint system

* format code

* add error logging

* add more logging

* add epoch of fund and release

* add more logging

* update info

* update bottom up

* update comment

* update log

* update logging

* remove subsystem file

---------

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* remove default gateway address const (#188)

---------

Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com>
Co-authored-by: Akosh Farkash <aakoshh@gmail.com>
Co-authored-by: adlrocha <adlrocha@tutamail.com>

* update comments

* Bottom up checkpoint proof (#199)

* add bottom up proof

* format proof

* wrap inner proof

* update comment

* Update src/manager/checkpoint/proof/mod.rs

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* update proof state

* Update src/manager/checkpoint/proof/v1.rs

---------

Co-authored-by: adlrocha <adlrocha@tutamail.com>

* update mem push method

* update response

* share key store

* add subnet to lotus client

* fix docs

* load config should not crash programme

* load config should not crash programme

* wait for config reload

* Update mod.rs

---------

Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com>
Co-authored-by: Akosh Farkash <aakoshh@gmail.com>
Co-authored-by: adlrocha <adlrocha@tutamail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@adlrocha adlrocha adlrocha approved these changes

@cryptoAtwill cryptoAtwill Awaiting requested review from cryptoAtwill

@jsoares jsoares Awaiting requested review from jsoares

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@aakoshh @adlrocha