docker: Include vigilantes and explorer in the localnet deployment #123

vitsalis · 2022-09-08T13:30:00Z

This PR makes some modifications to the docker-compose methodology to setup a simnet. More specifically:

Removes bitcoinsim from localnet-start. The reasoning for this is that localnet-start should only be about testing the Cosmos codebase itself and not the entire Babylon stack.
For testing the entire babylon stack, the make simnet-start command is introduced. It builds the bitcoinsim machine, a localnet of 4 babylon nodes, a vigilante reporter, and a vigilante submitter. The vigilante inclusion is possible due to the recent update on the Dockerfiles for the vigilantes.
A vigilante.yaml file is introduced. This file is useful for providing the configuration for the vigilante nodes. The aim is to ultimately remove this file by having the vigilante implement a command similar to babylond testnet to build a configuration file.
The bitcoinsim image is updated so that it stores the rpc.cert and rpc.key files in a directory that can be accessed by the vigilante nodes. The vigilantes need the certificates to communicate with BTC properly.
The shared volumes for the vigilante, BTC, and Babylon are stored under .testnets. This means that the .testnets directory now contains gentxs, node{i}, bitcoin, and vigilante directories.

Currently, the vigilante reporter on the above setup fail with the following error:

vigilante-reporter    | time="2022-09-08T04:25:16Z" level=info msg="Successfully loaded config file at /vigilante/vigilante.yaml" module=config
vigilante-reporter    | time="2022-09-08T04:25:16Z" level=info msg="Successfully created the BTC client and connected to the BTC server" module=btcclient
vigilante-reporter    | time="2022-09-08T04:25:16Z" level=info msg="Successfully subscribed to newly connected/disconnected blocks from BTC" module=btcclient
vigilante-reporter    | time="2022-09-08T04:25:16Z" level=debug msg="Babylon key directory: /babylonconfig" module=babylonclient
vigilante-reporter    | time="2022-09-08T04:25:16Z" level=debug msg="All Babylon addresses: map[]" module=babylonclient
vigilante-reporter    | time="2022-09-08T04:25:16Z" level=info msg="Successfully created the Babylon client" module=babylonclient
vigilante-reporter    | time="2022-09-08T04:25:16Z" level=info msg="Successfully generated TLS certificates for the RPC server" module=rpcserver
vigilante-reporter    | panic: rpc error: code = Unknown desc = unknown query path: unknown request
vigilante-reporter    |
vigilante-reporter    | goroutine 1 [running]:
vigilante-reporter    | github.com/babylonchain/vigilante/reporter.(*Reporter).Init(0x400014e780)
vigilante-reporter    |         /work/reporter/bootstrapping.go:23 +0x400
vigilante-reporter    | github.com/babylonchain/vigilante/cmd/reporter.cmdFunc(0x4000efe000?, {0x142caab?, 0x4?, 0x4?})
vigilante-reporter    |         /work/cmd/reporter/reporter.go:90 +0x24c
vigilante-reporter    | github.com/spf13/cobra.(*Command).execute(0x4000efe000, {0x4000e82880, 0x4, 0x4})
vigilante-reporter    |         /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x4d0
vigilante-reporter    | github.com/spf13/cobra.(*Command).ExecuteC(0x400029db80)
vigilante-reporter    |         /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x360
vigilante-reporter    | github.com/spf13/cobra.(*Command).Execute(...)
vigilante-reporter    |         /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
vigilante-reporter    | main.main()
vigilante-reporter    |         /work/cmd/main.go:22 +0x114

Still investigating the source of this, but I figured you people could take a look at the rest of the implementation while I do that or provide ideas as to why it might fail.

I would suggest that you try the make simnet-start command yourselves to check the stack. A future step would be to add the explorer to the stack as well.

aakoshh · 2022-09-08T22:55:13Z

The reasoning for this is that localnet-start should only be about testing the Cosmos codebase itself and not the entire Babylon stack.

It's difficult to say what localnet-start should be about. I thought that to test the Cosmos codebase you will need a BTC nodes to accept the checkpoints and the submitter/reporter to work as well, so a "local net" will need all of them. But if you found otherwise, for sure do whatever makes sense.

aakoshh

Looks nice! Have you found out the error?

README.md

contrib/images/bitcoinsim/Dockerfile

aakoshh · 2022-09-08T23:17:24Z

docker-compose-simnet.yaml

+      localnet:
+        ipv4_address: 192.168.10.6
+    volumes:
+      - ./.testnets/bitcoin:/bitcoinconf:Z


Why is :Z required?

The :Z flag denotes that the volume is shared between containers (see here`

vitsalis · 2022-09-09T01:51:18Z

@aakoshh Pushed a few new updates:

Remove the simnet-start command and do that in the localnet-start based on your comments. I agree that someone who wants to run a localnet, doesn't want to just run the Babylon nodes (which will do nothing by themselves) but wants to run the entire stack.
Added the explorer to the deployment.
Updated instructions to include the explorer.

The bug hasn't been resolved yet, will look for that right now.

vitsalis · 2022-09-09T03:53:28Z

Turns out the bug was related to the vigilante not using the latest version of Babylon. #114 changed the endpoints of the btccheckpoint module so the latest version was needed. Fixed that at the vigilante repo.

vigilante.yaml

docker-compose.yml

aakoshh · 2022-09-09T04:51:46Z

docker-compose.yml

+    restart: on-failure:10
+  vigilante-submitter:
+    container_name: vigilante-submitter
+    image: babylonchain/vigilante-submitter


The vigilante-submitter and vigilante-reporter are the same repo, same executables, no? Maybe one image with different command would be enough?

It would, but I believe it is better if we have those separate, since in the future they might be split up or work using different parameters etc.

aakoshh · 2022-09-09T04:55:15Z

Makefile

+build-bitcoinsim:
+	$(MAKE) -C contrib/images bitcoinsim


It's built automatically by docker-compose. Do we need a separate top level target?

Not necessary, but it can be useful imo.

docker-compose.yml

Makefile

aakoshh

Great! Must have been quite an effort to figure out all these cert files and align all the config 🙇

KonradStaniec · 2022-09-10T09:16:44Z

@vitsalis

tbh It would be nice to leave localnet-start without vigiliante and submitter just for the sake on integration tests. I planned to start without them.
For the future I would avoid merging pr-s with failing CI. If something is not easy fix and is really needed, I would prefer to disable failing check, and create separate task to fix it. Failing CI just breaks the flow for other people.

vitsalis · 2022-09-10T10:04:40Z

Oh I'm sorry about that, for some reason I was under the impression that the build was already failing. Added a fix on #125

KonradStaniec · 2022-09-10T10:10:11Z

Yea we have some flaky test for now i created issue for it: #124

docker: Add simnet command with vigilante

04f6420

vitsalis requested review from aakoshh and SebastianElvis September 8, 2022 13:30

aakoshh reviewed Sep 8, 2022

View reviewed changes

Updated instructions

aca5638

vitsalis changed the title ~~docker: Add simnet command with vigilante~~ docker: Include vigilantes and explorer in the localnet deployment Sep 9, 2022

Add note about explorer

dfc61b8

Update vigilante config

508a693

aakoshh reviewed Sep 9, 2022

View reviewed changes

vigilante.yaml Outdated Show resolved Hide resolved

aakoshh reviewed Sep 9, 2022

View reviewed changes

docker-compose.yml Show resolved Hide resolved

aakoshh reviewed Sep 9, 2022

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

aakoshh reviewed Sep 9, 2022

View reviewed changes

Makefile Outdated Show resolved Hide resolved

aakoshh approved these changes Sep 9, 2022

View reviewed changes

vitsalis and others added 5 commits September 9, 2022 14:20

updates

28726a1

Restrict bitcoinsim context to the minimum.

2a820b3

Ignore .testnets files in docker builds.

90573d1

Update cosmos version and testnet command

176dcce

Remove .testnets dir on localnet-stop

9293ed5

vitsalis merged commit 7454857 into main Sep 10, 2022

vitsalis deleted the vigilante-deployment branch September 10, 2022 03:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker: Include vigilantes and explorer in the localnet deployment #123

docker: Include vigilantes and explorer in the localnet deployment #123

vitsalis commented Sep 8, 2022

aakoshh commented Sep 8, 2022

aakoshh left a comment

aakoshh Sep 8, 2022

vitsalis Sep 9, 2022

vitsalis commented Sep 9, 2022

vitsalis commented Sep 9, 2022

aakoshh Sep 9, 2022

vitsalis Sep 9, 2022

aakoshh Sep 9, 2022

vitsalis Sep 9, 2022

aakoshh left a comment

KonradStaniec commented Sep 10, 2022

vitsalis commented Sep 10, 2022

KonradStaniec commented Sep 10, 2022

docker: Include vigilantes and explorer in the localnet deployment #123

docker: Include vigilantes and explorer in the localnet deployment #123

Conversation

vitsalis commented Sep 8, 2022

aakoshh commented Sep 8, 2022

aakoshh left a comment

Choose a reason for hiding this comment

aakoshh Sep 8, 2022

Choose a reason for hiding this comment

vitsalis Sep 9, 2022

Choose a reason for hiding this comment

vitsalis commented Sep 9, 2022

vitsalis commented Sep 9, 2022

aakoshh Sep 9, 2022

Choose a reason for hiding this comment

vitsalis Sep 9, 2022

Choose a reason for hiding this comment

aakoshh Sep 9, 2022

Choose a reason for hiding this comment

vitsalis Sep 9, 2022

Choose a reason for hiding this comment

aakoshh left a comment

Choose a reason for hiding this comment

KonradStaniec commented Sep 10, 2022

vitsalis commented Sep 10, 2022

KonradStaniec commented Sep 10, 2022