Optimizing deployment process (With the new release candidate of knuu) for large-scale network tests

[staheri14]

For the large-scale network test consisting of roughly 100 nodes, the current e2e tests require building an image for each node locally before deployment. This process is very time-consuming and slows down deployment as we need to build one image per node. knuu developers are already investigating this issue, and a new release v0.13.3.-rc.4 is on the way to make the process faster.

In an attempt to incorporate the first release candidate from that repository, a few errors have surfaced (for more details please see #3456). In this issue, I aim to investigate these issues, resolve them, and ultimately integrate the new release candidate v0.13.3.-rc.4 into our network tests to enable a faster deployment process.

[staheri14]

Cross posting from #3456
#3456 (comment)

Closes #3480

This is a test PR attempting to integrate the new release candidate v0.13.3-rc.4 of knuu and checking the e2e tests results against the updated version (primarily to use the more efficient deployment process by adding files and folders after creating the image). However, the following error occurs for the very first validator (just by merely bumping the knuu version without even swapping the order of AddFolder and Commit method calls):

Command to run:

go run ./test/e2e E2ESimple  -v 

go run ./test/e2e E2ESimple  -v 
test-e2e2024/05/09 13:56:44 === RUN E2ESimple
test-e2e2024/05/09 13:56:44 Running simple e2e test version 171a547
WARN[0000] The .env file does not exist, continuing without loading environment variables. 
INFO[2024-05-09T13:56:44-07:00]knuu/knuu.go:175 LOG_LEVEL: info                              
WARN[2024-05-09T13:56:44-07:00]knuu/knuu.go:77 KNUU_NAMESPACE is deprecated. Scope overridden to: test-sanaz 
INFO[2024-05-09T13:56:44-07:00]knuu/knuu.go:80 Initializing knuu with scope: test-sanaz     
test-e2e2024/05/09 13:56:54 Creating testnet validators
test-e2e2024/05/09 13:56:54 Creating account
{"level":"info","name":"alice","pk":"PubKeySecp256k1{02A80BAE7BD0A69597FE3CA95073EC49AE381564C68DB8F2E8456AE0550AEE5A0B}","time":"2024-05-09T13:56:54-07:00","message":"txsim account created and added to genesis"}
test-e2e2024/05/09 13:56:54 Setting up testnets
{"level":"info","name":"val0","directory":"/var/folders/gl/8f7g64t91rxc6hsz_gj0mqfh0000gn/T/val0","time":"2024-05-09T13:56:57-07:00","message":"Creating validator's config and data directories"}
{"level":"info","name":"val1","directory":"/var/folders/gl/8f7g64t91rxc6hsz_gj0mqfh0000gn/T/val1","time":"2024-05-09T13:57:17-07:00","message":"Creating validator's config and data directories"}
{"level":"info","name":"val2","directory":"/var/folders/gl/8f7g64t91rxc6hsz_gj0mqfh0000gn/T/val2","time":"2024-05-09T13:57:34-07:00","message":"Creating validator's config and data directories"}
{"level":"info","name":"val3","directory":"/var/folders/gl/8f7g64t91rxc6hsz_gj0mqfh0000gn/T/val3","time":"2024-05-09T13:57:51-07:00","message":"Creating validator's config and data directories"}
test-e2e2024/05/09 13:58:08 Starting testnets
2024/05/09 13:59:09 failed to start testnets: node val0 failed to start: error waiting for instance 'val0-64b6f7c2' to be running: timeout while waiting for instance 'val0-64b6f7c2' to be running
exit status 1

Logs from val0-64b6f7c2:

2024-05-09 14:01:28.603	
cp: can't create directory '/knuu/home/celestia/.celestia-app/data': No such file or directory
2024-05-09 14:01:28.603	
cp: can't create directory '/knuu/home/celestia/.celestia-app/config': No such file or directory
2024-05-09 13:59:56.545	
cp: can't create directory '/knuu/home/celestia/.celestia-app/data': No such file or directory
2024-05-09 13:59:56.545	
cp: can't create directory '/knuu/home/celestia/.celestia-app/config': No such file or directory
2024-05-09 13:59:06.527	
cp: can't create directory '/knuu/home/celestia/.celestia-app/data': No such file or directory
2024-05-09 13:59:06.527	
cp: can't create directory '/knuu/home/celestia/.celestia-app/config': No such file or directory
2024-05-09 13:58:40.556	
cp: can't create directory '/knuu/home/celestia/.celestia-app/data': No such file or directory



2024-05-09 13:58:40.556	
cp: can't create directory '/knuu/home/celestia/.celestia-app/config': No such file or directory
2024-05-09 13:58:23.305	
cp: can't create directory '/knuu/home/celestia/.celestia-app/data': No such file or directory
2024-05-09 13:58:23.305	
cp: can't create directory '/knuu/home/celestia/.celestia-app/config': No such file or directory

After investigating with @jrmanes, it has been established that the fix for this issue is to ensure calls to Commit always precede calls to AddFolder.

[staheri14]

There is another issue under the radar (resulting from the rc.5) and that is the port forwarding started failing. It happens in the invocation of PortForwardTCP func, with the following error message forwarding port 26657: error forwarding port after 5 retries.
Replicating the issue:

• Fetch the branch in test: bumps knuu version to v0.13.3-rc.5 #3456
• Run go run ./test/e2e E2ESimple -v

[tty47]

hello @staheri14

I have good news, I was able to run your tests with the latest version: v0.13.3, please try to run the tests and let us know if you find any issues

thanks @smuu !