Investigate the performance and time taken by the mass_deployments workflow

[mohamedamer453]

Description

Investigate the performance and how long it takes for the mass_deployments workflow to run.

The latest workflow run took almost 3 hours for 100 deployments. mainly because some of the failed deployments took ~10 minutes to timeout as seen in https://github.com/threefoldtech/tfgrid-sdk-ts/actions/runs/6727307503/job/18284925859#step:6:3325

[AlaaElattar]

Work in Progress (WIP):

• I got errors at first then opened this issue Farm endpoint gives bad gateway when randomize query is used with the other queries tfgrid-sdk-go#473
• I'm still investigating but with deleting randomization in both farms and nodes until the issue is solved

[AlaaElattar]

Work in Progress (WIP):

• I'm trying to start by running only 10 deployments and track their time.
• But I've got the same error for them all, Error getting node x, didn't respond after 10 seconds on dev.
• I've tried to run on qa but got same error for 10 deployments.
• That's all with removing randomize from both nodes and farms.

[AlaaElattar]

Work in Progress (WIP):

• I've increased timeout to 20 seconds.
• I got Error: Error getting node 2: Error: Failed to send request to twinId 9 with command: zos.statistics.get, payload: due to 102 code: 102 for all 10 deployments but without timeout part.

[AlaaElattar]

Work in Progress (WIP):

• Blocked till this issue is solved deployment failed with code 102 #1379

[xmonader]

Issue blocked on dysfunctional devnet

[AlaaElattar]

Investigation :

• If deployment created successfully will take avg 18 to 30 seconds. (If network is slow, the deployment will take 1 minute)
• If failed because the error of sending request (code 102) will take 10 seconds (timeout of request).
• If failed because Node 128 is not available for user with twinId: 214, maybe it's rented by another user or node is dedicated. use capacity planning with availableFor option. will take avg 3 to 4 seconds. (That's an issue needs to be handled here or from gridproxy)

[AlaaElattar]

Work Completed::

• Script updated to deploy batches of vms together.
• For example, 1000 vms will be deployed each 100 batch together. If one on the batch failed, the whole batch will be failed.
• This will decrease the time alot.

Investigation:

• There's something called matrix in github workflows that runs many instances of the same workflow at same time.
• I'm investigating how to use it.

[AlaaElattar]

• Now I'm blocked as the account running the script on mainnet contains 0 TFTs, I need it to test time of worflow not just test it locally.

[AlaaElattar]

Work In Progress (WIP)

• After running script on mainnet, I found that if the vm deployed successfully it will take avg 1 min which is still alot.
• There're 2 errors many times either the node didn't respont after 10 seconds or can't get wg_ports .
• I'm investigating these errors.

[AlaaElattar]

Work In Progress (WIP)

• I've updated that any offline node will be excluded from filterNodes.
• Also, I've found that the batch takes less 1 min locally.
• Still investigating any errors prevent batch from being deployed

[AlaaElattar]

Work Completed:

• I've updated pinging node to be parallel with using promise.all
• Also, we've found that filter nodes takes alot (40 sec), so we will apply it once and then choose random from list.
• The batch of 50 deployment takes about 10 min.

Work in progress (WIP)

• working on using matrices to run 4 instances of mass_deployment each one of them has 250 vm.
• each instance will use different account.

[AlaaElattar]

Work Completed:

• Creating deployments updated to be parallel also.
• Instead of using github matrix, we've decided to make mass deployment run 250 vm and schedule it every 6 hours.
• 50 vms take from 7 to 8 min.

[AlaaElattar]

Work In Progress (WIP):

• working on pr comments.

[AlaaElattar]

Work Completed:

• All pr comments applied.

[A-Harby]

Verified,

The mass deployment is now running in much less time than before, taking only about 20 minutes to run the full batch.

But the workflow sometimes passes and sometimes fails. I created a new issue for it here: #1699.