Allow setting node taints and custom resource definitions

[leninmehedy]

No description provided.

[leninmehedy]

Currently it is possible for a user to set custom values using a values file solo network deploy --values-file values-deployment.yaml:

deployment:
  podAnnotations: {}
  podLabels: {}
  nodeSelector: {}
  tolerations:

So it is not super critical to have custom flags for all those options.

[leninmehedy]

Ok, found an issue with how tolerations are set currently and we need a better solution.

Problem

Currently tolerations are hardcoded at 8 places for various resources in the values.yaml file, see below:

• https://github.com/hashgraph/full-stack-testing/blob/main/charts/fullstack-deployment/values.yaml#L215
• https://github.com/hashgraph/full-stack-testing/blob/main/charts/fullstack-deployment/values.yaml#L254
• https://github.com/hashgraph/full-stack-testing/blob/main/charts/fullstack-deployment/values.yaml#L299
• https://github.com/hashgraph/full-stack-testing/blob/main/charts/fullstack-deployment/values.yaml#L323
• https://github.com/hashgraph/full-stack-testing/blob/main/charts/fullstack-deployment/values.yaml#L336
• https://github.com/hashgraph/full-stack-testing/blob/main/charts/fullstack-deployment/values.yaml#L349
• https://github.com/hashgraph/full-stack-testing/blob/main/charts/fullstack-deployment/values.yaml#L424
• https://github.com/hashgraph/full-stack-testing/blob/main/charts/fullstack-deployment/values.yaml#L389

Proposal

solo could offer custom flags as below for solo network deploy command:

• --tolerations
• --pod-annotations
• --pod-labels
• --node-selector

If these flags are set, solo may do either of the below to apply those values across the network resources:

1. Generate a new values file duplicating those values for all resource (8 types of resources as of now)
2. Add new values using --set

Warning: remember to apply these for all third party resources as currently hardcoded in the values.yaml so that all full-stack-network resources are scheduled on the nodes expected.

Complexity

For each of those flags, user would have to enter a complex YAML data structure. So we need to support adding Yaml in the console. However, this is not too crazy to have since other CLI tools like curl provides --data-binary flag for user to enter multi-line JSON data input like below

curl -0 -v -X POST http://www.example.com/api/users \
  -H "Expect:" \
  -H 'Content-Type: application/json; charset=utf-8' \
  --data-binary @- << EOF
{
  "field1": "test",
  "field2": {
    "foo": "bar"
  }
}

Solution options

1. Values file: User provide a custom values file (e.g. values-deployment.yaml) using the --values-file switch containing the deployment section that solo duplicates for all network resources (at 8 locations as of now) . In this case, solo needs to handle the deployment section in the provided values file differently and user can be relieved from typing multi-line yaml in the console.

2. "Have-it-all": Have both options (values file and command flags supporting multiline yaml)

[jeromy-cannon]

Another option is that the user could enter the toleration in json format. definitely would probably have to use --set-json when adding it to the command line option of the helm install call.

[leninmehedy]

We are going to follow gsutil pattern to set node-taints and labels etc: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/update#--node-taints

[leninmehedy]

• We need separate taints and labels for for mirror-node.
• We also need node affinity and anti-affinity, it could be specified with a file. However this should be least preferable option.