Bootstrap does not support arbitrary passwords/tokens + Overwrite behavior not configurable

[stealthybox]

So I tried a few things with the GitLab provider based off this Slack thread: https://cloud-native.slack.com/archives/CLAJ40HV3/p1616777615167100

@springroll12: In the flux terraform provider, is it possible to split up the "install" secret from the "sync" secret? For example we would like to have the flux sync secret have read-only privileges, where the flux clearly needs "write" permissions to write to the repo initially.

@phillebaba and I had a theory that maybe we could use the idempotent design of bootstrap to pre-create the token-secret.

I made two separate PAT's: flux-bootstrap & flux-readonly

I was able to create a proper secret in the cluster before the bootstrap like so:

k create ns flux-system --dry-run=client -oyaml | k apply -f-
k create secret generic flux-system -n flux-system \
  --from-literal=username=flux-readonly \
  --from-file=password=./flux-readonly.pat \
  --dry-run=client -oyaml | k apply -f-

Unfortunately, running bootstrap without --token-auth sets the GitRepository URL protocol to SSH, so while it attempts to use the pre-existing Secret credentials, the clone fails with:

auth error: invalid 'flux-system' secret data: required fields 'identity' and 'known_hosts'

I was hopeful that using --token-auth would fix the protocol, so I created a new cluster and tried the same steps.
Unfortunately, --token-auth overwrites the Secret instead of leaving it alone like the normal SSH bootstrap.
( this behavior makes sense when you're trying to switch protocols from SSH DeployKeys to HTTPS tokens on a live-cluster )

Currently, the only option for using a different token or username/password currently is to patch the Secret after the bootstrap. This works but is undesirable for good reason:

springroll12: Honestly I wouldn't want the bootstrap credentials to be in k8s at all. Is there no way to feed it to the flux source controller directly (e.g. as a cli parameter)? Sure it's easy enough to patch a secret but in my view it really doesn't need to be "stored" at all.

While @hiddeco is working through bootstrap, we should keep in mind that the difficulty in selecting the protocol and override behavior here has some consequences in how bootstrap interacts with other systems.

Considerations for several related use-cases that we've heard from before:

• Bring your own SSH Key (CLI flag / terraform option)
• Bring your own password (CLI flag / terraform option)
• Pre-existing HTTPS/token secret in cluster
• Pre-existing SSH secret in cluster
• Flag to force protocol (in-cluster Secret can already become ambiguous when you bootstrap with SSH and switch to --token-auth)
• Flag to enable/disable overwrite behavior

Additional use-case:
GitLab supports Deploy Tokens which are a per-repo HTTPS token auth method similar to Deploy Keys.
We may also consider how that fits into the bootstrap flow.

[hiddeco]

I think this issue should be non-blocking for #968 as everything introduced there is in internal (because we have some other - not yet documented - changes in mind to ease making adjustments to the manifests), and the current state of the PR is already a gigantic step forward to better UX.

[stefanprodan]

GitLab Deploy Tokens were implemented long time ago.