docs: Add some more info to the Scaling Doc

[juliev0]

Fixes: #10566

Motivation

Want to provide more information to those who are trying to scale Argo Workflows.

[terrytangyuan]

Burst may not work as expected. See #8576

[juliev0]

So, that's actually a different "Burst". That's for the RateLimiter configured in the workflow-controller-configmap controlling the Pod creation rate.

This setting just gets passed directly into the Kubernetes Client to control the rate of outgoing K8S API requests.

[agilgur5]

Great additions! Tuning has a good amount of nuance to it, so I think guidance here could be very helpful!
Completes the docs request in #11457 (comment) as well 🙂

Mostly made copy-edits below

[agilgur5]

We may want to list the defaults for these flags here as well so that there is a comparison point for when they are not explicitly set

[agilgur5]

LGTM, thanks for the copy edits!

Listing defaults would be good, but that can be done as a follow-up, this is already a solid improvement as is

[juliev0]

We may want to list the defaults for these flags here as well so that there is a comparison point for when they are not explicitly set

I suppose I was reluctant to do that since the values may change and we may forget to update the documentation, although admittedly I did say that the defaults were low, which could potentially change as well.

[agilgur5]

I suppose I was reluctant to do that since the values may change and we may forget to update the documentation

We don't change defaults that often though. Right now I think it's more likely that users have no idea what the defaults are 😬

In #11457 (comment) I also mentioned that flag options would probably be a good use-case for docgen. So maybe we consider docgen as future state for these but may still want something in the interim as docgen is non-trivial.