Investigate effective storage limits for completed runs

[imjasonh]

We know that storing details of completed Runs eventually results in too many stored resources/bytes and unresponsive behavior from etcd and the K8s API server. We don't have a good idea exactly how many resources/bytes it takes to start causing problems.

We should explore this on a standard GKE cluster and document (even if it's just in this issue) our findings about what symptoms we observed, roughly how many resources it took to see them, etc.

If anybody else has experienced this on their own clusters and could contribute data, even anecdata, that would be helpful.

Related #454
cc @wlynch

[wlynch]

tektoncd/experimental#479 is also related for a cronjob to clean up these completed resources.

[psschwei]

Anecdotally, I started seeing some minor sluggishness with about 1600 completed taskruns (this was doing an artificial test on minikube, where I just spammed it with new taskruns of the "hello world" task from the tutorial).

Going to spend some time over the next couple of days doing some more rigorous testing and documenting the results in this issue.

[psschwei]

/assign @psschwei

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[imjasonh]

/reopen
/lifecycle frozen

Excessive etcd storage is still an issue when results and pruning aren't configured.

We should run some tests to get a rough idea how many resources can be reliably stored in etcd storage at different resource levels, both as documentated guidance to operators and as a sales pitch for enabling results and/or pruning to avoid these issues.

While we're doing this, we should collect some symptoms of an overloaded cluster (what behavior does the cluster exhibit under excessive etcd load, what error messages can people google to find our docs)