Add TTL to garbage collect old workflows

[jlewi]

FEATURE REQUEST

A general problem in K8s is that batch jobs can accumulate overwhelming the master.
Kubeflow has experienced this with Argo workflows (as well as other resources kubeflow/testing#87).

There is a proposal in K8s to solve this by adding a TTL field to K8s jobs and implementing a TTL controller that could eventually be used with other resources.

kubernetes/kubernetes#64470
Design doc: https://goo.gl/YxtxTi.

I think Argo workflows should have a similar field and eventually support garbage collecting old workflows using the TTL controller.

[jessesuen]

Thanks for pointing this issue. Similar to the ability to do a cascaded deletion via owner references, this will actually save a lot of unnecessary work in the workflow-controller if there is an upcoming mechanism to GC old workflows.

[jessesuen]

Looking at the proposal a bit closer, it's going to be a long while before the TTL support CRDs (if ever). It'll be better to support this earlier in our types/controller, and remove the logic if & when the TTL controller supports custom resources.