Adds custom annotations via env variables

[tylerpotts]

Implementation of the work described here: #1370

Adds the ability to add custom annotations to the kubernetes jobs via an environment variable

[tylerpotts]

@savingoyal or @shrinandj could you take a look when you have a moment?

[saikonen]

Took this for a spin and it is working as intended for the Kubernetes side. Also applies custom annotations first so does not make it possible to override system annotations (f.ex. metaflow/*) as expected 👍

Small question though, as the original discussion was focused on the safe-to-evict control annotation, but now we're introducing arbitrary annotations, do we want this to work with Argo Workflows as well? Currently there is no setup for the custom annotations for Argo Workflows, so that could still be added (similar to how labels are treated in argo_workflows.py).

[tylerpotts]

@saikonen Agreed that it makes sense to have argo included in this PR. I just added it in the recent commit and it's ready for your review

[nflx-mf-bot]

Testing[445] @ 86f1412

[tylerpotts]

@savingoyal Could you merge this when you have a moment?

[savingoyal]

approved! i will make this part of the next release.