[enhancement] configure prefect agent labels and environment variables

[jhamman]

Description

I would like to set labels and environment variables on my prefect agents. Is this possible today? Or does this require a new feature in qhub?

Suggestion

Perhaps the prefect configuration section could look something like this:

prefect:
  enabled: true
  labels: ["dev", "staging"]
  env_vars: {"KEY": "VALUE", "KEY2": "VALUE2"}

Context

Our prefect cloud account has multiple agents attached to it. Exposing labels and env_vars via the qhub config would allow us to preferentially direct flows to the correct agent.

[Adam-D-Lewis]

Thanks for opening this. There isn't a way do this currently, however, we would welcome a PR.

If you did decide to work on this then I'd suggest that we modify the helm chart currently being used to deploy Prefect to be closer to the one used in the official Prefect Server helm chart (https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/agent/deployment.yaml). I see that they do have a way to include arbitrary prefect labels, but not arbitrary environment variables. We could add the arbitrary environment variables with the range action in helm.

The rest of the changes are fairly minor modifications to qhub. Would you be interested in contributing a PR, @jhamman?

[jhamman]

@Adam-D-Lewis - see #768 for an initial attempt.

[dharhas]

@leej3 @trallard @kcpevey have been looking at changing how QHub and Prefect are deployed together. Tagging for awareness.

[leej3]

Some thoughts here

[trallard]

I see that they do have a way to include arbitrary prefect labels, but not arbitrary environment variables. We could add the arbitrary environment variables with the range action in helm.

Just adding a bit more context here - the Prefect Helm chart does not have specific fields for env vars but one can define them at the job level as per https://docs.prefect.io/orchestration/agents/kubernetes.html#flow-configuration

is this idea of having env-vars at an Agent level @jhamman to allow for all your flows within a specific agent to access them rather than explicitly pass them on a per-flow basis?

[Adam-D-Lewis]

We have a working version now merged in from #813 which allows you to define arbitrary environment variables on the prefect agent and add labels to the prefect agent. Closing this issue since this is now resolved.