Change schema of hub.extraEnv to allow use of valueFrom

[mattjbray]

This was discussed back in #228, but was not done because of difficulties with toYaml. It looks like indent hadn't been discovered then.

Motivation: I'm adding tracing to the hub component, and I need to use valueFrom to access the downward API.

[mattjbray]

Note: if we need to maintain backwards-compatibility, I'd happily create a new config setting that can co-exist with the existing hub.envVar.

[yuvipanda]

Thanks for the patch, @mattjbray! I would like us to be backwards compatible - we've done this for extraConfig with some type introspection in gotmpl. Do you think you can do something similar for this too?

[mattjbray]

@yuvipanda Good idea - I've updated the PR.

[minrk]

Nice! Does it make sense to do the same for singleuser.extraEnv? Or does the fact that that gets passed through jupyterhub config make that not work?

[yuvipanda]

Thanks for the PR, @mattjbray! Congratulations on getting your first PR here merged! \o/ Hope to see more :)

@minrk yeah, we need to fix that up on the kubespawner side I believe.

[yuvipanda]

@mattjbray can you also tell us a little more about the tracing you are doing? I'm very curious!

[mattjbray]

@yuvipanda thanks!

Sure, at the moment we're using DataDog, a proprietary service.

This DataDog helm chart adds a DaemonSet that runs the DataDog agent on each node. The agent collects metrics about the host (including pods running, etc.), and also provides a collector service much like Zipkin, to which other applications running on the host can send their own metrics.

DataDog provide automatic instrumentation for Python applications (e.g. the tornado server for hub). You just have to wrap the program with their ddtrace executable. To do that we're customizing the hub Dockerfile:

FROM jupyterhub/k8s-hub:4b122ad

USER root

# Install ddtrace and create a script that wraps jupyterhub with ddtrace-run.
RUN pip3 install 'ddtrace==0.11.0'
RUN mv /usr/local/bin/jupyterhub /usr/local/bin/jupyterhub-orig
COPY scripts/jupyterhub /usr/local/bin/jupyterhub

USER jovyan

Where scripts/jupyterhub is simply:

#!/bin/bash

# Wrap with datadog tracer
ddtrace-run /usr/local/bin/jupyterhub-orig "$@"

Then in our values.yaml we tell ddtrace running in the hub deployment how to connect to the collector. This is where extraEnv comes in:

jupyterhub:
  hub:
    extraEnv:
      - name: "DATADOG_TRACE_AGENT_HOSTNAME"
        valueFrom:
          fieldRef:
            fieldPath: status.hostIP
      - name: "DATADOG_TRACE_AGENT_PORT"
        value: "8126"
      - name: "DATADOG_PATCH_MODULES"
        value: "tornado:true,sqlite:true"

datadog:
  datadog:
    apmEnabled: true
  daemonset:
    useHostPort: true

[manics]

I thought I'd mention a use case for keeping the backwards compatibility in future too. If extraEnv is a dict you can easily merge a public config in a git repo with private variables set on the command line during, e.g.

hub:
  extraEnv:
    OAUTH2_AUTHORIZE_URL: "https://login.elixir-czech.org/oidc/authorize"
    OAUTH2_TOKEN_URL: "https://login.elixir-czech.org/oidc/token"
    OAUTH_CALLBACK_URL: https://example.org/hub/oauth_callback
    OAUTH_CLIENT_ID: xxx

and set the private variables separately during deployment: --set hub.extraEnv.OAUTH_CLIENT_SECRET=XXX, but to my knowledge you can't do this when extraEnv is a list.

[consideRatio]

Ah! @manics this was useful knowledge for me!