Allow environment variables to have a default value

[jrcamp]

Is your feature request related to a problem? Please describe.

When using environmental variables you may want to have a default that is used if a user does not override it. For example:

config:
  endpoint: ${ENDPOINT:-localhost:8080}

Describe the solution you'd like
Allow bash-like variable substitution.

Describe alternatives you've considered
Not allowing it and require all defaults to be set external to the process. The downside is that the config and defaults now live in two separate places.

[jcchavezs]

I can work on this.

[bogdandrutu]

@jcchavezs please wait couple of days until we publish a document about the syntax we want for this.

/cc @tigrannajaryan @pjanotti

[jrcamp]

Will be addressed by #2469 with general default syntax.

[ttomsu]

Any progress made here? We've got some env vars that have (multiple) fallbacks, so these default values would be really nice.

[jrcamp]

It is part of the configsource in Splunk distribution but I'm not sure if there's a timeline of when it will be upstreamed https://github.com/signalfx/splunk-otel-collector/tree/main/internal/configsource/envvarconfigsource @pjanotti?

[ringerc]

A workaround for the lack of this capability in k8s based deployments is to use a layered configmap reference.

In a podTemplate container spec such as the .spec.template.spec.containers[] of a Deployment or DaemonSet, add e.g.:

envFrom:
  - configMapRef:
      name: my-collector-defaults
  - configMapRef:
      name: my-collector-overrides
      optional: true

Deploy my-collector-defaults with your component's usual manifests bundle.

Create my-collector-overrides with keys you wish to override default environment values for. The last-found value will be the one that takes effect. Be warned that values in env: override envFrom:.

The same thing works with env e.g.

          - name: DEBUG_EXPORTER_VERBOSITY
            valueFrom: 
              configMapKeyRef:
                name: otel-node-agent-settings-default
                key: DEBUG_EXPORTER_VERBOSITY
          - name: DEBUG_EXPORTER_VERBOSITY 
            valueFrom:
              configMapKeyRef:
                name: otel-node-agent-settings-override
                key: DEBUG_EXPORTER_VERBOSITY
                optional: true

If you use flux for your manifest deployments, you can annotate your overrides configmap with e.g.

metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/prune: disabled
    kustomize.toolkit.fluxcd.io/reconcile: disabled

to ensure flux won't try to delete it.

(You can avoid having the defaults configmap by defining env literals in the app manifest instead but I dislike doing so. The defaults and overrides pattern makes it a bit more discoverable.)