Is there an alternative to env. variables object substitution?

[orhantoy]

As of version 7.7 the ability to substitute objects into the config via env. variables were removed. We're currently relying on that ability in Enterprise Search, for instance to set custom ES headers:

output.elasticsearch:
  # ...
  hosts: ["${ELASTICSEARCH_HOST}"]
  # ...
  headers: ${ELASTICSEARCH_HEADERS}

elastic/go-ucfg#139 is the PR that removed support for this, I think.

Is there an alternative to this other than setting a command line argument like

-E output.elasticsearch.headers='{ "X-Custom-Header": "..." }'

?

[blakerouse]

Have you confirmed this doesn't work?

That change did not prevent the replacement from environment variables it removed the ability for double substitution. A variable that relied on another variable.

[orhantoy]

Just tried with a fresh copy of Filebeat 7.7.1 and it exited with an error:

$ ELASTICSEARCH_HEADERS='{"X-Custom-Header":"..."}' ./filebeat -e -v
[...]
2020-06-18T16:13:31.558+0200    INFO    instance/beat.go:411    filebeat stopped.
2020-06-18T16:13:31.558+0200    ERROR   instance/beat.go:932    Exiting: error initializing publisher: required 'object', but found 'string' in field 'output.elasticsearch.headers' (source:'filebeat.yml')
Exiting: error initializing publisher: required 'object', but found 'string' in field 'output.elasticsearch.headers' (source:'filebeat.yml')

The same command works on 7.6.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[acrewdson]

Can confirm this is a problem in 7.8 too – affects both the headers and ssl config fields. I think this might be related to the changes here:

The module was accepting configuration as JSON strings for some parameters (http_headers, http_request_body, pagination, rate_limit, ssl) which is inconsistent with how other parts of Beats are configured so I removed this. Now these options expect regular YAML objects for values. None of these options are required to use the module so the impact to users should be minimal.