Variables substitution doesn’t work from keystore in username/password fields of metricbeat.yml in an autodiscover section

[nevmerzhitsky]

How can I use keystore variables in autodiscover.providers.templates.config section of metricbeat.yml? It looks like broken.

• Version: Metricbeat 7.1.0
• Operating System: containers from https://www.docker.elastic.co/
• Discuss Forum URL: https://discuss.elastic.co/t/variables-substitution-doesnt-work-from-keystore-in-username-password-fields-of-metricbeat-yml-in-autodiscover-providers-templates-config-section/186125

Steps to Reproduce

1. Setup Docker for running next services
2. Setup Elasticsearch and configure Metricbeat to connect to it
3. Run a RabbitMQ service with an username and a password for authorization
4. Add QUEUE_USERNAME and QUEUE_PASSWORD to Metricbeat keystore via metricbeat keystore add with appropriate values
5. Add a block like this to metricbeat.yml:

  autodiscover:
    providers:
      - type: docker
        templates:
          - condition:
              # You can use any other way to match the service here
              equals:
                docker.container.labels.com.docker.swarm.service.name: my_rabbitmq_service
            config:
              - module: rabbitmq
                period: 10s
                hosts: ['${data.host}:${data.port}']
                username: ${QUEUE_USERNAME}
                password: ${QUEUE_PASSWORD}

6. Start the beat

Expected:
Metricbeat successfully connected to RabbitMQ service.

But it doesn't.

Interesting additional info:

1. For debugging purposes, you can use "${QUEUE_PASSWORD}" as a value for field output.elasticsearch.hosts of metricbeat.yml and you will see the finish value of QUEUE_PASSWORD variable from the keystore in logs of the beat (in error messages about the connection to Elasticsearch). Therefore the keystore really store the correct value.

2. If you replace ${QUEUE_USERNAME} and ${QUEUE_PASSWORD} by hard-coded values in metricbeat.yml and restart the beat it will work. Therefore Metricbeat can connect to RabbitMQ with required auth and username/password fields of the metricbeat.yml really works.

3. Even if you return the vars back to metricbeat.yml and add 2 Linux env vars QUEUE_USERNAME and QUEUE_PASSWORD and restart the beat it will work too. Therefore environ variables substitution work in the autodiscover.providers.templates.config section. But keystore variables substitution doesn't work.

[exekias]

Thank you for opening this! @ph I'm guessing the problem is happening here: https://github.com/elastic/beats/blob/2c70a2b3ee480e3d2e57da84130213513e4636d5/libbeat/autodiscover/template/config.go, any ideas?

[ChrsMark]

Hi @nevmerzhitsky! As we found with @exekias when looking into sth relevant Keystore is not supported with Autodiscover right now.

This is expected because this could be a security issue when Autodiscover is based on hints. Imagine that you deploy a pod with autodiscover hints that would have access to all the keystore secrets of Metricbeat. However this should be allowed with basic autodiscover where Metricbeat configuration is static and provided by the user that handles keystore as well.

[nevmerzhitsky]

Hi @nevmerzhitsky! As we found with @exekias when looking into sth relevant Keystore is not supported with Autodiscover right now.

This is expected because this could be a security issue when Autodiscover is based on hints. Imagine that you deploy a pod with autodiscover hints that would have access to all the keystore secrets of Metricbeat. However this should be allowed with basic autodiscover where Metricbeat configuration is static and provided by the user that handles keystore as well.

Yeah, I agree with you in security aspects. I'm personally required for support in the case of static configuration.

[exekias]

We should consider adding support for this, always for static configs (not hints based autodiscover). I don't see any harm on doing that. We are currently working on #8847, let's make sure the solution we find can fit this in the future