Allow autoconf templates in docker labels

[xvello]

What does this PR do?

Mirroring the kubernetes annotation autodiscovery source, lookup autoconf template in container labels. These templates are expected in the JSON format.

If k8s annotation are present, they take precedence. Else, if valid json is found in the following labels:

• com.datadoghq.ad.check_names
• com.datadoghq.ad.init_configs
• com.datadoghq.ad.instances

Warning: the label names changed from the original PR, to reflect the s/ServiceDiscovery/AutoDiscovery/ transition

a template is rendered from them, bypassing the auto_conf files lookup.

If a K/V store backend is configured, this feature is disabled, like k8s annotations.

Testing Guidelines

• Added the test_get_check_tpls_labels test method. We could go further with integration testing.
• For manual testing, one can use the workbench recipe jmx:template_in_labels from add new template_in_labels jmx variant workbench-recipes#6 and the datadog/dev-dd-agent:jmx-xvello_template_in_labels image

[jhmartin]

The / character in the label name will not work with Amazon ECS Task Definitions, as it enforces the label format recommendations at https://docs.docker.com/engine/userguide/labels-custom-metadata/. Attempting to add such a label in ECS results in:

Docker label key service-discovery.datadoghq.com/check_names contains invalid characters, does not match pattern ^[_\-a-zA-Z0-9.]+$

The strict pattern according to the docker page would be^[a-z0-9-.]+$, and until recently was the pattern required by AWS.

Additionally the reverse-dns style recommendation implies it should becom.datadoghq.service-discovery as the prefix, so perhaps com.datadoghq.service-discovery.check-names and so on.

[xvello]

Good catch, I'm renaming the labels to be valid. Here is an example label section for a basic jmx check:

    labels:
      com.datadoghq.sd.check_names: '["jmx"]'
      com.datadoghq.sd.init_configs: '[{}]'
      com.datadoghq.sd.instances: '[{"host": "%%host%%", "port": "%%port%%"}]'

The test docker image is rebuilding as I post this, could you give it another try?

[jhmartin]

This appears to work.

My docker-compose.yml was:

version: "2"
services:
  tomcat:
    image: tomcat:8-alpine
    ports:
     - "8080:8080"
     - "8099:8099"
    environment:
     - JVM_HEAP=-Xmx512m -Xms512m
     - CATALINA_OPTS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
    labels:
      com.datadoghq.sd.check_names: "[\"jmx\"]"
      com.datadoghq.sd.init_configs: "[{}]"
      com.datadoghq.sd.instances: "[{\"host\": \"%%host%%\", \"port\": \"%%port_1%%\",  \"tags\" : { \"env\" : \"jm\", \"app\" : \"appnamehere\" }}]"
  datadog:
    image: datadog/dev-dd-agent:jmx-xvello_template_in_labels
    depends_on:
      - tomcat
    environment:
     - API_KEY
     - SD_JMX_ENABLE=yes
     - SD_BACKEND=docker
     - DD_HOSTNAME=jmtest
     - LOG_LEVEL=DEBUG
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /proc:/host/proc:ro
     - /sys/fs/cgroup:/host/sys/fs/cgroup:ro

I had to add escaping else compose complained about the YAML format.

Using this image and configuration I saw the JMX metrics appear in DataDog under app:appnamehere and env:jm.

A couple of things I noticed:
/etc/init.d/datadog-agent configtest isn't aware of this so shows no JMX metrics, same for /etc/init.d/datadog-agent jmx. It is not a fatal problem but does complicate troubleshooting.

[xvello]

Hi @jhmartin,

Thanks for your feedback. Happy to know this is working for you! As for escaping, if you surround your JSON with single quotes, you don't need to escape double quotes, please have a look at the example I posted

Indeed, JMX is kind of a special child for now as it runs in a separate java process, and does not appear in configtest. You can have a look at /opt/datadog-agent/run/jmx_status.yaml to make sure confs are indeed taken into account.

[hkaj]

kube_config --> docker_config same for the 2 lines above

[hkaj]

same here, you can call it config or whatever but it's not k8s specific anymore

[hkaj]

left nitpick comments and will give it a spin before we merge but it looks 👍

[xvello]

Hi @jhmartin,

I'm just merging that PR, so 5.17 will have support for templates as labels.
Please note we changed the label names to improve consistency.

Regard

[jhmartin]

@xvello Saw that, no problem.

[jhmartin]

@xvello
Please note that https://docs.docker.com/engine/userguide/labels-custom-metadata/#key-format-recommendations does say:

label key is the left-hand side of the key-value pair. Keys are alphanumeric strings which may contain periods (.) and hyphens (-).
...
These guidelines are not currently enforced and additional guidelines may apply to specific use cases.

It currently works, but relying on underscore in a label name may cause an issue in the future. Better to change it to -, or accept both.