[auto_conf] Allow multiple instances per check

[olivielpeau]

What does this PR do?

Allows configuring multiple instances of the same check with auto_conf.

Motivation

In some cases (and for some checks), it makes sense to have multiple
instances of the same check running against a container. Example:
php_fpm check when the user wants to run the check against multiple status_urls.

Testing Guidelines

I did some basic manual testing. Haven't written unit tests but if needed I can add some.

Additional Notes

Opening this PR after discussing with @sdwr98 on a use case for #3259 (the use case being: having the ability to define multiple instances of the same check with auto_conf, which is useful for php_fpm for example)

[aerostitch]

Could this also affect service discovery? I'm currently experiencing a similar issue with service discovery / kubernetes annotations taking only the 1st instance defined in the annotations for a check.

[olivielpeau]

@aerostitch This issue shouldn't be affecting kubernetes annotations (or, at least, what the present PR changes should have no effect on the way kubernetes annotations are pulled by the agent).

Feel free to contact our support team at support@datadoghq.com and send them a flare if you're having issues setting up kubernetes annotations for the Agent, they'll be able to ask for more specific information about your setup.

[aerostitch]

I've already done that, I was just checking, as the behavior seems very similar, if that was related.
And the fact that the get_check_tpls function was skipping to autoconf for the annotations here:

        # it makes the method skip directly to the auto_conf
        if kwargs.get('auto_conf') is True:
            # When not using a configuration store on kubernetes, check the pod
            # annotations for configs before falling back to autoconf.
            kube_annotations = kwargs.get(KUBE_ANNOTATIONS)
            kube_container_name = kwargs.get(KUBE_CONTAINER_NAME)

made me think that this could be what I was looking for as, when I use the info command, I get this message:
2017-05-10 20:35:05,125 | WARNING | dd.collector | utils.service_discovery.config(config.py:31) | No configuration backend provided for service discovery. Only auto config templates will be used.

I'm actually not sure if the issue is when the config is pulled from Kubernetes or when it's pushed in the datadog config.

[olivielpeau]

Oh okay, hmm the comments on top of the snippet you copied are a bit misleading, but for what this PR fixes the issue lies in the function that's called here: https://github.com/DataDog/dd-agent/blob/5.13.0/utils/service_discovery/abstract_config_store.py#L287 , so it's definitely unrelated with your issue.

[aerostitch]

Thanks for the quick answer.
I'll try to narrow down exactly to where my issue is in the code while talking to Stephen from the support.

[aerostitch]

Just a note: once #3341 is merged you could use the image_3 and image_4 examples I added to the tests for this. Should be pretty straightforward.

[xvello]

LGTM, we could indeed extend the tests for that though, see previous comment

[hkaj]

LGTM, fits well with #3341 (will also require some documentation)