First implementation of dynamic query support

[IzabellaRaulin]

Related to #676

Summary of changes:

• adding dynamic query support (wildcard and tuple)
• adding validation of metric's namespace:
  • (part 1) not allowed characters, especially that ones which can be invalid interpreted as regexp tokens
  • (part 2) a wildcard in the end of namespace - that has been implemented yet - added after revising of all existing collector plugins; only one collector had such like metrics and was refactored.

Testing done:

• checking a query support for plugin with no-dynamic metrics
• checking a query support for plugin with dynamic metrics (if plugin accepts wildcards in this location the framework does not attempt to evaluate that)

@intelsdi-x/snap-maintainers

[geauxvirtual]

These comments should be removed as the below lines are not commented out, nor should the lines be commented out.

[IzabellaRaulin]

Thanks, I decided to remove this commented out code. When I finish revising that all plugins do not exposed a wildcard in the end of namespace + refactoring that one which has such like metrics, I will add this part of validation + unit test which covers that again in another pull request.

Already even plugin has metrics which ends with wildcard, it will works properly and can be used by users before refactoring.

[jcooklin]

You can reuse metricType (defined here in metrics.Go) instead of creating this definition.

[IzabellaRaulin]

Thanks, done

[IzabellaRaulin]

Code after suggested changes. Feel free to comment this PR

[jcooklin]

Loading mock1 and running the following task fails.

---
  version: 1
  schedule:
    type: "simple"
    interval: "1s"
  workflow:
    collect:
      metrics:
        /intel/*: {}
      config:
        /intel/mock:
          user: "root"
          password: "secret"

To fix this line should be config: wf.configTree.Get(ns),.

[IzabellaRaulin]

Yes, now it only works for "/intel/mock/*" and for others variations of queries (e.g like /intel/mock/(foo|bar)), but they have to start with "/intel/mock" to get this config.

I looked on that and I suppose that turning m.Namespace() into ns is not the only thing which needs to been done in this case. Look on scheduler/job.go#L221 - I do not refactor this file (generally my approach was changing as little as possible), but might this should be changed too - according that, 2 things have to be refactored:

• in scheduler/job.go func (c *collectorJob) Run()
• in control/control.go func (p *pluginControl) CollectMetrics(..)

@jcooklin - what do you think about it?

[IzabellaRaulin]

After refactoring, for metric "/intel/*" in task manifest, the config section is retrieved properly

[candysmurf]

nice work! Do we have a doc that outlines the current behavior and the differences this enhancement may bring?

[IzabellaRaulin]

@candysmurf - already there is only that: docs/TASKS.md

But did you mean a doc for this pull request or generally more detailed section in snap/doc?

@IzabellaRaulin - Thanks for the pointer. It's a very nice doc. It's clear that what are supported. I'm confused that I was able to use wildcard feature already in my plugin development prior to this enhancement. What new things this introduces. If you can elaborate more, that would help. Or I will catch you for my doubts.

[IzabellaRaulin]

Code after suggested changes by @jcooklin - after refactoring, for metric "/intel/*" in task manifest, the config section is retrieved properly. This fix required (ndirectly) the separation of ExpandWildcard() to two functions, which are:

• ExpandWildcards([]string ns) ([][]string, serror.SnapError) -> now it only retrieves the matched metrics namespaces from metricCatalog.mKey, does NOT process the matching itself
• MatchQueryToNamespaces([]string ns) ([][]string, serror.SnapError) -> it performs the matching process. It is called in scheduler/scheduler.go, in gatherMetricsAndPlugins() when task is creating and metrics are retrieving from task manifest (they are matched with these ones in the catalog). The matching results are held in metricCatalog.mKey as before.

BEFORE there was only ExpandWildcards() which performed the matching process when there were not any matched results for this 'metric to collect' which is taken from task manifest.

[jcooklin]

LGTM. 🚢 it.

[IzabellaRaulin]

I've just added labels to show that this pull request is a good example of using that ones.