queries for multiple tags

[carlivar]

We would like the ability to query multiple tags via DNS and HTTP.

An example of this would be using environment name as a tag, but also needing additional tag functionality such as master/slave. For example to get the master mysql server in the qa environment, we'd like the service to publish both those tags and allow a DNS query such as:

dig @127.0.0.1 -p 8600 master.qa.mysql.service.consul. ANY

Or for HTTP:

curl 'http://localhost:8500/v1/catalog/service/mysql?tag=master&tag=qa

Or whatever HTTP param syntax is sensible (I suppose multiple params named the same would not work. Perhaps an array syntax is needed).

For DNS, I'm not sure how easy it would be to support multiple tags in the format I suggested, so a delimiter could also work. Perhaps master_qa.mysql.service.consul using underscore as a delimiter. Although this would remove the ability to put underscore in tags.

This is somewhat related to #290 since it solves the multiple environments problem for those that are willing to use tags. After much internal discussion, we are.

[mfischer-zd]

+1 -- tag intersections should be supported, at least for the HTTP API. DNS is another matter: not really sure how one could implement it given its hierarchical design.

[ecweaver]

+1 ... if multiple tag= args could form an implicit AND / intersection, that would help a lot.

[broberts-zd]

+1 - this would be very useful.

[grosser]

👍 having multiple hosts + different services -> we need different tags

for dns maybe double-underscore as delimiter

[grobie]

👍

[babbottscott]

👍 Though simple and/not/or/grouping syntax would quickly become desirable. For DNS could it be as simple as
dig ... '(foo&bar)|!baz).mysvc.service.consul.'

[mfischer-zd]

Those aren't legal characters in DNS, @babbottscott

[babbottscott]

Not for hostnames, but for queries, this should be fine.

[mfischer-zd]

Many forwarding resolvers will return errors before the queries ever get to Consul's DNS server. So it's not really practical.

[armon]

I can certainly see it working in HTTP, DNS will require more thought to avoid breaking backwards compatibility

[mphoratiu]

👍 In most architectures having the possibility to query based on more than one tag would be a great feature.

[hanshasselberg]

isn't possible to use dots as tag delimiter in DNS?

[armon]

@i0rek The problem is a tag could be "foo.bar", which is hard to disambiguate from "foo" and "bar" or just "foo.bar".

[hanshasselberg]

@armon it already is a problem, no? service "foo.bar" VS service "bar" and tag "foo".

[armon]

Yes, currently any additional dots are considered to be part of the tag. So if your service name contains a period you cannot resolve it via DNS.

[mfischer-zd]

We should probably forbid that.. (separate issue though)

[hanshasselberg]

I think if dots are forbidden in service names the same should apply to tag names. Then we can have dots as tag delimiter and the lookup should be easy too.

[wolfspyre]

+1 for consistency amongst reserved characters. Since DNS is one of the primary APIs for consul it seems reasonable to reserve . as a delimiter across the board.

[xakraz]

👍

[mdhelgen]

👍

[mfischer-zd]

Any idea when the HTTP tag intersection behavior might be released? I'm sort of surprised that it didn't end up in 0.5.0.

[armon]

No idea currently. Part of the issue is simultaneously migrating tags from a flag list of strings into a richer K/V representation. That complicates the syntax as it's no longer flat.

[breedx]

+1 for multiple tags

[adubkov]

'http://localhost:8500/v1/catalog/service/mysql?tag=[master,qa]

why not just add something like that? no need make tags overcomplicated, they need just for filtering. And not have ability to filter by multiple parameters that's... i was very frustrated.

[mfischer-zd]

@blacked, because that's not how one typically specifies multiple values for a named parameter in HTTP request parameters. The idiomatic form is ?tag=master&tag=qa.

[sfncook]

👍

[rodrigopr]

+1 for multiple tags

[ybogdanov]

any progress on this?

[ghost]

+1 for multiple tags search over DNS with '.' prohobited in tag

[loicalbertin]

👍 with dns search support.

[ninthnails]

+1

[calvn]

+1 for this feature. It would be nice for multi-environment teams that are using/hosting the same services.

[nickwales]

Related to this topic, I was trying to provide some extra DNS information without using tags. So I can create a service called test.app which the API lets me query but not DNS. I can't see any specific documentation for what the name of the service should include but it appears periods are not supported for DNS resolution.

e.g.
curl consul/v1/catalog/service/test.app <-- works
dig @consul -p8600 test.app.service.datacenter.consul <-- doesn't return anything

[endzyme]

+1 for the multi-tag filtering

[isegal]

+1 yes this would be extremely useful. For example, a mysql sever could be staging or production, but it could also be master or slave. Allowing multi-tag filtering would keep the number of service name definitions down to a minimum.
A potential disadvantage would be the order of tags, from DNS perspective production.master.mysql.service.consul and master.production.mysql.service.consul are supposed to be separate hosts, but for service discovery it would be the same. A small price to pay I presume.

[rgarver]

👍 for this. Seems like the DNS complications could be managed with a canonical ordering of tags (eg: alphabetical) and CNAME non-canonical versions to the canonical version. That's if we really care that production.primary.mysql.service.consul and primary.production.mysql.service.consul are the same. Personally I think that's a non-issue that only affects DNS caching and frankly not in anyway that seems important given the context.

[slackpad]

Prepared queries (released in Consul 0.6.0) provides a solution for this - https://www.consul.io/docs/agent/http/query.html.

You can register a query that filters with multiple tags (in addition to several more features) and execute that query via DNS or HTTP request.

[chatu]

👍

[slackpad]

Closing this out now that prepared queries are available and allow queries using multiple tags via DNS and HTTP.

[rogeralsing]

Is this really a viable replacement for the actual request?

Prepared queries have a name, you can not pass any arguments to it.
Thus you cannot really use it for dynamic queries.

e.g. say I want to find a service with the name Customer and the tags VIP and Sweden.
Then I would be forced to have one Query for VipSweden and then another for VipNorway etc etc.
No?

If it was possible to use multiple tags in the API directly, one could do what the OP suggested:

curl 'http://localhost:8500/v1/catalog/service/mysql?tag=VIP&tag=Sweden

Currently the only option for building context aware service discovery is to query for all healthy instances and then filter it on the client.

[slackpad]

@rogeralsing prepared query templates let you parameterize so you can make a single query that can handle "vip" + "something else". We opened #1781 for a solution to ad-hoc queries with multiple tags as PQs still need to be created before they can be executed.

[rogeralsing]

Perfect, I added a querstion about prepared query templates here https://groups.google.com/forum/#!topic/consul-tool/iaPwF9gLT0w

I get that I can use regex to extract parts of the name, but how would one go about to extract multiple tags? e.g. should that match contain a comma separated list when it is inserted into the template?
That is, how do I make this multiple matches?
"Tags": ["${match(2)}"]
Sorry for the crosspost