Add test case for vector matches on the promql compliance

[alanprot]

Adding test case to cover cases like thanos-io/thanos#5799

[juliusv]

Thanks! A couple of points:

• This should go somewhere in the binary operator test queries section:

  compliance/promql/promql-test-queries.yml

  Lines 99 to 134 in d128251

  	# Binary operators.
  	- query: '1 * 2 + 4 / 6 - 10 % 2 ^ 2'
  	- query: 'demo_num_cpus + (1 {{.compBinOp}} bool 2)'
  	variant_args: ['compBinOp']
  	- query: 'demo_memory_usage_bytes {{.binOp}} 1.2345'
  	variant_args: ['binOp']
  	- query: 'demo_memory_usage_bytes {{.compBinOp}} bool 1.2345'
  	variant_args: ['compBinOp']
  	- query: '1.2345 {{.compBinOp}} bool demo_memory_usage_bytes'
  	variant_args: ['compBinOp']
  	- query: '0.12345 {{.binOp}} demo_memory_usage_bytes'
  	variant_args: ['binOp']
  	- query: '(1 * 2 + 4 / 6 - (10%7)^2) {{.binOp}} demo_memory_usage_bytes'
  	variant_args: ['binOp']
  	- query: 'demo_memory_usage_bytes {{.binOp}} (1 * 2 + 4 / 6 - 10)'
  	variant_args: ['binOp']
  	# Check that vector-scalar binops set output timestamps correctly.
  	- query: 'timestamp(demo_memory_usage_bytes * 1)'
  	# Check that unary minus sets timestamps correctly.
  	# TODO: Check this more systematically for every node type?
  	- query: 'timestamp(-demo_memory_usage_bytes)'
  	- query: 'demo_memory_usage_bytes {{.binOp}} on(instance, job, type) demo_memory_usage_bytes'
  	variant_args: ['binOp']
  	- query: 'sum by(instance, type) (demo_memory_usage_bytes) {{.binOp}} on(instance, type) group_left(job) demo_memory_usage_bytes'
  	variant_args: ['binOp']
  	- query: 'demo_memory_usage_bytes {{.compBinOp}} bool on(instance, job, type) demo_memory_usage_bytes'
  	variant_args: ['compBinOp']
  	# Check that __name__ is always dropped, even if it's part of the matching labels.
  	- query: 'demo_memory_usage_bytes / on(instance, job, type, __name__) demo_memory_usage_bytes'
  	- query: 'sum without(job) (demo_memory_usage_bytes) / on(instance, type) demo_memory_usage_bytes'
  	- query: 'sum without(job) (demo_memory_usage_bytes) / on(instance, type) group_left demo_memory_usage_bytes'
  	- query: 'sum without(job) (demo_memory_usage_bytes) / on(instance, type) group_left(job) demo_memory_usage_bytes'
  	- query: 'demo_memory_usage_bytes / on(instance, job) group_left demo_num_cpus'
  	- query: 'demo_memory_usage_bytes / on(instance, type, job, non_existent) demo_memory_usage_bytes'
  	# TODO: Add non-explicit many-to-one / one-to-many that errors.
  	# TODO: Add many-to-many match that errors.

• That section is currently woefully insufficient. It hasn't been testing ignoring at all (only on), so maybe adding similar queries with ignoring there as the currently existing on ones could already uncover this issue as well.
• Currently the testing setup only scrapes data from the demo service (https://github.com/juliusv/prometheus_demo_service), and new queries should stick to that as well. You can play with all the demo_ metrics here: https://demo.promlabs.com/graph.

[alanprot]

Thanks!

About the last point those metrics are in fact from the demo server as we can see below.

You mean that ideally we should use just metrics that starts with demo_?

[juliusv]

Ah right, it does get emitted as part of the standard client library metrics from the demo job, but I'd prefer to stick to the metrics that are explicitly created by the demo service itself if we can. So far all other examples only query those demo_ metrics.