Fix gRPC query service cmux

[csweichel]

fixes #1754

breaking change: grpc/grpc-go#2406
workaround described in:

• Notice: grpc-go will stop sending RPCs before HTTP/2 SETTINGS frame is received soheilhy/cmux#64
• https://github.com/soheilhy/cmux#limitations

[codecov]

Codecov Report

Merging #1756 into master will decrease coverage by 0.04%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1756      +/-   ##
==========================================
- Coverage   98.26%   98.22%   -0.05%     
==========================================
  Files         195      195              
  Lines        9533     9527       -6     
==========================================
- Hits         9368     9358      -10     
- Misses        131      134       +3     
- Partials       34       35       +1

Impacted Files	Coverage Δ
cmd/query/app/server.go	90.47% <100%> (-0.3%)	⬇️
cmd/query/app/static_handler.go	83.33% <0%> (-3.51%)	⬇️
model/span.go	100% <0%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4978c15...37b2e54. Read the comment docs.

[yurishkuro]

Thanks for the research & the fix, @32leaves .

I would like to have a unit test that uses grpc client to hit grpc endpoint via cmux, so that we don't have regressions like these in the future.

[yurishkuro]

I don't think this fix is helping, I added a unit test #1758 for connecting to grpc server via cmux, and it's failing both before and after the change to matchers. Also, the behavioral change in grpc/grpc-go#2406 was only released in grpc-go 1.23, but we're currently pinning grpc to 1.21.1

[yurishkuro]

cc @annanay25

[csweichel]

I just pushed the test I used while working on the fix (didn't see you had already added a test as well - I'm happy to remove mine again). With this fix I can successfully call GetServices using the gRPC interface.

[annanay25]

Thanks for the PR @32leaves. Travis is failing with -

cmd/query/app/server_test.go:69:97: grpc.WithTimeout is deprecated: use DialContext and context.WithTimeout instead.  (SA1019)

We should be able to go forward and land once this is fixed :)
Ref: #1758

[yurishkuro]

My mistake, my PR was using the wrong port.

It's fixed now, I prefer to merge my PR because it includes a bunch of other clean-up. I pulled the main commit from here for the actual fix.

[yurishkuro]

superseded by #1758