remote-read: [1] Extended remote read proto with negotiation and chunked response. #5701

bwplotka · 2019-06-21T15:11:57Z

First part of chained PRs to add streamed, chunked remote read.

Part of: #4517 and thanos-io/thanos#488

Notes for reviewers:

To fully understand decisions and how usage will look like feel free to see all changes together here.

.. and caller side changes (Thanos): thanos-io/thanos#1268

I thought it will be much easier to discuss changes in much smaller chunks.

PTAL @brian-brazil @tomwilkie @gouthamve @krasi-georgiev and others. Feedback very welcome.

Concerns from my side:

Should we support non-streamed XOR Chunk. IMO only streamed makes sense.
Should we support streamed Sampled response? IMO not.
Should we do content negotiation based on request headers instead? I think proto is way more explicit and will help in maintaing compatibility.
Should we split responseType and negotiate streaming encoding and compression in three different enums / ReadRequest parameters? It will be awful on implementation site and user will expect every permutation to work. That's why I left only two options: SAMPLES and STREAMED_XOR_CHUNKS

Initial benchmarks are available here: #5703

More details on some decisions: https://docs.google.com/document/d/1JqrU3NjM9HoGLSTPYOvR217f5HBKBiJTqikEB9UiJL0/edit#

Signed-off-by: Bartek Plotka bwplotka@gmail.com

bwplotka · 2019-06-21T16:00:00Z

Pinged Cortex team for feedback: cortexproject/cortex#1472

prompb/remote.proto

juliusv · 2019-06-21T21:53:02Z

prompb/remote.proto

@@ -43,3 +65,12 @@ message QueryResult {
  // Samples within a time series must be ordered by time.
  repeated prometheus.TimeSeries timeseries = 1;
 }
+
+// ChunkedReadResponse is a response when response_type equals STREAMED_XOR_CHUNKS.
+// We strictly stream full series by series. Single frame can contain partion of the single series.


Not sure what "We strictly stream full series by series" means. I thought it means that each single series wouldn't be broken up, but then the next sentence says the opposite?

It rather means that we expect series to be be sequentially send.

So you cannot send series ABC [1 10] then XYZ [1-20] then ABC [11-20] but you can ABC [1 10] then ABC [11-20] then XYZ [1-20] (:

Tried to be more explicit. PTAL. Also not 100% sure if we want to be that strict on interface side.

prompb/types.proto

bwplotka · 2019-06-24T13:17:38Z

Comments fixed, PTAL.

prompb/remote.proto

bwplotka · 2019-06-25T10:31:36Z

@brian-brazil Moved to accepted_response_types (: Updating other PRs in chain.

prompb/remote.proto

bwplotka · 2019-06-25T12:58:51Z

@tomwilkie any feedback on this?

prompb/remote.proto

tomwilkie · 2019-06-26T18:15:12Z

prompb/types.proto

+
+  // We require this to match chunkenc.Encoding.
+  enum Encoding {
+    UNKNOWN = 0;


When is UNKOWN ever used?

no, but please see comment, it helps to match chunkenc.Encoding exact values of enum (int).

tomwilkie · 2019-06-26T18:18:00Z

prompb/remote.proto

+// ChunkedReadResponse is a response when response_type equals STREAMED_XOR_CHUNKS.
+// We strictly stream full series after series, optionally split by time. This means that a single frame can contain
+// partition of the single series, but once a new series is started to be streamed it means that no more chunks will
+// be sent for previous one.


Whilst this might be true, it seems like an unnecessary restriction that we might want to change in the future (esp for very long queries). Should we consider not having this restriction as part of the interface?

Can we say anything about the order of the series we return?

It's something that the other end can rely on for efficiency, and if there's a new TSDB implementation at some point where a different way makes sense we can always have a new type covering it. For now I'd say let users depend on this, rather than sacrificing this to an unknown future change.

Hm. I can agree with @tomwilkie that it will change soon and then we will need to do deprecation process or different type which might be painful, but without that restriction as @brian-brazil says callers cannot assume that useful invariant and gain benefits which would have bad conequences on Prometheus PromQL current shape.

This means that I think I am bit more on restricting it as @brian-brazil suggests, but maybe it makes sense then to discuss / be prepared / have an idea how we will do it if we will need to change that restriction.

Maybe leave the current shape and once new restriction come in, add some enum SeriesOrder to ReadRequest? In theory we can do that gradually once the different restriction arise? And keep it YAGNI for now.

I'm still not seeing how we'd be changing this, and we can always extend the proto.

tomwilkie · 2019-06-26T18:18:18Z

prompb/types.proto

+// ChunkedSeries represents single, encoded time series.
+message ChunkedSeries {
+  repeated Label labels = 1 [(gogoproto.nullable) = false];
+  repeated Chunk chunks = 2 [(gogoproto.nullable) = false];


Is there anything to be said for the order of the chunks we return? Will they overlap?

Good point. I think the Prometheus server implementation will not give overlaps, but not sure if all known caller implementation assumes. I think both Prometheus and Thanos handles overlaps well.

In terms of sorting 🤷‍♂️ need to double check.

I'd say that they shouldn't overlay, and will be sorted.

tomwilkie · 2019-06-26T18:18:36Z

prompb/types.proto

+
+// ChunkedSeries represents single, encoded time series.
+message ChunkedSeries {
+  repeated Label labels = 1 [(gogoproto.nullable) = false];


We tried elsewhere in the remote interfaces to guarantee ordering of the labels IIRC.

yes. Also the in most other places in TSDB. Happy to add such comment and check implementation on later chained PRs.

However, what do you think about having more complex logic - if we send consequtive chunks for exactly the same series in multiple frames it is enough to send labels just on first one, first? But might be optimization we can add later (which will be another interface change ;p)

That'd require a bit more state on the reader side for a pretty small saving, I'd keep it simple and always include it.

bwplotka · 2019-06-28T18:51:19Z

Addressed comments @tomwilkie @brian-brazil

…response. Context: https://docs.google.com/document/d/1JqrU3NjM9HoGLSTPYOvR217f5HBKBiJTqikEB9UiJL0/edit# Signed-off-by: Bartek Plotka <bwplotka@gmail.com>

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>

bwplotka · 2019-07-05T07:36:53Z

Closing -> the squashed commits are now part of this PR: #5703

bwplotka changed the title ~~remote-read: [1] Extended remote read proto with negotiation and and chunked response.~~ remote-read: [1] Extended remote read proto with negotiation and chunked response. Jun 21, 2019

brian-brazil reviewed Jun 21, 2019

View reviewed changes

prompb/remote.proto Show resolved Hide resolved

juliusv reviewed Jun 21, 2019

View reviewed changes