Request for feedback on Prometheus streamed, chunked proto interface.

[bwplotka]

Hi 👋

As agreed on the recent Prometheus dev summit we need to all have consensus on the end shape of proto interface for extended remote read: prometheus/prometheus#4517

As Cortex is using (or plans to use) streamed remote read we would love to have common agreement to have this being reusable for both Cortex, Thanos and others.

I would say let's discuss this on proposed proto changes PR: prometheus/prometheus#5703

I already implemented caller and server code for this and performed some benchmarks, details are here: https://docs.google.com/document/d/1JqrU3NjM9HoGLSTPYOvR217f5HBKBiJTqikEB9UiJL0/edit#

Looking forward to your feedback! ❤️

[bboreham]

Thanks for this.

First thing that comes to my mind is: I want streamed remote write! The effort to decompress raw samples inside Cortex distributor is huge (most of the cost is in garbage-collection, but still).

Was this discussed?

[bwplotka]

Well, our discussion was touching overall if we want and how to stream or encode those Prometheus APIs. The clear decision for now was only on streamed and chunked version for remote read. In terms of if write chunks instead of samples for remote write (or if stream in different way), problem with chunks is that AFAIK remote write takes samples from WAL and there are no chunks at that point so it has to be encoded into chunks somewhere. Now is the question should it be on caller or receiver side. Definitely some benchmarks and design might useful here Anyway this change definitely can be related as we are defining now how we expose chunks in protobuf and how to stream (e.g varint delimited protobuf messages), so it might reusable at some point. This means that I would recommend taking a look on what we propose. (:

…

On Fri, Jun 21, 2019, 20:33 Bryan Boreham ***@***.***> wrote: Thanks for this. First thing that comes to my mind is: I want streamed remote write! The effort to decompress raw samples inside Cortex distributor is huge (most of the cost is in garbage-collection, but still). Was this discussed? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1472?email_source=notifications&email_token=ABVA3O3APHQP5DL77DB4TQTP3UUHXA5CNFSM4H2S53DKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYJMQLY#issuecomment-504547375>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABVA3O5R6M2VTBHQO42NLTTP3UUHXANCNFSM4H2S53DA> .

[gouthamve]

Closing this as it's done in prometheus/prometheus#5703