Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sidecar: Optimize Prometheus Remote Read API. #488

Closed
bwplotka opened this issue Aug 24, 2018 · 3 comments · Fixed by #1268
Closed

sidecar: Optimize Prometheus Remote Read API. #488

bwplotka opened this issue Aug 24, 2018 · 3 comments · Fixed by #1268
Assignees
@bwplotka
Labels
feature request/improvement priority: P0

Comments

@bwplotka
Copy link
Member

bwplotka commented Aug 24, 2018 •
edited
Loading

Due to non-streamed Prometheus remote read that sends series with samples it is highly inefficient with sidecar. This is because sidecar has to buffer whole protobuf and then stream it out.

On recent Prometheus dev summit (May 2019), we finally agreed on shape of streamed, chunked remote read API. I plan to contribute streamed chunked remote read API to Prometheus, and then make sidecar using it.

All info & design available here: prometheus/prometheus#4517
@bwplotka bwplotka self-assigned this Aug 24, 2018
@bwplotka bwplotka mentioned this issue Sep 13, 2018
Closed
@bwplotka bwplotka mentioned this issue Feb 6, 2019
Closed
@bwplotka bwplotka mentioned this issue May 10, 2019
Closed
bwplotka added a commit that referenced this issue May 28, 2019
@bwplotka
Moved flagutil to separate package, expanded blockgen
f39ecb9 
This is needed for remote read benchmarks as the part of #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue May 30, 2019
@bwplotka
Moved flagutil to separate package; expanded blockgen; blockgen docke…
1eee12f 
…rfile added.

This is needed for remote read benchmarks as the part of #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
@bwplotka bwplotka mentioned this issue May 30, 2019
Closed
bwplotka added a commit that referenced this issue May 31, 2019
@bwplotka
Moved flagutil to separate package; expanded blockgen; blockgen docke…
9b80e6f 
…rfile added.

This is needed for remote read benchmarks as the part of #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
@bwplotka
Copy link
Member Author

I am actively working on it now.

bwplotka added a commit that referenced this issue Jun 21, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
38f56b8 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
@bwplotka bwplotka mentioned this issue Jun 21, 2019
Merged
1 task
bwplotka added a commit that referenced this issue Jun 21, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
f5ddb66 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Jun 21, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
9fb1f3f 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
This was referenced Jun 21, 2019
Closed
Closed
Merged
@bwplotka
Copy link
Member Author

bwplotka commented Jun 21, 2019 •
edited
Loading

Back to business!

Protobuf changes, delimited streaming, server: prometheus/prometheus#5703
Caller usage example (Thanos): #1268

Living document: https://docs.google.com/document/d/1JqrU3NjM9HoGLSTPYOvR217f5HBKBiJTqikEB9UiJL0/edit#

bwplotka added a commit that referenced this issue Jul 3, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
c75b1a0 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Jul 3, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
7bd205f 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Jul 5, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
cedf673 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
@bwplotka
Copy link
Member Author

bwplotka commented Jul 5, 2019 •
edited
Loading

First MVP should be safe to use. I build 2 docker images if someone wants to try:

No config changes, just move docker images to that Prometheus and Thanos sidecar tags and enjoy!

2 big requests against Prometheus + sidecar in upgraded version look like this memory and CPU wise:
image

Versus old one with non-streaming sampled read (3GB of mem!):
image

Still it's work in progress and there are more optimization to make, but you can already gain from this.

bwplotka added a commit that referenced this issue Aug 16, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
5641810 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 16, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
2679e0b 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 16, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
3038dda 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 18, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
d0bff9c 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 18, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
98dc30e 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 18, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
9f71df8 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 18, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
04bfeda 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 18, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
acf922d 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 19, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
053d20f 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 19, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
f28e1a1 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 19, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
de650e9 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 19, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
b372d14 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 19, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
6243e4d 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 19, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
4f31ca5 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit to prometheus/prometheus that referenced this issue Aug 19, 2019
@bwplotka
remote-read: streamed chunked server side; Extended protobuf; Added c…
48b2c9c 
…hunked, checksumed reader (#5703)

Part of: #4517 and thanos-io/thanos#488

Changes:
* Extended protobuf for chunked remote read and negotation.
* Added checksumed, chunked Writer/Reader.
* Added Server side implementation for chunked streamed remote-read.


Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 20, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
3df065c 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 20, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
ec9a9d3 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 20, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
ef5d17b 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 20, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
a460fa6 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 22, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
4dc0dc9 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>

# Conflicts:
#	Makefile
#	pkg/store/prompb/remote.pb.go
bwplotka added a commit that referenced this issue Aug 22, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
6094485 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>

# Conflicts:
#	Makefile
#	pkg/store/prompb/remote.pb.go
bwplotka added a commit that referenced this issue Aug 22, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
431aaf5 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>

# Conflicts:
#	Makefile
#	pkg/store/prompb/remote.pb.go
bwplotka added a commit that referenced this issue Aug 22, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
e57811c 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>

# Conflicts:
#	Makefile
#	pkg/store/prompb/remote.pb.go
bwplotka added a commit that referenced this issue Aug 27, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
c216e97 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 27, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
d7b4aef 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 27, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
c6d10fc 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 27, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read.
78ebeea 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Aug 27, 2019
@bwplotka
sidecar: Added support for streaming, chunked remote read. (#1268)
fbffc4b 
Fixes: #488

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bwplotka bwplotka

Labels
feature request/improvement priority: P0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sidecar: Added support for streaming, chunked remote read. thanos-io/thanos
1 participant
@bwplotka