Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: switch back to gogoproto, rm stringlabels #7790

Merged
merged 9 commits into from
Oct 1, 2024
Merged

*: switch back to gogoproto, rm stringlabels #7790

merged 9 commits into from
Oct 1, 2024

Conversation

GiedriusS
Copy link
Member

@GiedriusS GiedriusS commented Oct 1, 2024

I suggest changing course back to gogoproto. It seems to me right now that the best course of action is to focus on implementing a new RPC protocol that has zero-copy and would give us tight control of allocations/memory usage. Here are some of the reasons why I think we should do that:

  • protoc-gen-go forces []*T instead of []T. This makes sense from proto perspective but it's bad for us because it means a lot of extra allocations: not just for the data itself but also for pointers to each variable.
  • protoc-gen-go forces creation of extra fields for protobuf features that we don't need: https://github.com/protocolbuffers/protobuf-go/blob/03df6c145d9653d4607107dbb3168d05a9b769c8/cmd/protoc-gen-go/internal_gengo/main.go#L370-L396. We could workaround this by custom coding marshaling/unmarshaling functions but that just feels like fighting against the RPC layer/framework.
  • String labels are bad for us because they force another allocation. If we reuse the same Name/Value string format, then we can cast with zero copies, but if labels.Labels is different, then we would be forced to allocate again to accommodate the custom layout.
  • grpc-go is bad for us because sending a message there happens at some unspecified time in the future and hence it is impossible to reuse memory when one Querier is connected to another Querier
  • I have found that there is some speed regression with the v2 encoding; just reverting back to an older version gave me a non-trivial speed up. Didn't have time to dig into it more.

This reverts commit a2113fd.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
This reverts commit 735db72.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
This reverts commit 30f453e.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
This reverts commit 8cd3fae.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
This reverts commit 883fade.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
This reverts commit ca8ab90.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
This reverts commit a8e7109.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
This reverts commit 97710f4.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
@GiedriusS GiedriusS force-pushed the rm_vtprotobuf branch 2 times, most recently from e1f3dae to 2d06594 Compare October 1, 2024 11:44
This reverts commit 8c8a88e.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
@GiedriusS GiedriusS marked this pull request as ready for review October 1, 2024 12:42
@fpetkovski
Copy link
Contributor

I don’t know if we should revert stringlabels altogether. It is very useful for receivers which are very memory-heavy.

Maybe be want stringlabels specific implementations for ZLabelsToPromLabels and ZLabelsFromPromLabels using the same build tag that prometheus uses?

@GiedriusS
Copy link
Member Author

I don’t know if we should revert stringlabels altogether. It is very useful for receivers which are very memory-heavy.

Maybe be want stringlabels specific implementations for ZLabelsToPromLabels and ZLabelsFromPromLabels using the same build tag that prometheus uses?

👍 I think let's distribute two versions of Thanos so that users could run Receive with stringlabels. It's awkward but should still be helpful. I'll do that in a separate PR.

@GiedriusS GiedriusS merged commit bcd20a3 into main Oct 1, 2024
21 of 22 checks passed
jnyi pushed a commit to jnyi/thanos that referenced this pull request Oct 17, 2024
* Revert "store: add chunk pooling (thanos-io#7771)"

This reverts commit a2113fd.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Revert "query/store: memoize PromLabels() call (thanos-io#7767)"

This reverts commit 735db72.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Revert "store: compare labels directly (thanos-io#7766)"

This reverts commit 30f453e.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Revert "store: don't create intermediate labels (thanos-io#7762)"

This reverts commit 8cd3fae.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Revert "*: build with stringlabels (thanos-io#7745)"

This reverts commit 883fade.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Revert "*: enable gRPC pooling (thanos-io#7742)"

This reverts commit ca8ab90.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Revert "*: switch to vtprotobuf (thanos-io#7721)"

This reverts commit a8e7109.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Revert "*: removing gogoproto extensions (thanos-io#7718)"

This reverts commit 97710f4.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Revert "*: rm ZLabels (thanos-io#7675)"

This reverts commit 8c8a88e.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

---------

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants