Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VStream API: allow cells to be specified for picking source tablets to stream from #10294

Merged

Conversation

rohit-nayak-ps
Copy link
Contributor

@rohit-nayak-ps rohit-nayak-ps commented May 12, 2022

Description

The VTGate VStreamAPI, internally needs to source data from tablets using the RowStreamer and VStreamer APIs. It uses a TabletPicker to choose a tablet per shard. If a REPLICA tablet type is specified, currently the picker looks at tablets in the same cell as the vtgate to choose a replica to source from. However it is possible that there is no replica in the cell of the vtgate one connects to.

This PR adds the ability to specify a list of (comma separated) cell names/cell aliases. Instead of changing the API signature we add a new Cells field to the VStreamFlags struct for this purpose.

Sample usage:

options := &vtgatepb.VStreamFlags{
	Cells: "zone1, zone4"
}
reader, err := conn.VStream(ctx, topodatapb.TabletType_REPLICA, vgtid, filter, options)

The functionality is validated by a e2e test.

Related Issue(s)

Checklist

  • "Backport me!" label has been added if this change should be backported
  • Tests were added or are not required
  • Documentation was added or is not required

…or candidates in

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
@rohit-nayak-ps rohit-nayak-ps added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: VReplication release notes labels May 12, 2022
@deepthi
Copy link
Member

deepthi commented May 12, 2022

I assume this will also work with cell-aliases?

@rohit-nayak-ps
Copy link
Contributor Author

I assume this will also work with cell-aliases?

Yes, because it is just calls the tablet picker with the cell names and we already resolve aliases there.

@github-actions
Copy link
Contributor

github-actions bot commented May 13, 2022

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has the correct release notes label. release notes none should only be used for PRs that are so trivial that they need not be included.
  • If a new flag is being introduced, review whether it is really needed. The flag names should be clear and intuitive (as far as possible), and the flag's help should be descriptive.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should either include a link to an issue that describes the bug OR an actual description of the bug and how to reproduce, along with a description of the fix.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
@rohit-nayak-ps
Copy link
Contributor Author

@mattlord, @deepthi ping for review, thx.

@mattlord
Copy link
Contributor

Sorry, @rohit-nayak-ps ! Somehow I completely missed this one. I will review in the next few hours.

Copy link
Contributor

@mattlord mattlord left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had some questions and nits, but nothing major. Will approve so that you're not blocked if I misunderstood something or you disagree.

go/test/endtoend/vreplication/vreplication_test.go Outdated Show resolved Hide resolved
go/test/endtoend/vreplication/vreplication_test.go Outdated Show resolved Hide resolved
go/vt/vtgate/vstream_manager.go Show resolved Hide resolved
go/vt/vtgate/vstream_manager.go Outdated Show resolved Hide resolved
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
@rohit-nayak-ps rohit-nayak-ps merged commit f82d8bc into vitessio:main May 27, 2022
@rohit-nayak-ps rohit-nayak-ps deleted the rn-vstream-api-allow-specifying-cells branch May 27, 2022 08:11
DeathBorn pushed a commit to vinted/vitess that referenced this pull request Apr 11, 2024
…o stream from (vitessio#10294)

* Add parameter Cells in vstream api flags, for tablet picker to look for candidates in

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Add e2e test for the new Cells vstream flag

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Self-review

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Address review comments

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Vilius Okockis <vilius.okockis@vinted.com>
DeathBorn pushed a commit to vinted/vitess that referenced this pull request Apr 12, 2024
…o stream from (vitessio#10294)

* Add parameter Cells in vstream api flags, for tablet picker to look for candidates in

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Add e2e test for the new Cells vstream flag

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Self-review

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Address review comments

Fix gofmt

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Vilius Okockis <vilius.okockis@vinted.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: VReplication Type: Enhancement Logical improvement (somewhere between a bug and feature)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants