feat(sink): support columns subset for cassandra and doris sink #16821
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xxhZs
changed the title
fuyufjh
reviewed
May 20, 2024
fix fmt
wenym1
reviewed
May 20, 2024
| @@ -143,6 +143,7 @@ impl HeaderBuilder { | |||
| self | |||
| } | |||
|
|
|||
| /// Only use in Starrocks | |||
There was a problem hiding this comment.
How does user to enable partial_update on starrocks sink? I think they can reuse the same with options on user side, but just set different header for different implementations.
There was a problem hiding this comment.
like create sink with(partial_update = 'true');
wenym1
reviewed
May 20, 2024
src/connector/src/sink/doris.rs
Outdated
| @@ -125,8 +127,11 @@ impl DorisSink { | |||
| .collect(); | |||
|
|
|||
| let rw_fields_name = self.schema.fields(); | |||
| if rw_fields_name.len().ne(&doris_columns_desc.len()) { | |||
| return Err(SinkError::Doris("The length of the RisingWave column must be equal to the length of the doris column".to_string())); | |||
| if rw_fields_name.len().gt(&doris_columns_desc.len()) { | |||
There was a problem hiding this comment.
Use > here instead of gt.
And should we allow gt only when partial update is enabled in the with options?
There was a problem hiding this comment.
In theory, these are two things, gt thought that the column of rw was a downstream subset, while partial_update means opening starrocks/doris's feature
|
@xxhZs Any updates? |
fuyufjh
approved these changes
Jun 18, 2024
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Fixes #16633
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
we can add partial_columns in doris sink with option.
like
create sink with(doris.partial_columns = 'true')
https://doris.apache.org/zh-CN/docs/2.0/data-operate/update/update-of-unique-model/