Table addition validation #2469
Open
+200
−33
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Amogh-Bharadwaj
force-pushed
the
add-table-validate
branch
from
66a17c0 to
80d31a7
Compare
| @@ -190,3 +196,93 @@ func (h *FlowRequestHandler) CheckIfMirrorNameExists(ctx context.Context, mirror | |||
|
|
|||
| return nameExists.Bool, nil | |||
| } | |||
|
|
|||
| func (h *FlowRequestHandler) ValidateTableAdditions(ctx context.Context, req *protos.ValidateTableAdditionsRequest) (*protos.ValidateCDCMirrorResponse, error) { | |||
There was a problem hiding this comment.
why do we want to split the API into validating and then doing the actual operation? it seems more reasonable to have the existing endpoint check and then error out, would map better to the ClickPipes UI as well
| Requires publication name as string and table values in the form of (schema::text, table::text) pairs. | ||
| The schema and table names should be quoted and escaped. | ||
| */ | ||
| func (c *PostgresConnector) CheckIfTablesAreInPublication(ctx context.Context, pubName string, tableValues []string) error { |
There was a problem hiding this comment.
can share code with the check done in AddTablesToPublication
serprex
reviewed
Feb 12, 2025
Comment on lines
+172
to
179
| err = chPeer.CheckDestinationTables(ctx, connclickhouse.ClickHouseDestinationCheckInput{ | ||
| TableMappings: req.ConnectionConfigs.TableMappings, | ||
| TableNameSchemaMapping: res, | ||
| SyncedAtColName: req.ConnectionConfigs.SyncedAtColName, | ||
| Resync: req.ConnectionConfigs.Resync, | ||
| DoInitialSnapshot: req.ConnectionConfigs.DoInitialSnapshot, | ||
| }) | ||
| if err != nil { |
There was a problem hiding this comment.
Suggested change
| err = chPeer.CheckDestinationTables(ctx, connclickhouse.ClickHouseDestinationCheckInput{ | |
| TableMappings: req.ConnectionConfigs.TableMappings, | |
| TableNameSchemaMapping: res, | |
| SyncedAtColName: req.ConnectionConfigs.SyncedAtColName, | |
| Resync: req.ConnectionConfigs.Resync, | |
| DoInitialSnapshot: req.ConnectionConfigs.DoInitialSnapshot, | |
| }) | |
| if err != nil { | |
| if err := chPeer.CheckDestinationTables(ctx, connclickhouse.ClickHouseDestinationCheckInput{ | |
| TableMappings: req.ConnectionConfigs.TableMappings, | |
| TableNameSchemaMapping: res, | |
| SyncedAtColName: req.ConnectionConfigs.SyncedAtColName, | |
| Resync: req.ConnectionConfigs.Resync, | |
| DoInitialSnapshot: req.ConnectionConfigs.DoInitialSnapshot, | |
| }); err != nil { |
serprex
reviewed
Feb 12, 2025
Comment on lines
+272
to
+280
| err = chPeer.CheckDestinationTables(ctx, connclickhouse.ClickHouseDestinationCheckInput{ | ||
| TableMappings: req.AddedTables, | ||
| TableNameSchemaMapping: res, | ||
| SyncedAtColName: req.SyncedAtColName, | ||
| // TODO: Implement resync and cdc-only for table addition | ||
| Resync: false, | ||
| DoInitialSnapshot: true, | ||
| }) | ||
| if err != nil { |
There was a problem hiding this comment.
Suggested change
| err = chPeer.CheckDestinationTables(ctx, connclickhouse.ClickHouseDestinationCheckInput{ | |
| TableMappings: req.AddedTables, | |
| TableNameSchemaMapping: res, | |
| SyncedAtColName: req.SyncedAtColName, | |
| // TODO: Implement resync and cdc-only for table addition | |
| Resync: false, | |
| DoInitialSnapshot: true, | |
| }) | |
| if err != nil { | |
| if err := chPeer.CheckDestinationTables(ctx, connclickhouse.ClickHouseDestinationCheckInput{ | |
| TableMappings: req.AddedTables, | |
| TableNameSchemaMapping: res, | |
| SyncedAtColName: req.SyncedAtColName, | |
| // TODO: Implement resync and cdc-only for table addition | |
| Resync: false, | |
| DoInitialSnapshot: true, | |
| }); err != nil { |
serprex
reviewed
Feb 12, 2025
| return nil, errors.New("source peer config is not postgres") | ||
| } | ||
|
|
||
| pgPeer, err := connpostgres.NewPostgresConnector(ctx, nil, sourcePeerConfig) |
There was a problem hiding this comment.
We need to stop creating postgres connectors like this in endpoints
See #2525 where I put validation behind a connector interface
serprex
reviewed
Feb 12, 2025
Comment on lines
+240
to
+244
| publicationErr := pgPeer.CheckIfTablesAreInPublication(ctx, req.PublicationName, addedTableValues) | ||
| if publicationErr != nil { | ||
| h.alerter.LogNonFlowWarning(ctx, telemetry.EditMirror, req.FlowJobName, publicationErr.Error()) | ||
| return nil, publicationErr | ||
| } |
There was a problem hiding this comment.
Suggested change
| publicationErr := pgPeer.CheckIfTablesAreInPublication(ctx, req.PublicationName, addedTableValues) | |
| if publicationErr != nil { | |
| h.alerter.LogNonFlowWarning(ctx, telemetry.EditMirror, req.FlowJobName, publicationErr.Error()) | |
| return nil, publicationErr | |
| } | |
| if err := pgPeer.CheckIfTablesAreInPublication(ctx, req.PublicationName, addedTableValues); err != nil { | |
| h.alerter.LogNonFlowWarning(ctx, telemetry.EditMirror, req.FlowJobName, err.Error()) | |
| return nil, err | |
| } |
I know it seems nice to avoid shadowing, but what happens is someone makes a change to the error message or something, assumes the variable is called err, & ends up having their code reference some always-nil err from earlier in the code
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces validation for table addition. The validation consists of:
A new Flow API endpoint has been made for this, and the UI sends a request to this if there are additional tables selected.