Fix enum & set values after COPY phase #92
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
notfelineit
commented
Dec 13, 2023
|
|
||
| // Takes enum or set column type like ENUM('a','b','c') or SET('a','b','c') | ||
| // and returns a slice of values []string{'a', 'b', 'c'} | ||
| func parseEnumOrSetValues(columnType string) []string { |
There was a problem hiding this comment.
Putting the "parsing" logic here does result in an extra parse on every enum/set column value. If this is too much processing, we could "save" the parsed values somewhere upstream - however, that would require refactoring a great deal more of this code since, as it is currently, we don't pass a lot of metadata around during the READ phase.
Phanatic
approved these changes
Dec 14, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses https://github.com/planetscale/airbyte-source/issues/56 and adds extra support for mapping
ENUMandSETvalues after the initial COPY phase of replication.During the replicating phase, we expect enum and set values to appear as their indices (
"1") rather than their value ("apple"in a column of typeenum('apple','banana','orange')). Because of this, we need extra logic to map them to their actual value before passing the value on to Airbyte.