-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
example pokeapi with new protocol #17489
Conversation
}, | ||
"weight": { | ||
"type": ["null", "integer"] | ||
"$ref": "WellKnownTypes.json#definitions/Integer" | ||
}, | ||
"abilities": { | ||
"type": ["null", "array"], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
as seen here: non-primitive types (arrays, objects, oneof, etc) are not defined in WellKnownTypes, because they need additional configuration to fully describe their contents
@@ -0,0 +1,60 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file is just the JSON-ified version of WellKnownTypes.yaml. In the real usage, this file will be provided as part of the CDK - I'm copying it in manually because I don't want to set up the full yaml -> json -> copy pipeline just for this example.
@@ -3,45 +3,51 @@ | |||
"type": "object", | |||
"properties": { | |||
"id": { | |||
"type": ["null", "integer"] | |||
"$ref": "WellKnownTypes.json#definitions/Integer", | |||
"airbyte_attributes": { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
here's an example of airbyte_attributes
. The $ref
technically "overrides" this, but that won't matter: All the components that care about airbyte_attributes
don't actually want to resolve the $ref
. For example, normalization would prefer to just check for $ref == [...]Integer
rather than resolving it to the full {"type": "string", "pattern": "..."}
.
This won't be available in our v0 implementation, but is something that we'll keep in mind for future enhancements. It's purely additive, so it'll be pretty easy as a bolt-on.
@@ -3,45 +3,51 @@ | |||
"type": "object", | |||
"properties": { | |||
"id": { | |||
"type": ["null", "integer"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
null
is redundant: these properties aren't required
, so the source can just not set them at all.
If we find that sources actually want to output {"id": null}
, then we can always update the well-known types to use type: [null, integer]
.
Does |
/test connector=connectors/source-pokeapi
Build PassedTest summary info:
|
An example port of source-pokeapi to the updated protocol version described in #17486. Specifically:
source.py
uses the default type transformer to wrap numeric values into stringsThis PR also demonstrates
airbyte_attributes
.See comments below for further details - #17489 (review)