[Low-Code CDK] Write the component schema and use it during manifest validation #20422

brianjlai · 2022-12-13T10:05:26Z

Closes #19759

What

Together Catherine, Maxime, and I handwrote json schema that documents all the available components in the low-code language and their respective relationships. This handwritten schema will then be used as the source of truth for the Pydantic models generated, a reference document for low-code developers, and hopefully the UI.

It is also incorporated into the manifest validation flow.

How

A majority of the work was writing the schema by hand. We went through this checklist to avoid duplicates and ensure that we had consistent formatting and styling for the YAML.

The other aspect was incorporating this into the validation flow. We read in the new schema YAML file and perform the same validator check against the incoming schema. The one nuance here is that because we want to deprecate the factory once we have finished the refactor, the new validation flow doesn't use any part of the existing factory.py to modify the incoming manifest. That means that we're not doing some of the behind the scenes preprocessing in order to adhere to the component schema.

This is handled in the form of the manifest_component_transformer.py, which adds certain fields to components so that it can be validate against our ideal schema:

Applies default types to components that are missing type and have a defined default value
Propagates $options dictionary to all subcomponents

Recommended reading order

low_code_component_schema.yaml
manifest_component_transformer.py
manifest_declarative_source.py

- ApiKeyAuthenticator - BasicHttpAuthenticator - BearerAuthenticator - DeclarativeOauth2Authenticator - NoAuth

…ilter components

…on 'type' not being defined)

… output

… interpolators

…rm validation against the handwritten schema

brianjlai · 2022-12-19T08:46:46Z

/publish-cdk dry-run=true

❌ https://github.com/airbytehq/airbyte/actions/runs/3729823953

brianjlai · 2022-12-19T08:58:21Z

/publish-cdk dry-run=true

🕑 https://github.com/airbytehq/airbyte/actions/runs/3729893236
✅ https://github.com/airbytehq/airbyte/actions/runs/3729893236

brianjlai · 2022-12-19T09:07:06Z

/publish-cdk dry-run=false

🕑 https://github.com/airbytehq/airbyte/actions/runs/3729962091
✅ https://github.com/airbytehq/airbyte/actions/runs/3729962091

…included when package is published

brianjlai · 2022-12-19T19:55:00Z

/publish-cdk dry-run=true

🕑 https://github.com/airbytehq/airbyte/actions/runs/3734712600
✅ https://github.com/airbytehq/airbyte/actions/runs/3734712600

brianjlai · 2022-12-19T20:08:13Z

/publish-cdk dry-run=false

🕑 https://github.com/airbytehq/airbyte/actions/runs/3734807069
✅ https://github.com/airbytehq/airbyte/actions/runs/3734807069

brianjlai · 2022-12-19T20:41:46Z

/approve-and-merge reason="This version of the airbyte-cdk has already been released (w/ a patch fix also done this morning) so merging this is a formality more than a release"

octavia-approvington · 2022-12-19T20:42:32Z

This is really good

franzwilhelm · 2022-12-20T00:10:22Z

@octavia-approvington @brianjlai I started getting issues in my local low code development workflow after this push. Running https://pypi.org/project/airbyte-cdk/0.16.1/ currently.
With this config in low-code:

authenticator:
      type: "OAuthAuthenticator"
      token_refresh_endpoint: "{{ config['token_refresh_endpoint'] }}"
      client_id: "{{ config['client_id'] }}"
      client_secret: "{{ config['client_secret'] }}"
      refresh_token: ""
      refresh_request_body: 
        audience: "{{ config['graphql_endpoint'] }}"
      grant_type: "client_credentials"

This is output from python main.py spec

Traceback (most recent call last):
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 155, in _validate_source
    validate(propagated_manifest, declarative_component_schema)
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: 'OAuthAuthenticator' is not one of ['ApiKeyAuthenticator']

Failed validating 'enum' in schema[0]['properties']['type']:
    {'enum': ['ApiKeyAuthenticator'], 'type': 'string'}

On instance['type']:
    'OAuthAuthenticator'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/main.py", line 12, in <module>
    source = SourceGraphql()
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/source_graphql/source.py", line 18, in __init__
    super().__init__(**{"path_to_yaml": "graphql.yaml"})
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/yaml_declarative_source.py", line 21, in __init__
    super().__init__(source_config, debug)
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 62, in __init__
    self._validate_source()
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 157, in _validate_source
    raise ValidationError("Validation against json schema defined in declarative_component_schema.yaml schema failed") from e
jsonschema.exceptions.ValidationError: Validation against json schema defined in declarative_component_schema.yaml schema failed

But as soon as i remove the refresh_token key from my config I get this:

Traceback (most recent call last):
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/main.py", line 12, in <module>
    source = SourceGraphql()
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/source_graphql/source.py", line 18, in __init__
    super().__init__(**{"path_to_yaml": "graphql.yaml"})
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/yaml_declarative_source.py", line 21, in __init__
    super().__init__(source_config, debug)
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 62, in __init__
    self._validate_source()
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 131, in _validate_source
    streams = [self._factory.create_component(stream_config, {}, False)() for stream_config in self._stream_configs()]
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 131, in <listcomp>
    streams = [self._factory.create_component(stream_config, {}, False)() for stream_config in self._stream_configs()]
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 129, in create_component
    return self.build(
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 146, in build
    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 146, in <dictcomp>
    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 209, in _create_subcomponent
    return self.create_component(definition, config, instantiate)()
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 129, in create_component
    return self.build(
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 146, in build
    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 146, in <dictcomp>
    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 209, in _create_subcomponent
    return self.create_component(definition, config, instantiate)()
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 129, in create_component
    return self.build(
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 146, in build
    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 146, in <dictcomp>
    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 200, in _create_subcomponent
    return self.create_component(definition, config, instantiate)()
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 129, in create_component
    return self.build(
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 167, in build
    validate(component_definition, schema)
  File "/Users/franz/Programming/aco.airbyte-cdk/airbyte-integrations/connectors/source-graphql/.venv/lib/python3.9/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: 'refresh_token' is a required property

Failed validating 'required' in schema['allOf'][1]:
    {'properties': {'_token_expiry_date': {},
                    'access_token_name': {'anyOf': [{'$ref': '#/definitions/InterpolatedString'},
                                                    {'type': 'string'}],
                                          'default': 'access_token'},
                    'client_id': {'anyOf': [{'$ref': '#/definitions/InterpolatedString'},
                                            {'type': 'string'}]},
                    'client_secret': {'anyOf': [{'$ref': '#/definitions/InterpolatedString'},
                                                {'type': 'string'}]},
                    'config': {'type': 'object'},
                    'expires_in_name': {'anyOf': [{'$ref': '#/definitions/InterpolatedString'},
                                                  {'type': 'string'}],
                                        'default': 'expires_in'},
                    'grant_type': {'anyOf': [{'$ref': '#/definitions/InterpolatedString'},
                                             {'type': 'string'}],
                                   'default': 'refresh_token'},
                    'refresh_request_body': {'type': 'object'},
                    'refresh_token': {'anyOf': [{'$ref': '#/definitions/InterpolatedString'},
                                                {'type': 'string'}]},
                    'scopes': {'items': {'type': 'string'},
                               'type': 'array'},
                    'token_expiry_date': {'anyOf': [{'$ref': '#/definitions/InterpolatedString'},
                                                    {'type': 'string'}]},
                    'token_expiry_date_format': {'type': 'string'},
                    'token_refresh_endpoint': {'anyOf': [{'$ref': '#/definitions/InterpolatedString'},
                                                         {'type': 'string'}]}},
     'required': ['token_refresh_endpoint',
                  'client_id',
                  'client_secret',
                  'refresh_token',
                  'config'],
     'type': 'object'}

On instance:
    {'$options': {'name': 'graphql', 'path': '/'},
     'client_id': "{{ config['client_id'] }}",
     'client_secret': "{{ config['client_secret'] }}",
     'config': {},
     'grant_type': 'client_credentials',
     'name': 'graphql',
     'path': '/',
     'refresh_request_body': {'audience': "{{ config['graphql_endpoint'] "
                                          '}}'},
     'token_refresh_endpoint': "{{ config['token_refresh_endpoint'] }}"}

Could you help me out here, or should I submit this as an issue?

brianjlai · 2022-12-20T00:36:47Z

hey @franzwilhelm thanks for pointing this out! This is an issue on our end and I see what the problem is. I'll start working on a fix for this

franzwilhelm · 2022-12-20T00:51:46Z

Workaround for now was to pin the airbyte cdk to 0.15.0. Thanks!

MAIN_REQUIREMENTS = [
    "airbyte-cdk~=0.15.0",
]

brianjlai · 2022-12-20T21:16:14Z

hey @franzwilhelm I just pushed an update. Can you try 0.16.2 and see if that gets rid of your issue in the authenticator?

brianjlai and others added 30 commits December 6, 2022 23:42

handwritten low code manifest example components

6934d5f

add MinMaxDatetime to jsonschema

8f871ad

add a basic gradle command to generate manifest components

828614e

Add auth components to handwritten component schema

48ba3f9

- ApiKeyAuthenticator - BasicHttpAuthenticator - BearerAuthenticator - DeclarativeOauth2Authenticator - NoAuth

Respect optional properties in DeclarativeOauth2Authenticator

ada53df

Fix Dict[str, Any] mapping in auth components

329f623

add default error handler composite error handler and http response f…

c5c9346

…ilter components

[low code component schema] adding backoff strategies to schema

c214724

[low code component schema] fix float types

a91e26c

[low code component schema] add RecordFilter

6edbc29

Remove config from auth components

ee6e142

[low code component schema] add Interpolation (with pending question …

f52248d

…on 'type' not being defined)

Add CartesianProductStreamSlicer & DatetimeStreamSlicer

36b9480

Add ListStreamSlicer, and fix nesting of DatetimeStreamSlicer

b735028

[low code component schema] add InterpolatedRequestOptionsProvider

7b74a2d

Add slicer components, and fix a couple of components after reviewing…

f482ad1

… output

[low code component schema] adding transformations and adding type to…

6bf1c17

… interpolators

adding spec and a few small tweaks

43f0628

Add DefaultSchemaLoader

11eb8d5

[low code component schema] attempt on custom class

607bed4

Add descriptions for auth components

f4f8909

add RequestOption

6769968

remove interpolated objects from the schema in favor of strings only

ac099f8

a few schema fixes and adding some custom pagination and stream slicer

37dedfe

[low code component schema] fix CustomBackoffStrategy

414bfd6

Add CustomRecordExtractor

72ae128

add some description and add additional properties

3e34503

insert a transformer to hydrate default manifest components and perfo…

fbadf8f

…rm validation against the handwritten schema

[low code component schema] validating existing schemas

9d39977

[low code component schema] clean validation script

2408f41

flash1293 mentioned this pull request Dec 16, 2022

🪟🎉 Connector builder: Allow defining inputs #20431

Merged

brianjlai added 6 commits December 16, 2022 15:51

Merge branch 'master' into low_code_handwritten_component_schema

c6fa661

fixing schema bugs from testing and new components

f94fdf5

Merge branch 'master' into low_code_handwritten_component_schema

4dfacba

last few small tweaks and fixes and fix the script

320ec02

Merge branch 'master' into low_code_handwritten_component_schema

fa36e8a

bump the version before publsih

42a6ab5

brianjlai added 2 commits December 19, 2022 11:44

include the declarative_component_schema.yaml as package data so its …

8f77835

…included when package is published

roll the version to publish

fc39d3f

octavia-approvington approved these changes Dec 19, 2022

View reviewed changes

octavia-approvington merged commit b7113a2 into master Dec 19, 2022

octavia-approvington deleted the low_code_handwritten_component_schema branch December 19, 2022 20:42

brianjlai mentioned this pull request Dec 20, 2022

fix a bug where the OAuthAuthenticator was misnamed #20700

Merged

github-actions bot mentioned this pull request Dec 27, 2022

Bump helm chart version reference to 0.43.9 #20892

Merged

This was referenced Jan 6, 2023

Bump Airbyte version from 0.40.26 to 0.40.27 #21092

Closed

Bump Airbyte version from 0.40.26 to 0.40.27 #21121

Closed

This was referenced Jan 6, 2023

Bump Airbyte version from 0.40.26 to 0.40.27 #21130

Closed

Bump Airbyte version from 0.40.26 to 0.40.27 #21135

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Low-Code CDK] Write the component schema and use it during manifest validation #20422

[Low-Code CDK] Write the component schema and use it during manifest validation #20422

brianjlai commented Dec 13, 2022 •

edited

Loading

brianjlai commented Dec 19, 2022 •

edited by github-actions bot

Loading

brianjlai commented Dec 19, 2022 •

edited by github-actions bot

Loading

brianjlai commented Dec 19, 2022 •

edited by github-actions bot

Loading

brianjlai commented Dec 19, 2022 •

edited by github-actions bot

Loading

brianjlai commented Dec 19, 2022 •

edited by github-actions bot

Loading

brianjlai commented Dec 19, 2022

octavia-approvington commented Dec 19, 2022

franzwilhelm commented Dec 20, 2022

brianjlai commented Dec 20, 2022

franzwilhelm commented Dec 20, 2022

brianjlai commented Dec 20, 2022

[Low-Code CDK] Write the component schema and use it during manifest validation #20422

[Low-Code CDK] Write the component schema and use it during manifest validation #20422

Conversation

brianjlai commented Dec 13, 2022 • edited Loading

What

How

Recommended reading order

brianjlai commented Dec 19, 2022 • edited by github-actions bot Loading

brianjlai commented Dec 19, 2022 • edited by github-actions bot Loading

brianjlai commented Dec 19, 2022 • edited by github-actions bot Loading

brianjlai commented Dec 19, 2022 • edited by github-actions bot Loading

brianjlai commented Dec 19, 2022 • edited by github-actions bot Loading

brianjlai commented Dec 19, 2022

octavia-approvington commented Dec 19, 2022

franzwilhelm commented Dec 20, 2022

brianjlai commented Dec 20, 2022

franzwilhelm commented Dec 20, 2022

brianjlai commented Dec 20, 2022

brianjlai commented Dec 13, 2022 •

edited

Loading

brianjlai commented Dec 19, 2022 •

edited by github-actions bot

Loading

brianjlai commented Dec 19, 2022 •

edited by github-actions bot

Loading

brianjlai commented Dec 19, 2022 •

edited by github-actions bot

Loading

brianjlai commented Dec 19, 2022 •

edited by github-actions bot

Loading

brianjlai commented Dec 19, 2022 •

edited by github-actions bot

Loading