Update source-gnews manifest to use inline stream schemas #20405

clnoll · 2022-12-12T23:58:19Z

What

Updates the source-gnews low-code connector to include the stream schema inline in the manifest.
See #19499.

This is blocked by #20375.

TODO: update the airbyte-cdk version used by source-gnews once #20375 has been released.

How

Updates gnews.yaml to include a yaml version of the schema previously stored in the schemas directory.
Deletes the existing schema files.

🚨 User Impact 🚨

No breaking changes; the output of the discover method is the same with the inline stream schemas as it was with the on-disk schema files.

Pre-merge Checklist

Unit & integration tests passing.
Code reviews completed
Documentation updated
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

girarda · 2022-12-13T02:35:12Z

airbyte-integrations/connectors/source-gnews/source_gnews/gnews.yaml

@@ -1,5 +1,109 @@
 version: "0.1.0"

+schemas:
+  search_stream_schema:
+    "$schema": http://json-schema.org/draft-07/schema#


do we need this line? I think it makes sense at the top level of a yaml file to specify the version of jsonschema to use to validate, but I doubt it does anything here

Ah, that can be removed. Just kept it in so I could diff the output of discover before and after the schema was included inline in the manifest.

@girarda just pushed the update removing these lines. Did you want to review the change?

clnoll · 2022-12-13T21:43:28Z

/test connector=connectors/source-gnews

🕑 connectors/source-gnews https://github.com/airbytehq/airbyte/actions/runs/3689604894
❌ connectors/source-gnews https://github.com/airbytehq/airbyte/actions/runs/3689604894
🐛 https://gradle.com/s/t7ssi5iez6fem

Build Failed

Test summary info:

Could not find result summary

brianjlai · 2022-12-13T22:06:59Z

airbyte-integrations/connectors/source-gnews/source_gnews/gnews.yaml

@@ -44,7 +146,12 @@ definitions:
    nullable: "{{ ','.join(config['nullable']) }}"
    from: "{{ stream_slice['start_time'] }}"
    to: "{{ stream_slice['end_time'] }}"
+  schema_loader:
+    type: StaticSchemaLoader


This should be InlineSchemaLoader correct?

😬 yep! Fixed.

brianjlai · 2022-12-13T22:12:50Z

airbyte-integrations/connectors/source-gnews/setup.py

@@ -6,7 +6,7 @@
 from setuptools import find_packages, setup

 MAIN_REQUIREMENTS = [
-    "airbyte-cdk~=0.1",
+    "airbyte-cdk~=0.13.2",


We should remove the patch version from here. Right now, this will cause us to only pull the airbyte-cdk from 0.13.x onward as per PEP 440. Whereas ideally we probably want to continue to get minor bumps as the CDK continues to be changed in a non-breaking way.

@brianjlai this change does rely on CDK version 0.13.2 or greater. WDYT about changing it to airbyte-cdk>=0.13.2?

I agree that we should definitely pin it to use 0.13 or above since it relies on a new feature, but the problem with explicitly marking it with the patch version 2 is that next time we build, we will only pull airbyte-cdk where minor version is 13. So in the next month if we progress to CDK 0.20.0, this connector will continue to be pinned at 0.13.x since it will only look for the latest patch. And I think based on PEP 440 ~= and >= are interchangeable. Basically I think we want to continue to get latest minors and specifying a patch in setup.py will inhibit that.

And actually given that this is a new non-breaking feature, we should probably also be bumping this to 0.14.0.

Got it, thanks for clarifying. Sounds like I should have bumped the CDK change to 0.14.0. Looks like that does exist now so I'll change this to use airbyte-cdk~=0.14.

ah sorry my bad, I was getting some PRs mixed. Up this was more that in hindsight that we should've minor version bumped the cdk schema change to 14 since it was a feature add. But regardless 0.14.0 is still fine since that was released earlier today.

brianjlai · 2022-12-13T22:13:57Z

airbyte-integrations/connectors/source-gnews/source_gnews/gnews.yaml

@@ -1,5 +1,107 @@
 version: "0.1.0"

+schemas:


nit, but can we move the schemas to the bottom of the file? i think the definitions and streams are likely to be the parts of the manifest that change most, so it's more intuitive to have them at the top.

ah sorry you are correct and my mistake. I thought we did some preparsing to allow for forward references even if defined after, but doesn't seem like we do, so it's fine to have this at the top.

No worries (sorry, deleted my comment because I thought maybe I'd spoken prematurely and was looking more closely at the issue). Going to write up an issue for updating the parser to handle forward references, since I do think that will make the yaml more readable.

sherifnada · 2022-12-15T00:32:45Z

airbyte-integrations/connectors/source-gnews/source_gnews/gnews.yaml

  search_stream:
+    schema_loader:
+      $ref: "*ref(definitions.schema_loader)"


fwiw I would find it more understandable if this said

schema_loader: type: InlineSchemaLoader

instead of the ref

clnoll · 2022-12-16T16:42:05Z

@girarda @brianjlai @sherifnada did you want to rereview this? It needs an approval to merge, since we don't want to wait on the potential forward reference change.

sherifnada · 2022-12-16T16:42:59Z

@clnoll no review needed from me! I just happened to see that one line where I made a comment :)

clnoll · 2022-12-16T17:23:48Z

/test connector=connectors/source-gnews

clnoll · 2022-12-16T19:13:07Z

/publish connector=connectors/source-gnews

🕑 Publishing the following connectors:
connectors/source-gnews
https://github.com/airbytehq/airbyte/actions/runs/3715878899

Connector	Did it publish?	Were definitions generated?
connectors/source-gnews	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

octavia-squidington-iv added connectors/source/gnews area/connectors Connector related issues labels Dec 12, 2022

clnoll force-pushed the source-gnews-use-inline-schema branch from 2a994f6 to 1d073ae Compare December 12, 2022 23:59

octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Dec 13, 2022

clnoll mentioned this pull request Dec 13, 2022

Enable low-code CDK users to specify schema in the manifest #20375

Merged

5 tasks

clnoll force-pushed the write-stream-schemas-to-manifest branch from f3d0a00 to 71782a8 Compare December 13, 2022 00:16

clnoll requested a review from a team as a December 13, 2022 00:16

octavia-squidington-iv added the CDK Connector Development Kit label Dec 13, 2022

clnoll force-pushed the write-stream-schemas-to-manifest branch from 5bbfe37 to 625d612 Compare December 13, 2022 01:05

clnoll force-pushed the source-gnews-use-inline-schema branch from 2624e82 to 14e30e9 Compare December 13, 2022 01:07

octavia-squidington-iv removed the CDK Connector Development Kit label Dec 13, 2022

girarda reviewed Dec 13, 2022

View reviewed changes

Base automatically changed from write-stream-schemas-to-manifest to master December 13, 2022 04:44

clnoll force-pushed the source-gnews-use-inline-schema branch from df526c5 to 4e8a721 Compare December 13, 2022 21:15

clnoll force-pushed the source-gnews-use-inline-schema branch from 4e8a721 to f326718 Compare December 13, 2022 21:45

brianjlai reviewed Dec 13, 2022

View reviewed changes

Update source-gnews manifest to use inline stream schemas

5d5e5a5

clnoll force-pushed the source-gnews-use-inline-schema branch from f326718 to 5d5e5a5 Compare December 14, 2022 00:25

clnoll and others added 4 commits December 13, 2022 19:42

Fix gnews.md table formatting

1678bb2

Use airbyte-cdk~=0.14

febe06f

Merge branch 'master' into source-gnews-use-inline-schema

10d7558

Merge branch 'master' into source-gnews-use-inline-schema

e39c2d9

sherifnada reviewed Dec 15, 2022

View reviewed changes

clnoll added 2 commits December 16, 2022 11:40

Update gnews changelog

c7620fe

Merge branch 'master' into source-gnews-use-inline-schema

a8b6946

girarda approved these changes Dec 16, 2022

View reviewed changes

Update Dockerfile io.airbyte.version

4a351c1

auto-bump connector version

56b0895

octavia-squidington-iii temporarily deployed to more-secrets December 16, 2022 19:34 — with GitHub Actions Inactive

octavia-squidington-iii temporarily deployed to more-secrets December 16, 2022 19:35 — with GitHub Actions Inactive

Merge branch 'master' into source-gnews-use-inline-schema

a69fbb6

clnoll temporarily deployed to more-secrets December 16, 2022 19:46 — with GitHub Actions Inactive

Merge branch 'master' into source-gnews-use-inline-schema

8221d2e

clnoll temporarily deployed to more-secrets December 16, 2022 22:44 — with GitHub Actions Inactive

clnoll temporarily deployed to more-secrets December 16, 2022 22:45 — with GitHub Actions Inactive

Merge branch 'master' into source-gnews-use-inline-schema

f9f1700

clnoll temporarily deployed to more-secrets December 19, 2022 14:39 — with GitHub Actions Inactive

clnoll temporarily deployed to more-secrets December 19, 2022 14:40 — with GitHub Actions Inactive

clnoll merged commit ddab3ce into master Dec 19, 2022

clnoll deleted the source-gnews-use-inline-schema branch December 19, 2022 15:32

github-actions bot mentioned this pull request Dec 27, 2022

Bump helm chart version reference to 0.43.9 #20892

Merged

This was referenced Jan 6, 2023

Bump Airbyte version from 0.40.26 to 0.40.27 #21092

Closed

Bump Airbyte version from 0.40.26 to 0.40.27 #21121

Closed

This was referenced Jan 6, 2023

Bump Airbyte version from 0.40.26 to 0.40.27 #21130

Closed

Bump Airbyte version from 0.40.26 to 0.40.27 #21135

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update source-gnews manifest to use inline stream schemas #20405

Update source-gnews manifest to use inline stream schemas #20405

clnoll commented Dec 12, 2022 •

edited

Loading

girarda Dec 13, 2022

clnoll Dec 13, 2022

clnoll Dec 13, 2022

clnoll commented Dec 13, 2022 •

edited by github-actions bot

Loading

brianjlai Dec 13, 2022

clnoll Dec 14, 2022

brianjlai Dec 13, 2022

clnoll Dec 14, 2022

brianjlai Dec 14, 2022

clnoll Dec 14, 2022

brianjlai Dec 14, 2022

brianjlai Dec 13, 2022

brianjlai Dec 14, 2022

clnoll Dec 14, 2022

sherifnada Dec 15, 2022

clnoll Dec 15, 2022

clnoll commented Dec 16, 2022

sherifnada commented Dec 16, 2022

clnoll commented Dec 16, 2022

clnoll commented Dec 16, 2022 •

edited by github-actions bot

Loading

Update source-gnews manifest to use inline stream schemas #20405

Update source-gnews manifest to use inline stream schemas #20405

Conversation

clnoll commented Dec 12, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Tests

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clnoll commented Dec 13, 2022 • edited by github-actions bot Loading

Build Failed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clnoll commented Dec 16, 2022

sherifnada commented Dec 16, 2022

clnoll commented Dec 16, 2022

clnoll commented Dec 16, 2022 • edited by github-actions bot Loading

clnoll commented Dec 12, 2022 •

edited

Loading

clnoll commented Dec 13, 2022 •

edited by github-actions bot

Loading

clnoll commented Dec 16, 2022 •

edited by github-actions bot

Loading