[WIP] run all CATs locally #24254

erohmensing · 2023-03-20T21:43:48Z

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

…ally

…or first

github-actions · 2023-03-20T21:47:39Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (1)

Connector	Version	Changelog	Publish
`source-railz`	`0.1.1`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

…il_on_extra_columns

…codes. Fix max_concurrency flag

…onnector builder module (#24173) * wip * fix unit test * fix other unit test * format * reset * format * missing unit test * yield a LogMessage on error * format * format * fix unit tests * yield a trace message instead of a log message * format * fix bad merge

* #1681 source file: do not read whole file on check and discover * #1681 source file: upd changelog * #1681 source file: add allowed hosts * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* #1692 source linkedin-pages remove authSpec * #1692 source linkedin-pages: upd changelog * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* added view and storage to pages, updated SAT * added unittests * added changelog, fixed type in docs * added checking for stream Audit to skip it if user doesn't have permissions * refactored plan validation * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* [23172] Destination-snowflake: added host validation by a pattern on UI

* fixed spec, added expected records, enabled high strictness, added unit tests * updated docs, formatting * Fix SAT * Automated Change * Fix typo in docs * auto-bump connector version --------- Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com> Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com> Co-authored-by: lazebnyi <lazebnyi@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* Update current schema vs catalog validation to omit airbyte metadata fields * Bumped versions for postgres, mssql, and mySQL + changelog * Bumped dockerfile version of strict encrypt * Manually generate definitions for mysql * Manually generate definitions for mssql --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* destination-s3-glue: Fix data type for emitted at column The `_ab_emitted_at` column is actually a numeric type and not a string. This updates the generated schema to treat it as the proper type. * fix tests * rollback seed dest spec update * auto-bump connector version --------- Co-authored-by: Tobias Macey <tmacey@mit.edu> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* add grouping and collapsing fields to postgres source * add auth group to github source connector * revert postgres field order changes and adjust group of schemas field * inject group into ssh tunnel spec for postgres only, through overloaded methods * Automated Change * bump Dockerfile versions and update changelogs * bump strict encrypt version as well * fix postgres acceptance test * fix acceptance test again * fix all postgres acceptance tests * add newline * undo other changes to postgres readme file * add security group to tunnel_method in expected_spec.json * bump version of strict encrypt * manually bump versions in seed files --------- Co-authored-by: lmossman <lmossman@users.noreply.github.com>

* #1689 source chargebee: ignore http 404 errors * #1689 source chargebee: upd changelog * merge two handlers * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

This reverts commit 740d629.

Co-authored-by: Topher Lubaway <asimplechris@gmail.com>

CLAassistant · 2023-03-22T22:34:57Z

All committers have signed the CLA.

erohmensing · 2023-03-22T22:35:26Z

Yikes ain't that a big failed rebase

erohmensing added 17 commits March 12, 2023 23:58

Pass argument along, add test that should pass and test that should fail

f596e3f

Add tests with additionalProperties

c88305e

Set additionalproperties=false when not set|

1f91a59

Parametrize test cases

a8ef276

Make the behavior 'optional'

3e5cf1f

Fix parametrization for all combinations

a7eb6f6

Improve CI credentials README and rename param

2545f8f

Update naming to be clearer about columns only

356b12a

record_has_unexpected_field > record_has_unexpected_column

a53c31a

Hacking the CAT dockerfile and run script to test my changes specific…

646b372

…ally

First crack at running CAT on all connectors

fa72e00

Write during instead of after all tests

be92aef

Async-ify it

96eb3fc

Add ability to define max concurrency

9fc650c

Write successes

ede50bd

ci_credentials: fix overwriting 'data' before getting nextPageToken

cea154a

Adjustible num_semaphores, check to make sure it's an airbyte connect…

a2569e4

…or first

erohmensing changed the base branch from master to ella/no-extra-fields March 20, 2023 21:44

erohmensing and others added 4 commits March 20, 2023 22:02

Automated Change

63a273b

Make create_issues and create_prs more configurable, add issue for fa…

f729dc2

…il_on_extra_columns

Add ability to pass in sources as a list or from a txt file

1aa9644

Add logs to issue, make project nullable

6535110

erohmensing mentioned this pull request Mar 21, 2023

ci_credentials: fix overwriting 'data' before getting nextPageToken #24265

Merged

erohmensing and others added 6 commits March 21, 2023 02:30

Automated Change

0e0491d

Migrate multiple connectors

5257d07

Add cli args

62e51c1

use ruamel.yaml to preserve ordering

601f07b

Separate config loading from config migration

69e6c61

Add ability to pass in lists of sources to test. Sort output by exit …

fef08e5

…codes. Fix max_concurrency flag

girarda and others added 21 commits March 22, 2023 17:34

🤖 Bump patch version of Airbyte CDK

a56d83e

Source linkedin-pages: remove authSpec (#24352)

3b4d365

* #1692 source linkedin-pages remove authSpec * #1692 source linkedin-pages: upd changelog * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

Source Xero: releaseStage "beta" (#24231)

1f81ed4

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Source Trello: releaseStage "beta" (#24200)

c2d536c

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Source Pipedrive: releaseStage "beta" (#23930)

6fc7eb4

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

fix path (#24290)

a845db8

Fixed Typo in README (#23591)

a92ed7c

Add missing type hints (#24345)

ab3fddc

Destination-snowflake: added host validation by pattern on UI (#23904)

e8b4b55

* [23172] Destination-snowflake: added host validation by a pattern on UI

Revert "Azure Synapse Compatibility (#18294)" (#24371)

4f1b306

This reverts commit 740d629.

Added Relevance AI widget to documentation (#24289)

3193fbf

Co-authored-by: Topher Lubaway <asimplechris@gmail.com>

Default to testing only beta and GA connectors

adffde9

Always write test output when available

5551523

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit normalization labels Mar 22, 2023

erohmensing changed the base branch from ella/no-extra-fields to master March 22, 2023 22:34

erohmensing closed this Mar 22, 2023

erohmensing deleted the ella/run-all-CATS branch March 22, 2023 22:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] run all CATs locally #24254

[WIP] run all CATs locally #24254

erohmensing commented Mar 20, 2023

github-actions bot commented Mar 20, 2023 •

edited

Loading

✅ Sources (1)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

CLAassistant commented Mar 22, 2023 •

edited

Loading

erohmensing commented Mar 22, 2023

[WIP] run all CATs locally #24254

[WIP] run all CATs locally #24254

Conversation

erohmensing commented Mar 20, 2023

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

github-actions bot commented Mar 20, 2023 • edited Loading

Affected Connector Report

✅ Sources (1)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

CLAassistant commented Mar 22, 2023 • edited Loading

erohmensing commented Mar 22, 2023

github-actions bot commented Mar 20, 2023 •

edited

Loading

CLAassistant commented Mar 22, 2023 •

edited

Loading