Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bmoric/remove dep connector worker #17977

Merged
merged 55 commits into from
Oct 18, 2022
Merged

Conversation

benmoriceau
Copy link
Contributor

What

Remove the dependency between the connectors and the worker

@benmoriceau benmoriceau requested a review from a team as a code owner October 13, 2022 22:09
@github-actions github-actions bot added the area/platform issues related to the platform label Oct 13, 2022
@benmoriceau benmoriceau temporarily deployed to more-secrets October 18, 2022 00:37 Inactive
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-pubsub
  • source-oracle
  • destination-oracle-strict-encrypt
  • destination-pulsar
  • destination-elasticsearch
  • destination-local-json
  • source-alloydb
  • source-cockroachdb
  • destination-postgres-strict-encrypt
  • destination-dev-null
  • source-clickhouse
  • source-postgres-strict-encrypt
  • destination-mssql-strict-encrypt
  • source-clickhouse-strict-encrypt
  • destination-databricks
  • source-elasticsearch
  • destination-keen
  • destination-bigquery
  • source-kafka
  • source-mssql-strict-encrypt
  • destination-azure-blob-storage
  • source-scaffold-java-jdbc
  • source-mongodb-v2
  • destination-tidb
  • destination-mysql
  • destination-cassandra
  • source-postgres
  • source-cockroachdb-strict-encrypt
  • destination-r2
  • destination-elasticsearch-strict-encrypt
  • source-db2
  • source-bigquery
  • source-tidb
  • source-mongodb-strict-encrypt
  • source-alloydb-strict-encrypt
  • source-sftp
  • source-mssql
  • source-db2-strict-encrypt
  • destination-aws-datalake
  • destination-oracle
  • source-oracle-strict-encrypt
  • destination-postgres
  • destination-mongodb-strict-encrypt
  • source-e2e-test
  • destination-mongodb
  • destination-csv
  • destination-meilisearch
  • destination-redshift
  • destination-mqtt
  • destination-gcs
  • destination-clickhouse
  • source-e2e-test-cloud
  • destination-clickhouse-strict-encrypt
  • destination-kafka
  • source-redshift
  • destination-jdbc
  • destination-dynamodb
  • destination-rockset
  • destination-redis
  • destination-mariadb-columnstore
  • destination-snowflake
  • destination-mssql
  • destination-s3
  • source-jdbc
  • destination-kinesis
  • destination-bigquery-denormalized
  • destination-mysql-strict-encrypt
  • source-mysql
  • source-snowflake
  • destination-e2e-test
  • source-mysql-strict-encrypt
  • destination-scylla

@benmoriceau benmoriceau temporarily deployed to more-secrets October 18, 2022 15:35 Inactive
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-postgres-strict-encrypt
  • source-postgres
  • destination-tidb
  • destination-dev-null
  • source-snowflake
  • destination-mssql-strict-encrypt
  • destination-bigquery
  • source-redshift
  • source-e2e-test
  • source-scaffold-java-jdbc
  • destination-postgres
  • destination-elasticsearch
  • destination-mysql
  • source-oracle-strict-encrypt
  • source-clickhouse-strict-encrypt
  • source-bigquery
  • destination-mqtt
  • source-oracle
  • destination-dynamodb
  • destination-kafka
  • destination-redis
  • destination-mysql-strict-encrypt
  • source-mongodb-v2
  • destination-mssql
  • source-jdbc
  • destination-elasticsearch-strict-encrypt
  • source-clickhouse
  • destination-aws-datalake
  • destination-cassandra
  • destination-bigquery-denormalized
  • source-tidb
  • destination-azure-blob-storage
  • destination-clickhouse
  • source-kafka
  • destination-gcs
  • destination-r2
  • source-sftp
  • destination-oracle
  • destination-mongodb
  • destination-mariadb-columnstore
  • destination-snowflake
  • source-elasticsearch
  • destination-s3
  • source-mssql
  • destination-pulsar
  • destination-pubsub
  • source-db2-strict-encrypt
  • destination-meilisearch
  • source-e2e-test-cloud
  • source-mssql-strict-encrypt
  • destination-postgres-strict-encrypt
  • destination-local-json
  • destination-keen
  • source-alloydb
  • source-db2
  • source-mongodb-strict-encrypt
  • destination-jdbc
  • destination-rockset
  • destination-databricks
  • source-alloydb-strict-encrypt
  • source-mysql-strict-encrypt
  • destination-redshift
  • destination-csv
  • destination-kinesis
  • source-cockroachdb-strict-encrypt
  • destination-scylla
  • destination-mongodb-strict-encrypt
  • destination-e2e-test
  • source-cockroachdb
  • source-mysql
  • destination-clickhouse-strict-encrypt
  • destination-oracle-strict-encrypt

@benmoriceau benmoriceau temporarily deployed to more-secrets October 18, 2022 15:50 Inactive
Copy link
Contributor

@jdpgrailsdev jdpgrailsdev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit: @benmoriceau We can also update the settings.gradle to move the airbyte-workers module into the platform section so that it is only built when the platform is built.

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-postgres-strict-encrypt
  • source-elasticsearch
  • source-clickhouse
  • destination-bigquery
  • destination-mongodb
  • destination-meilisearch
  • destination-bigquery-denormalized
  • destination-clickhouse-strict-encrypt
  • source-alloydb-strict-encrypt
  • destination-keen
  • destination-scylla
  • destination-pubsub
  • source-postgres-strict-encrypt
  • destination-snowflake
  • destination-mqtt
  • source-mysql
  • source-db2
  • destination-elasticsearch
  • destination-mysql-strict-encrypt
  • destination-elasticsearch-strict-encrypt
  • source-scaffold-java-jdbc
  • source-bigquery
  • source-alloydb
  • destination-csv
  • source-cockroachdb
  • source-clickhouse-strict-encrypt
  • destination-tidb
  • destination-mariadb-columnstore
  • destination-cassandra
  • source-jdbc
  • source-mysql-strict-encrypt
  • source-db2-strict-encrypt
  • source-snowflake
  • destination-clickhouse
  • destination-gcs
  • source-oracle
  • source-mongodb-v2
  • source-redshift
  • source-sftp
  • destination-aws-datalake
  • destination-mongodb-strict-encrypt
  • destination-s3
  • destination-r2
  • destination-kinesis
  • destination-postgres
  • source-postgres
  • destination-mssql
  • destination-databricks
  • destination-pulsar
  • destination-kafka
  • source-e2e-test-cloud
  • destination-local-json
  • destination-e2e-test
  • destination-dev-null
  • destination-mysql
  • source-mssql-strict-encrypt
  • destination-mssql-strict-encrypt
  • destination-rockset
  • destination-redis
  • destination-jdbc
  • destination-oracle
  • destination-redshift
  • source-tidb
  • destination-azure-blob-storage
  • source-kafka
  • source-mssql
  • destination-oracle-strict-encrypt
  • source-oracle-strict-encrypt
  • source-mongodb-strict-encrypt
  • destination-dynamodb
  • source-e2e-test
  • source-cockroachdb-strict-encrypt

@benmoriceau benmoriceau temporarily deployed to more-secrets October 18, 2022 16:11 Inactive
@benmoriceau
Copy link
Contributor Author

:shipit: @benmoriceau We can also update the settings.gradle to move the airbyte-workers module into the platform section so that it is only built when the platform is built.

That's a good point, I will do it. It turns out that it was needed to move more class. Will do it once I have a green build.

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-tidb
  • destination-gcs
  • destination-cassandra
  • destination-mongodb
  • destination-keen
  • source-db2
  • destination-snowflake
  • destination-jdbc
  • source-oracle
  • destination-mssql-strict-encrypt
  • destination-bigquery-denormalized
  • source-alloydb
  • source-alloydb-strict-encrypt
  • source-sftp
  • destination-redshift
  • destination-bigquery
  • source-redshift
  • destination-redis
  • destination-mqtt
  • destination-pubsub
  • source-tidb
  • destination-meilisearch
  • source-postgres
  • source-mongodb-v2
  • source-bigquery
  • source-scaffold-java-jdbc
  • source-cockroachdb
  • source-clickhouse-strict-encrypt
  • destination-oracle-strict-encrypt
  • destination-mongodb-strict-encrypt
  • source-mongodb-strict-encrypt
  • destination-postgres
  • source-kafka
  • destination-s3
  • destination-postgres-strict-encrypt
  • source-cockroachdb-strict-encrypt
  • source-jdbc
  • destination-csv
  • source-e2e-test
  • source-elasticsearch
  • destination-e2e-test
  • destination-kafka
  • destination-azure-blob-storage
  • source-e2e-test-cloud
  • destination-r2
  • destination-scylla
  • destination-mysql-strict-encrypt
  • destination-dynamodb
  • destination-dev-null
  • destination-clickhouse-strict-encrypt
  • destination-mariadb-columnstore
  • source-mssql-strict-encrypt
  • destination-aws-datalake
  • destination-pulsar
  • destination-rockset
  • destination-kinesis
  • source-snowflake
  • destination-mysql
  • destination-mssql
  • source-clickhouse
  • source-mysql
  • source-postgres-strict-encrypt
  • source-oracle-strict-encrypt
  • destination-clickhouse
  • destination-elasticsearch
  • source-db2-strict-encrypt
  • source-mysql-strict-encrypt
  • source-mssql
  • destination-local-json
  • destination-oracle
  • destination-databricks
  • destination-elasticsearch-strict-encrypt

@benmoriceau benmoriceau temporarily deployed to more-secrets October 18, 2022 16:37 Inactive
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-tidb
  • destination-tidb
  • destination-databricks
  • destination-kinesis
  • destination-e2e-test
  • source-alloydb-strict-encrypt
  • source-clickhouse-strict-encrypt
  • destination-mongodb-strict-encrypt
  • destination-postgres-strict-encrypt
  • source-oracle-strict-encrypt
  • source-kafka
  • destination-jdbc
  • source-mssql-strict-encrypt
  • destination-redis
  • source-elasticsearch
  • destination-mariadb-columnstore
  • source-bigquery
  • destination-mssql
  • source-mongodb-strict-encrypt
  • destination-azure-blob-storage
  • destination-pubsub
  • source-mysql-strict-encrypt
  • source-jdbc
  • destination-oracle
  • destination-bigquery-denormalized
  • destination-mqtt
  • destination-clickhouse
  • source-alloydb
  • destination-aws-datalake
  • source-e2e-test
  • destination-redshift
  • destination-elasticsearch
  • destination-elasticsearch-strict-encrypt
  • destination-kafka
  • destination-keen
  • destination-dynamodb
  • destination-pulsar
  • source-cockroachdb-strict-encrypt
  • destination-rockset
  • source-snowflake
  • source-mssql
  • source-postgres-strict-encrypt
  • source-sftp
  • destination-mysql-strict-encrypt
  • destination-cassandra
  • destination-r2
  • destination-mongodb
  • destination-bigquery
  • destination-s3
  • destination-mysql
  • source-oracle
  • destination-clickhouse-strict-encrypt
  • destination-scylla
  • source-postgres
  • source-clickhouse
  • destination-gcs
  • destination-postgres
  • source-cockroachdb
  • destination-csv
  • source-db2-strict-encrypt
  • destination-dev-null
  • destination-mssql-strict-encrypt
  • source-mongodb-v2
  • source-redshift
  • destination-meilisearch
  • destination-snowflake
  • source-e2e-test-cloud
  • destination-oracle-strict-encrypt
  • source-scaffold-java-jdbc
  • source-mysql
  • source-db2
  • destination-local-json

@benmoriceau benmoriceau temporarily deployed to more-secrets October 18, 2022 16:49 Inactive
@benmoriceau benmoriceau temporarily deployed to more-secrets October 18, 2022 17:33 Inactive
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-mssql-strict-encrypt
  • destination-dev-null
  • destination-pubsub
  • destination-mysql-strict-encrypt
  • destination-e2e-test
  • destination-r2
  • destination-mongodb
  • destination-s3
  • source-e2e-test-cloud
  • source-postgres
  • source-scaffold-java-jdbc
  • destination-jdbc
  • destination-scylla
  • source-mongodb-strict-encrypt
  • source-mysql
  • destination-tidb
  • destination-databricks
  • destination-mariadb-columnstore
  • source-bigquery
  • source-clickhouse
  • destination-elasticsearch-strict-encrypt
  • source-cockroachdb
  • destination-bigquery-denormalized
  • source-tidb
  • source-alloydb
  • destination-postgres-strict-encrypt
  • source-redshift
  • destination-dynamodb
  • source-db2-strict-encrypt
  • source-jdbc
  • source-cockroachdb-strict-encrypt
  • source-kafka
  • destination-azure-blob-storage
  • destination-local-json
  • destination-oracle-strict-encrypt
  • source-sftp
  • source-postgres-strict-encrypt
  • destination-pulsar
  • destination-postgres
  • source-clickhouse-strict-encrypt
  • source-e2e-test
  • destination-snowflake
  • destination-elasticsearch
  • destination-mssql-strict-encrypt
  • source-db2
  • destination-clickhouse-strict-encrypt
  • source-mysql-strict-encrypt
  • destination-cassandra
  • destination-kinesis
  • source-alloydb-strict-encrypt
  • source-elasticsearch
  • source-oracle
  • destination-keen
  • destination-redis
  • destination-mqtt
  • destination-mssql
  • destination-bigquery
  • destination-kafka
  • source-mongodb-v2
  • destination-mongodb-strict-encrypt
  • destination-redshift
  • destination-aws-datalake
  • destination-meilisearch
  • destination-clickhouse
  • destination-csv
  • destination-oracle
  • source-mssql
  • source-snowflake
  • destination-mysql
  • source-oracle-strict-encrypt
  • destination-rockset
  • destination-gcs

@benmoriceau benmoriceau temporarily deployed to more-secrets October 18, 2022 18:11 Inactive
@benmoriceau benmoriceau marked this pull request as ready for review October 18, 2022 18:50
@benmoriceau benmoriceau merged commit adf2c5f into master Oct 18, 2022
@benmoriceau benmoriceau deleted the bmoric/remove-dep-connector-worker branch October 18, 2022 18:51
letiescanciano added a commit that referenced this pull request Oct 19, 2022
* master: (304 commits)
  Bump helm chart version reference to 0.40.27 (#18152)
  Bump helm chart version reference to 0.40.26 (#18094)
  Update deployment.yaml (#18151)
  Publishes Postgres, MySQL, MSSQL source with changes from #18041 (#18086)
  Fix minor DBT Cloud Errors. (#18147)
  Sentry Integration : Stop reporting all non system-error error types. (#18133)
  Docs: Fix backoff stategy docs (#18143)
  🐛 Destination GCS: Fix error logs to log 'Gcs' rather than 'AWS' (#17901)
  Add openAPI spec for Connector Builder Server (#17535)
  Alex/mvp UI for dbt cloud integration (#18095)
  increased timeout for sat tests (#18128)
  Bmoric/remove dep connector worker (#17977)
  `recordsRead` should be a long (#18123)
  doc_update_oath_issue_gsc (#17967)
  🎉 Source Zendesk Chat: engagements data fix infinity looping + gradlew format (#18121)
  🐛 Source Zendesk Chat: engagements data fix infinity looping (#17745)
  Custom APM Tracing (#17947)
  11679 BigQuery-Denormalized Destination: improve code coverage (#17827)
  increased timeout for sat tests (#18114)
  docs: clarify language (#18090)
  ...
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
* test [ci skip]

* Autogenerated files

* Add missing annotation

* Remove unused json2Schema block from worker

* Move tess

* Missing deps and format

* Fix test build

* TMP

* Add missing dependencies

* PR comments

* Tmp

* [ci skip] Tmp

* Fix acceptance test and add the seed dependency

* Fix build

* For diff

* tmp

* Build pass

* make the worker to be  on the platform only

* fix setting.yaml

* remove dependency

* Fix pmd

* Fix Cron

* Add chart

* Fix cron

* Fix server build.gradle

* Fix jar conflict

* PR comments

* Add cron micronaut environemnt

* Update required build path

* remove all dependencies on worker

* Typo

* Add gradle exclusions

* Add exclusion to avoid conflict

* more exclusion

* Move more class

* Exclude worker from the general build

* Fix connector build
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants