Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 Fixed test for gcs, csv, json-local, mongodb and meilisearch destination #6134

Merged
merged 5 commits into from
Sep 17, 2021
Merged

🐛 Fixed test for gcs, csv, json-local, mongodb and meilisearch destination #6134

merged 5 commits into from
Sep 17, 2021

Conversation

andriikorotkov
Copy link
Contributor

@andriikorotkov andriikorotkov commented Sep 16, 2021 •
edited
Loading

What

Fixed test for gcs, csv, json-local, mongodb and meilisearch destination

How

Updated test for csv, json-local and gcs.
Added emitted_at field for meilisearch destination.
Changed N copies from 1000 to 400 for testSyncWithLargeRecordBatch test - (this test hangs with a lot of entries for mongodb destination)

Recommended reading order

  1. x.java
  2. y.python

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/SUMMARY.md
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions
  • Connector added to connector index like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions
  • Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • If adding a new generator, add it to the list of scaffold modules being tested
  • The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • Documentation which references the generator is updated as needed.

@github-actions github-actions bot added the area/connectors Connector related issues label Sep 16, 2021
@andriikorotkov andriikorotkov marked this pull request as ready for review September 16, 2021 18:12
@andriikorotkov
Copy link
Contributor Author

andriikorotkov commented Sep 16, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-meilisearch

🕑 connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/1242704329
✅ connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/1242704329
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets September 16, 2021 18:16 Inactive
davinchia
davinchia approved these changes Sep 17, 2021
View reviewed changes
Copy link
Contributor

@davinchia davinchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, 2 small comments to make sure we know what is happening.

Other things:

  • can we run all the tests for the modified connectors on the PR branch to confirm the modifications are right?
  • can you follow the Updating a Connector checklist on the PR description? one thing we are missing here is the connector changelog update.

...rc/main/java/io/airbyte/integrations/standardtest/destination/DestinationAcceptanceTest.java Show resolved Hide resolved
...ch/src/main/java/io/airbyte/integrations/destination/meilisearch/MeiliSearchDestination.java Show resolved Hide resolved
...va/io/airbyte/integrations/destination/meilisearch/MeiliSearchDestinationAcceptanceTest.java Show resolved Hide resolved
@andriikorotkov
Copy link
Contributor Author

andriikorotkov commented Sep 17, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-csv

🕑 connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/1244654185
✅ connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/1244654185
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets September 17, 2021 07:57 Inactive
@andriikorotkov
Copy link
Contributor Author

andriikorotkov commented Sep 17, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-json-local

🕑 connectors/destination-json-local https://github.com/airbytehq/airbyte/actions/runs/1244724382
❌ connectors/destination-json-local https://github.com/airbytehq/airbyte/actions/runs/1244724382
🐛

@jrhizor jrhizor temporarily deployed to more-secrets September 17, 2021 08:20 Inactive
@etsybaev etsybaev mentioned this pull request Sep 17, 2021
Merged
33 tasks
@andriikorotkov
Copy link
Contributor Author

andriikorotkov commented Sep 17, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1244823082
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1244823082
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets September 17, 2021 08:54 Inactive
@andriikorotkov
Copy link
Contributor Author

andriikorotkov commented Sep 17, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-mongodb

🕑 connectors/destination-mongodb https://github.com/airbytehq/airbyte/actions/runs/1244865222
✅ connectors/destination-mongodb https://github.com/airbytehq/airbyte/actions/runs/1244865222
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets September 17, 2021 09:08 Inactive
@andriikorotkov
Copy link
Contributor Author

andriikorotkov commented Sep 17, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-local-json

🕑 connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/1244942112
✅ connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/1244942112
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets September 17, 2021 09:32 Inactive
@andriikorotkov
Copy link
Contributor Author

@davinchia, I corrected all remarks and tested all destinations. Can I merge this pull request?

@davinchia
Copy link
Contributor

@andriikorotkov go for it!

@andriikorotkov andriikorotkov merged commit 0a027a1 into master Sep 17, 2021
@andriikorotkov andriikorotkov deleted the akorotkov/test-fix branch September 17, 2021 11:08
@andriikorotkov
Copy link
Contributor Author

andriikorotkov commented Sep 17, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-meilisearch

Error: No ref found for: akorotkov/test-fix

@andriikorotkov andriikorotkov restored the akorotkov/test-fix branch September 17, 2021 12:37
@andriikorotkov
Copy link
Contributor Author

andriikorotkov commented Sep 17, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-meilisearch

🕑 connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/1245489217
✅ connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/1245489217

@jrhizor jrhizor temporarily deployed to more-secrets September 17, 2021 12:39 Inactive
@jrhizor jrhizor mentioned this pull request Sep 18, 2021
Merged
@swyxio swyxio deleted the akorotkov/test-fix branch October 12, 2022 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexandr-shegeda alexandr-shegeda alexandr-shegeda approved these changes

@davinchia davinchia davinchia approved these changes

@etsybaev etsybaev etsybaev approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues connectors/destination/local-csv connectors/destination/local-json connectors/destination/meilisearch connectors/destinations-api connectors/destinations-files
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@andriikorotkov @davinchia @etsybaev @alexandr-shegeda @jrhizor @karinakuz