🎉 New Source: Yotpo [Low code CDK] #25532
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tor" This reverts commit 6615288b6d2441d9fdde2d927a5eb77b5d788442.
|
Please add hackathon-developer-week label |
|
Conflicts resolved :) |
|
@marcosmarxm conflicts resolved from #25145 |
|
/test connector=connectors/source-yopto
Build FailedTest summary info: |
|
Fixed some expected records as the platform is auto rejecting some of the seeded content for some reason
|
|
/test connector=connectors/source-yopto
Build FailedTest summary info: |
marcosmarxm
requested changes
Apr 27, 2023
airbyte-integrations/connectors/source-yotpo/source_yotpo/schemas/email_analytics.json
Outdated
Show resolved
Hide resolved
|
/test connector=connectors/source-yotpo
|
|
Comments resolved.
|
|
/test connector=connectors/source-yotpo
Build PassedTest summary info: |
marcosmarxm
approved these changes
May 1, 2023
|
/publish connector=connectors/source-yotpo
if you have connectors that successfully published but failed definition generation, follow step 4 here |
marcosmarxm
added a commit
to natalia-miinto/airbyte
that referenced
this pull request
Jun 8, 2023
* Initial commit * Add stream top_reviews * Add new stream: privacy_data * add new stream: privacy user, refactor * Add new streams, update docs, update definitions * EOF * Update spec.yaml * fix date_from, optimize manifest, update docs * delete privacy streams, refactor webhooks * add pagination * add streams raw_data, email_analytics * add streams unsubscribers, webhook_events * Fix pagination * revert doc * Merge branch 'yotpo' of https://github.com/btkcodedev/airbyte into yotpo * Revert "remove top_reviews and reviews_bottomline, Make common paginator" This reverts commit 6615288b6d2441d9fdde2d927a5eb77b5d788442. * remove streams top_reviews, reviews_bottomline * make common base and extractor * refactor, Checkpoint for full test passed * refactor base, requester, paginator * remove extra paginator property * Update oss_catalog.json * fix conflicts * fix conflict oss * fix conflict oss * Add stream top_reviews * Add new stream: privacy_data * add new stream: privacy user, refactor * Add new streams, update docs, update definitions * EOF * Source Stripe: expand tiers when syncing plans (airbytehq#19906) * Expand tiers when syncing plans * Update Plans stream schema * Re-bump * Bump * Remove string as a plan schema * bump * Update changelog --------- Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> * airbyte-ci: improve metadata validation perfs (airbytehq#25107) * metadata service: upload metadata files to gcs (airbytehq#25115) * metadata: cleanup workflows (airbytehq#25105) * Source Stripe: generate seed source file (airbytehq#25113) * Expand tiers when syncing plans * Update Plans stream schema * Re-bump * Bump * Remove string as a plan schema * bump * Update changelog * update source spec * Automated Change --------- Co-authored-by: Visnu Pitiyanuvath <visnup@gmail.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com> * Connector builder: set pages and slices limits (airbytehq#25121) * Set limits * refactor and add unit tests * Update as per comments * 🤖 Bump patch version of Airbyte CDK * Bump Airbyte version from 0.43.1 to 0.43.2 * source-postgres works with bigger datasets (airbytehq#25138) * create metadata files for all connectors (airbytehq#24964) * fix date_from, optimize manifest, update docs * Update spec.yaml * 🚨🚨 Source Amplitude: migrate to YAML (airbytehq#22362) * Source Amplitude: migrate to YAML * Source Amplitude: enable high strictness level * Source Amplitude: update docs * Source Amplitude: update cursor field + expected records * Source Amplitude: logging refactor * Source Amplitude: refactor extractors * Source Amplitude: refactor extractors * Source Amplitude: migrate to beta low-code * Source Amplitude: add docstring * Source Amplitude: fix unit tests * Source Amplitude: update expected records * Source Amplitude: fix response filter action * Source Amplitude: rename typo * Source Amplitude: fix migration * Source Amplitude: update docs * Source Amplitude: fix migration * Source Amplitude: update expected recrods * Source Amplitude: bump manifest version * Automated Change * Source Amplitude: Update http -> https * Source Amplitude: tests fix + pin cdk version * auto-bump connector version --------- Co-authored-by: artem1205 <artem1205@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source Zendesk Support: update expected records (airbytehq#25058) * Source Zendesk Support: skip stream * Add expected records for tags stream * Automated Change --------- Co-authored-by: arsenlosenko <arsenlosenko@users.noreply.github.com> * Source Surveymonkey: Fix spec.json required fields and update schema for surveys and survey_responses (airbytehq#25080) * Fixes required fields on SurveyMonkey UI * fix integration test schema update for surveys and survey response * fix expected records * update dockerfile and docs * auto-bump connector version --------- Co-authored-by: Ali Hassan <alihassan01264@gmail.com> Co-authored-by: Ali Hassan <62539376+Alihassanc5@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Add additional destination-bigquery datacenters (airbytehq#25122) * Add additional bigquery datacenters * bump dockerfile * revert source BQ bump * docs * builds depend on normalization now... * no -base * auto-bump connector version * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump versions in atelier dockerfile (airbytehq#25125) * Bump versions in atelier dockerfile * fix new version * temporarily skip build and publish * reset * skip some steps * Revert "skip some steps" This reverts commit e034d10. * Source: Amazon Ads - Add pk validator for reports when expected pk is not returned (airbytehq#25146) * Add pk validator for reports when expected pk is not returned * Update PR number * Fix formating * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🎉Source Twilio: New Stream Conversation Participants (airbytehq#23995) * Source Twilio: New Stream Conversation Participants * fix: format schema and fix to incremental stream * refactor: Add newline * fix: remove unwanted file * fix: schema fixes * fix: Avoid acceptance test bypass for full testing * fix: Fixed incremental behaviour, Updated docs, Updated VersionTag * fix: Fixed incremental behaviour, Updated docs, Updated VersionTag * fix: Fixed syntax, Updated docs * fix: schema, update docs * fix: Changed parent stream, expected record fixes, connector version bump * chore: EOL fix * fix expected records for conversations participant * Update Twilio version to 0.5.0 * update twilio version to 0.5.0 * remove conversation_participants from state file * add lines and add eof * update twilio version to 0.5.0 in source spec * removed conversation_participants from abnormal state * remove extra lines * update doc connector version * update json schema draft version * fix doc conflict * add expected records * auto-bump connector version --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source Iterable: update expected records (airbytehq#25144) * Source Facebook Pages: update insight metrics request params (airbytehq#25143) * Source Facebook Pages: update insight metrics request params * Update Dockerfile and changelog * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * fix strict-encrypt metadata dockerRepository (airbytehq#25159) * Source Slack: update expected records (airbytehq#25150) * dagger: upgrade to 0.5.0 (airbytehq#25171) * dagger: upgrade to 0.5.0 * clean class ConnectorPackageInstall(Step): * Fixed missing punctuations (airbytehq#25151) * Update README.md I gave a better description to the Airbyte Cloud section, converted a few passive words to active, and added a missing comma. * Update README.md I noticed several punctuation marks were missing, so I fixed them. I also want to use this opportunity to draw get the attention of the docs team. I would love to join Airbyte as a technical writer and be a part of the team. I am more than willing to handle these tasks, even if the organization does not have the current budget to onboard a new technical writer. Also, I filled the form to participate in the Airbyte Writer Program. I hope the team reviews my topic and allow me contribute to the data insight and tutorial articles. Cheers! * Update README.md --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> * refactor source harness and GitHub action to add source-mysql (airbytehq#25036) * refactor to add source-mysql * refactor to add source-mysql * refactor to add source-mysql * test * Automated Change * fix if condition * try multiline * new db * add mysql catalogs --------- Co-authored-by: rodireich <rodireich@users.noreply.github.com> * change google analytics dockerfile order to pip install source (airbytehq#23082) Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> * Always show client cert and key (airbytehq#25131) * make CA cert and key always show * make same change for mysql * update readmes * update strict-encrypt version and expected spec * fix expected spec for mysql strict encrypt * auto-bump connector version * manually bump seed for postgres and mysql --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🐛 Source File: fix column data types for numerical values (airbytehq#23759) * Fix column data types for numerical values * Update unit tests * Update Dockerfile version * update dockerfile version for source-file-secure * Update source-file in seed file * Update doc * Update source_specs.yaml --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> * `connectors.md` gains Github labels for connector issues (airbytehq#25182) * `connectors.md` gains github label * Automated Commit - Formatting Changes * comment * Automated Commit - Formatting Changes * Automated Change --------- Co-authored-by: evantahler <evantahler@users.noreply.github.com> * Housekeeping: Remove unused workflows. (airbytehq#25191) This is no longer used with Move's move to Zenhub. * increase sentry heartbeat - 43200->64800 (airbytehq#25185) * increase sentry heartbeat - 43200->64800 * Automated Change --------- Co-authored-by: colesnodgrass <colesnodgrass@users.noreply.github.com> * Bump Airbyte version from 0.43.2 to 0.44.0 * dagger CI: auth to dockerhub (airbytehq#25196) * auth to dockerhub sometime * map docker hub username and password to env var * Revert "auth to dockerhub sometime" This reverts commit d82bf88. * Revert "Revert "auth to dockerhub sometime"" This reverts commit bc9bc61. * login to dockerhub * login to dockerhub * revert changes * airbyte-ci: orchestrate the upload of modified metadata.yaml files to GCS (airbytehq#25044) * Source Hubspot: update expected records (airbytehq#25152) * Source Hubspot: update expected records * Update expected records * Source Github: update expected records (airbytehq#25148) * Source Github: update expected records * Update expected records * Move teams expected records to previous place * Source Pipedrive: update expected records (airbytehq#25199) * Source Pipedrive: update expected records * Update expected records * Removed expected record that is not needed * Source Zendesk Support: update expected records (airbytehq#25206) * Source Twilio: update expected records (airbytehq#25200) * connector builder: Emit message at start of slice (airbytehq#25180) * Move condition for yielding the slice message to an overwritable method * Automated Commit - Formatting Changes * yield the slice log messages * same for incremental * refactor * Revert "refactor" This reverts commit c594365. * move flag from factory to source * set the flag * remove debug print * halfmock * clean up * Add a test for a single page * Add another test * Pass the flag * rename --------- Co-authored-by: girarda <girarda@users.noreply.github.com> * 🤖 Bump patch version of Airbyte CDK * CI checks for airbytehq#23868 (airbytehq#24717) * feat(source-bing-ads): add dimensional columns for AccountPerformanceReport * feat(source-bing-ads): add dimensional columns for AdGroupPerformanceReport * feat(source-bing-ads): add dimensional columns for AdPerformanceReport * feat(source-bing-ads): add dimensional columns for KeywordPerformanceReport * feat(source-bing-ads): add dimensional columns for CampaignPerformanceReport * docs(source-bing-ads): update changelog * chore(source-bing-ads): bump version in Dockerfile * test(source-bing-ads): allow extra fields for basic read * Revert "test(source-bing-ads): allow extra fields for basic read" This reverts commit 961f0d6. * test(source-bing-ads): ignore community contributed fields * fix: updated expected_records of report streams * auto-bump connector version * Automated Change --------- Co-authored-by: Rachel RIZK <rachel.rizk@backmarket.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: sajarin <sajarin@users.noreply.github.com> * Source Facebook Marketing: fix data retention period validation (airbytehq#25204) * airbytehq#1678 source Facebook Marketing: fix data retention period validation * airbytehq#1678 source fb marketing: add validation for the date range in the future * airbytehq#1678 source fb marketing: another fix * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Add OSS getting started redirect (airbytehq#25213) * Add is_optional flag to connector tests (airbytehq#25211) * Run doc deploys on docosaurus changes (airbytehq#25215) Co-authored-by: Topher Lubaway <asimplechris@gmail.com> * Fix manifest_declarative_source + add unit tests (airbytehq#25217) * Fix + unit test * Add a test with pagination * Add a test with partition router * Make sure _fetch_next_page is called with the right arguments * Automated Commit - Formatting Changes * pagination with partitions * refactor * clean up * format --------- Co-authored-by: girarda <girarda@users.noreply.github.com> * 🤖 Bump patch version of Airbyte CDK * Document the config for docker workspace cleanup (airbytehq#25210) * delete privacy streams, refactor webhooks * add pagination * add streams raw_data, email_analytics * add streams unsubscribers, webhook_events * Fix pagination * revert doc * Merge branch 'yotpo' of https://github.com/btkcodedev/airbyte into yotpo * Revert "remove top_reviews and reviews_bottomline, Make common paginator" This reverts commit 6615288b6d2441d9fdde2d927a5eb77b5d788442. * remove streams top_reviews, reviews_bottomline * make common base and extractor * refactor, Checkpoint for full test passed * refactor base, requester, paginator * remove extra paginator property * Cleanup some logging in connectors (airbytehq#25227) * 🎉 New Source: Teradata (airbytehq#24221) * add teradata template * add teradata environment client * add teradata source connector * add docs * fix lowercase letter * fix compilation error & config test * fix timestamp * generate seed --------- Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com> * Add metadata gcs sensor to Dagster (airbytehq#25055) * Add solution skeleton * Add metadata blob resource * Wire up sensor * Fix report * Materialize metadata definitions from folder * Move metadata to own file * Run black * Add PR comments * Remove dockerhub from action * 🎉Source Stripe: New Stream - Subscription Schedule (airbytehq#23624) * bump connector version * Source stripe: New Stream Subscription Schedule * fix: schema fixes, updated docs, updated versionTag * fix: test * Update stripe.md * fix: Add empty stream * schema fix, version bump, source update * unwanted line fix * Resolve comments * fix schemas * add expected records and update schema * 🎉Source Twilio: New Stream Conversation Messages (airbytehq#22875) * bump connector version * Source Twilio: New Stream Conversation Messages * fix: Adding correction to syntax * fix: schema errors, incremental stream fix * fix: schema, updated docs, dockerVersion, removed unwanted files * Update twilio.md * fix schema, add expected records and update stream * remove extar empty line * format files * fix schema for conversation messages * fix conversation participants schema * remove conversation participants from incr ctg * remove unused logging * update schema * Update source_definitions.yaml * Update jira.md * Update expected_records.jsonl * Update twilio.md * Update twilio.md * Update twilio.md * auto-bump connector version --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * remove json schema from integration test * add schema json * Update jira.md * Update stripe.md * Update source_definitions.yaml * auto-bump connector version --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Add compatibility guide to sidebar (airbytehq#25224) * Shuffled `destination-jdbc` into `bases` folder (airbytehq#25120) * Testing changes for reshuffle * Resolves dependency/race-condition with base classes and docker creation * Removes dockerfile * Updates java connectors gradle dependencies * Run ./gradlew :airbyte-config:init:processResources (airbytehq#25250) * add note on security self-check to security.md (airbytehq#25235) * Source Copper: Add opportunities to streams (airbytehq#24824) * Add opportunities to copper streams * Fix version * update schema * update versions and fix schema formatting * auto-bump connector version --------- Co-authored-by: Mal Hancock <mhancock@archangelic.space> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * low-code: Clean up SessionTokenAuthenticator interface (airbytehq#25086) * Username and session token are optional fields * update * Add titles, descriptions, and examples * Automated Commit - Formatting Changes * fix a small typo --------- Co-authored-by: girarda <girarda@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * Add logging functionality to understand Xmin wraparound (airbytehq#25220) * Add logging functionality - Add logging for DB version (all JDBC sources) - Add logging for Xmin values * Automated Change * Renaming + comments * More comments * Bump docker version + docs * auto-bump connector version * Update failed source_specs.yaml --------- Co-authored-by: akashkulk <akashkulk@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * metadata: create specs secret mask (airbytehq#25201) * Connector builder: Authentication documentation (airbytehq#25112) * authentication documentation * some fixes * fix broken link * remove session token auth * review comments * first version of tutorial * Revert "first version of tutorial" This reverts commit bbdeb99. * review comments * review comments * Connector builder: Partitioning documentation (airbytehq#25209) * partitioning documentation * review comments * Source S3: Add start date filter for files (airbytehq#25010) * Source S3: Add start date filter for files * Source S3: add docs * Source S3: add unittest * Source S3: add unittest * Source S3: add unittest * Source S3: Fix spec test * Source S3: bump version * Source S3: fix tests * Source S3: fix description * auto-bump connector version * Source S3: refactor start_date filtering * Source S3: update setup * Source S3: serialize state for cache * Source S3: refactor skip file filter * Source S3: bump version + update docs * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source Zendesk Chat: update expected records (airbytehq#25271) * Source Sentry: update expected records (airbytehq#25270) * Source S3: handle block size related errors (airbytehq#25067) * Source S3: handle pyarrow block size errors * Source S3: bump version * Automated Change * Source S3: fix null field check * Revert "Automated Change" This reverts commit dc707f7. * Automated Change * Source S3: bump version + update docs * auto-bump connector version --------- Co-authored-by: artem1205 <artem1205@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source Zendesk Support: update expected records (airbytehq#25272) * Add Dagster Deployment Action skeleton (airbytehq#24929) * Add code skeleton * Remove pr deployments * remove connectors change * Add action git ignore * Update YAML to deploy properly * Add newline at eof * Setup python-executable deploy * Add temporary script * Update readme * Remove github workflow * Add dagger pipeline for deploy * Add github action * Update readme * Remove comment * Move script into dagger pipeline * Add run test helper and skip test logic * Have the dagster orchestrator properly generate the setup.py file * Remove deploy script * Address PR comments * Update lock * Automated Commit - Formatting Changes --------- Co-authored-by: bnchrch <bnchrch@users.noreply.github.com> * 🎉 Source Facebook Marketing: Increase pause interval time after reach MIN_RATE or MAX_RATE. (airbytehq#22990) * Change min rate and increase time in pauses * Update Dockerfile * Update source_definitions.yaml * Update facebook-marketing.md * auto-bump connector version --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🎉 New Source: Apple Search Ads [low-code CDK] (airbytehq#19557) * chore(source-apple-search-ads): add low-code boilerplate code * feat(source-apple-search-ads): add spec and 6 streams * test(source-apple-search-ads): set integration and acceptance tests * fix(source-apple-search-ads): camel case for all properties * docs(source-apple-search-ads): add bootstrap and general docs * docs(source-apple-search-ads): update changelog with right PR number and content * feat(source-apple-search-ads): add lookback window and set to 30d * chore(source-apple-search-ads): remove unused file * test(source-apple-search-ads): complete sample state * docs(source-apple-search-ads): rename documentation url * docs(source-apple-search-ads): remove 90 days limitation * feat(source-apple-search-ads): add exponential backoff * fix(source-apple-search-ads): fix acceptance test execution * chore(source-apple-search-ads): switch time interval to ISO 8601 duration * chore(source-apple-search-ads): update json schemas to the latest version * test(source-apple-search-ads): ignore fields that are not idempotent * test(source-apple-search-ads): align threshold days with lookback window for incremental test * test(source-apple-search-ads): adjust timeout * chore(source-apple-search-ads): add latest CDK changes to boilerplate code * refactor(source-apple-search-ads): use latest manifest for CDK * add icon and source seed * auto-bump connector version --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🎉 New Source: Aircall [Low code CDK] (airbytehq#25240) * Initial Commit * fix sample config * change date_from to start_date * optimize retriever method * paginate records * fix formating * auto-bump connector version --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Increase memory_limit for Amplitude (airbytehq#25282) * Increase memory_limit for Amplitude * Automated Change * Adds 400 errors and reauth suggestion to Airtable doc (airbytehq#25255) * Adds information about 400 and 401 errors that can be temporarily fixed by reauthing * Update docs/integrations/sources/airtable.md * 🐛 Source Jira: Add missing types to issues json schema (airbytehq#25275) * [FIX] add types Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [UPD] bump source-jira version Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [UPD] bump oss version Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * auto-bump connector version --------- Signed-off-by: Henri Blancke <blanckehenri@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * CAT tools/acceptance_test_config_migration: Make relatively configurable per migration, add (hacky) ability to run CAT tests locally for multiple connectors (airbytehq#24377) * Pass argument along, add test that should pass and test that should fail * Add tests with additionalProperties * Set additionalproperties=false when not set| * Parametrize test cases * Make the behavior 'optional' * Fix parametrization for all combinations * Improve CI credentials README and rename param * Update naming to be clearer about columns only * record_has_unexpected_field > record_has_unexpected_column * Automated Change * Hacking the CAT dockerfile and run script to test my changes specifically * First crack at running CAT on all connectors * Write during instead of after all tests * Async-ify it * Add ability to define max concurrency * Write successes * ci_credentials: fix overwriting 'data' before getting nextPageToken * Adjustible num_semaphores, check to make sure it's an airbyte connector first * Automated Change * Make create_issues and create_prs more configurable, add issue for fail_on_extra_columns * Add ability to pass in sources as a list or from a txt file * Add logs to issue, make project nullable * Migrate multiple connectors * Add cli args * use ruamel.yaml to preserve ordering * Separate config loading from config migration * Add ability to pass in lists of sources to test. Sort output by exit codes. Fix max_concurrency flag * Default to testing only beta and GA connectors * Always write test output when available * Revert "Add cli args" This reverts commit b538a8c. * Remove slash * Don't run on alpha connetors, handle older config style * Don't migrate to new format, preserve quotes and long lines * Automated Change * Update issue, don't run for alpha connectors * Automated Change * Add bypass for extra fields test * Add bypass for extra fields test * Rename run_tests script * Rename module * Update args usage, small changes * Refactor create_issues.py * Clean up run_tests.py * Sort out arg parsers * Pull out get_valid_definitions_from_args * Import definitions module instead of methods * Use config files to provide constants for each migration * Handle FileNotFoundError in create_issues.py, improve logging * Rename to migrations, reference name of folder via utils * Update readmes for migration modules, add script for getting outputs * Use tmp dir, correct path for issue reference * Fix bash script * Fix create command, pull out test results insertion * Update call to update_configuration * add precommit to requirements * Reorder README * README cleanup for test and create issues * README cleanup for create_prs and config_migration * More readmes! Readmes galore * allow_beta * Restore hacky changes to dockerfile and acceptance-test-docker * Handle 'other' release stages * Update readme * Remove TODO, add comments to shell script * format according to gradle * format * Fix formatting --------- Co-authored-by: marcosmarxm <marcosmarxm@users.noreply.github.com> Co-authored-by: erohmensing <erohmensing@users.noreply.github.com> * Dagster: rename catalog to registry (airbytehq#25254) * rename catalog to registry in metadata service * rename catalog to registry in metadata files * Run generate models * Fix missed renames * Add github personal access token * Run black * Automated Change --------- Co-authored-by: bnchrch <bnchrch@users.noreply.github.com> * Source Amplitude: Reduce request_time_range (airbytehq#25281) * Source Amplitude: Reduce request_time_range * Source Amplitude: Reduce request_time_range * Update docs/integrations/sources/amplitude.md Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com> * Update docs/integrations/sources/amplitude.md * auto-bump connector version --------- Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com> Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * increase postgres heartbeat from 3600->7200 (airbytehq#25304) * connectors-ci: fix java connectors testing (airbytehq#25268) * connectors-ci: include destination-azure-blob-storage gradle project as a destination build dep * fix integration test return value * Source Stripe: add account stream (airbytehq#25136) * add accounts stream * add docs * lint * fix * fix * fix test * fix tests * fix account schema * fix expected records and schema * remove account from abnormal state * update dockerfile and update docs * Update Dockerfile * Update source_definitions.yaml * Update test_source stream count * auto-bump connector version --------- Co-authored-by: Haithem Souala <haithem.souala@woopit.fr> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * remove session token auth (airbytehq#25142) * Connector builder: Tutorial (airbytehq#25153) * authentication documentation * some fixes * fix broken link * remove session token auth * review comments * first version of tutorial * Revert "first version of tutorial" This reverts commit bbdeb99. * first version of tutorial * review comments * improvements * improvements * review comments * Source Hubspot: update expected records (airbytehq#25309) * Source Hubspot: update expected records * Update expected records * Bump Airbyte version from 0.44.0 to 0.44.1 * 🎉 Source Google Ads - fixed schema type for "data_type: ENUM" and "is_repeated: true" to array of strings (airbytehq#24945) * fixed schema type for "data_type: ENUM" and "is_repeated: true" to array of strings * added SAT for GAQL * updated version * Added unittest for get_json_schema * slight enhancement * fixed empty lines * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * source-amplitude: Only fetch date_time_fields once per request (airbytehq#25315) * postgres-cdc checkpointing: fix LSN parsing bug + refactor for efficiency (airbytehq#24582) * Fix LSN parsing from Integer to Long * rebasing * Rebase * Rebase * Other casting * Lock the file only when reading, so the file is free when parsing the object. Increased from 1 to 166 checkpoints, and from skipping hundreds of checkpoints to never skip a state. * Update load function documentation * bump mysql and mssql * cdc: refactor to remove debezium dependency from connector packages * use gradle's shared dependency * more refactoring * upgrade docker version * resolve master merge conflicts * Automated Change * minor changes * resolve merge conflicts * avoid deserializing multiple times * simplify * enable checkpointing for Postgres * more improvements * enable assertions * changelog + bump version * auto-bump connector version * auto-bump connector version * manual bump --------- Co-authored-by: subodh <subodh1810@gmail.com> Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * use actual release tag (airbytehq#25311) * Source Twilio: update expected records (airbytehq#25320) * Builder documentation: Pagination documentation (airbytehq#25261) * add offset increment and page increment docs * add cursor pagination docs * add full example of requests submitted by connector * improve examples and explanation of page/cursor increment * improve descriptions * add section about accessing last record for cursor pagination * add recommendation about page size * add links to low-code and python CDKs * 🐛 Destination AWS Datalake: fix airbyte type in complex data types (airbytehq#24642) * [UPD] prefer airbyte type in complex data types Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [UPD] bump version Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [UPD] only for integers Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [FIX] changelog table Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [UPD] cast objects that can have additional props to string Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [ADD] tests Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * auto-bump connector version --------- Signed-off-by: Henri Blancke <blanckehenri@gmail.com> Co-authored-by: Sajarin <sajarindider@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Dagger: Return true on valid exit in Dagger Commands (airbytehq#25333) * Return true on valid exit * Update return type to bool * update to new version of dagger cli (airbytehq#25340) * 🎉Source Mailjet-Mail: Add `listrecipient` stream (airbytehq#24689) * add listrecipient stream * update schemas to match source api * bump dockerfile version * update changelog * auto-bump connector version --------- Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 24153 Destination Databricks: Fix support for external tables on S3 (airbytehq#24657) * 24153 Destination Databricks: Update s3 source external * 24153 Destination Databricks: Fixed databricks.md formatting * 24153 Destination Databricks: Update the version * auto-bump connector version --------- Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Add version info to publish comment (airbytehq#25343) * Update managing-airbyte-cloud.md (airbytehq#25341) * Update managing-airbyte-cloud.md * Update managing-airbyte-cloud.md * Update managing-airbyte-cloud.md * Destination BigQuery + BigQuery denormalized: add configurable file buffer count (airbytehq#25287) * add configurable file buffer count for dest bq + bq denorm * update changelog * changelog nits * auto-bump connector version * manually bump connector version for bq denorm --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * [JDBC sources] : Log database indexes per stream (airbytehq#25345) * [JDBC sources] : Log database indexes per stream * Docker & doc bump * auto-bump connector version * auto-bump connector version * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * update link (airbytehq#25349) * connectors-ci: build multi platform connector images (airbytehq#25286) * 🎉 New Destination: Cumul.io (airbytehq#23153) * Included Cumul.io Destination connector * add gradle file * add dockerfile and format files * Refactor DestinationCumulio class based on review This commit refactors the DestinationCumulio class by removing some unused dummy variables and commented code that used to push data in the check method. Next to that I fixed a potential bug as pointed out by @grishick who reviewed this code (thanks for that!). Previously, the state message could have been yielded before any buffers were flushed, which could result in incorrectly indicating a synchronization has been completed while data could still be inside the buffer. Now, all buffers will be flushed before yielding that message. * Chore: add Cumul.io logo svg to airbyte-config/init/src/main/resources/icons/ * Update two unit tests that were failing due to fixing bug in a previous commit * fix tests --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> * Source SFTP-Bulk: Support custom CSV separators (airbytehq#19224) * manage different separators * tests * update tests * add optional parameter for separator * add default separator * typo * add trailing newline * fix spec.json order * bump dockerfile version * update changelog * auto-bump connector version * retrigger checks * retrigger checks --------- Co-authored-by: Sunny Hashmi <6833405+sh4sh@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source Amplitude: refactor Events stream (airbytehq#25317) * Source Amplitude: Refactor Events Stream, based on Python CDK * Source Amplitude: bump docker version * Source Amplitude: Fix add get_updated_state * Revert "Increase memory_limit for Amplitude (airbytehq#25282)" This reverts commit 06354ae. * Source Amplitude: Fix backport compatibility with new config * Update source_definitions.yaml * Source Amplitude: update catalogs --------- Co-authored-by: artem1205 <artem1205@users.noreply.github.com> * low-code: Do not apply transforms on AirbyteLogMessages and AirbyteTraceMessages (airbytehq#25290) * Check the input type before applying transformations * format * remove debug prints * connectors-ci: use shared pip cache instead of locked (airbytehq#25370) * fix typo in GHA workflow (airbytehq#25372) * Dagster registry generation and persist (airbytehq#25260) * Define legacy and latest registry at same time * Fix file persist * Get persist to work * Expand generate models to all models * Add new registry types * Fix class name * Get valid registry type definition * Remove uuid hack * Fix ids for json schemas * Resolve issues * Update legacy assets * Add typed legacy registry * Fix icon issue * Regenerate models * Update spec mask to use registry type * Move v1 to v0 * Add json sanitized dict helper * Fix tests and format * Ensure we only get latest * Code review comments * fix missing spec error * Move registry code to helper * Destination Databricks: use hive_metastore if catalog not set (airbytehq#25366) * use hive_metastore if catalog not set * Automated Change * add test case * version bump + changelog * auto-bump connector version --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Source Salesforce: update fields in Account expected_records (airbytehq#25385) * Source Faker: update expected records (airbytehq#25387) * Source Freshsales: update expected records (airbytehq#25388) * Source Amplitude: update changelog (airbytehq#25392) * Update Issues Templates (airbytehq#25365) * update issue templates * add titles and fix multiline text * Automated Change * Update issue-cli.yaml * Update issue-connector.yaml * Update issue-platform.yaml * Update issue-connector.yaml * Docs: Add AWS_SECRET_MANAGER to list of available options for storing secrets in documentation (airbytehq#25374) * Source Snapchat Marketing: Add missing story metrics to basic stats (airbytehq#20897) * Add missing story metrics to basic stats * fix expected records and update dockerfile version * auto-bump connector version --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Fix Java destination template (airbytehq#25390) The currently generated Java destination code doesn't compile. Fix it up. * Connector builder: Incremental sync documentation (airbytehq#25238) * authentication documentation * some fixes * fix broken link * remove session token auth * review comments * first version of tutorial * Revert "first version of tutorial" This reverts commit bbdeb99. * review comments * partitioning documentation * incremental sync * improvements * review comments * Conenctor builder: Record processing page (airbytehq#25267) * authentication documentation * some fixes * fix broken link * remove session token auth * review comments * first version of tutorial * Revert "first version of tutorial" This reverts commit bbdeb99. * first version of tutorial * review comments * improvements * partitioning documentation * review comments * improvements * record processing doc * review comments * 🎉 New Destination: Starburst Galaxy (airbytehq#25399) Co-authored-by: Mayank Vadariya <48036907+mayankvadariya@users.noreply.github.com> * Add git checkout (airbytehq#25376) * Destination S3 Glue: Fix precision and scale of decimal type (airbytehq#25397) * destination-s3-glue: Fix precision and scale of decimal type The decimal type in Hive will default to a precision of 10 and a scale of 0, meaning that it cannot handle numbers larger than 10 digits. The maximum values for precision and scale are 38 and 38 respectively. This updates the decimal type specified in destination schemas to use the maximum values for the broadest compatibility with numeric values. If a table is using the default precision of 10 and a number is present which exceeds that many digits then the query engine will throw a `fieldValue cannot be null` error message which is very cryptic. * run seed generator * Automated Change * Update connectors.md * Automated Change --------- Co-authored-by: Tobias Macey <tmacey@mit.edu> * small docs fixes (airbytehq#25377) * Add descriptions and examples to component schema (airbytehq#25117) * Reference docs for backoff strategies * Docs for most authentication mechansims * Docs for CheckStream * MinMaxDatetime * DeclarativeStream * DefaultErrorHandler * CompositeErrorHandler * update * Update token expiry date description * DPath extractor * Add interpolation_context * HttpResponseFilter * RecordFilter * RecordSelector * DefaultPaginator * CursorPagination * OffsetIncrement * Page Increment * PrimaryKey * HttpRequester * request option and request path * Schemas * Spec * Add field * remove fields * no auth and no pagination * Delete deprecated comment * Missing description * Json Decoder * OAuthConfigSpecification * reorder * add titles, examples, and descriptions for partition routers and datetime based cursor * updates * Update * fix indentation * Automated Commit - Formatting Changes * Update as per feedback * html tag * generate models * Update name * do not use title as class name * Update * Add stream_interval and stream_partition to interpolation_context * Automated Commit - Formatting Changes * fix path * format --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> Co-authored-by: girarda <girarda@users.noreply.github.com> * Fix and document macros and interpolation variables (airbytehq#25305) * Fix and document macros * cleanup * dots * Add tests and refactor * Update * Add an example * Document variables * Mention now_local is not recommended * 🤖 Bump patch version of Airbyte CDK * Source Twilio: update messages stream expected records (airbytehq#25384) * 🎉 Source Stripe: add SetupIntents stream (airbytehq#23963) * add SetupIntents schema * add setup intents * add test & docs * update docs * add additionalProperties * update date * fix schema * update schema and expected records * fix sub schedule schema * update expected records for sub schedule * update expected records * update source def seed * auto-bump connector version --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Mal Hancock <mhancock@archangelic.space> Co-authored-by: Mal Hancock <mallory@archangelic.space> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * connectors-ci: document the CLI (airbytehq#25391) * connectors-ci: package normalization inside destinations images supporting it (airbytehq#25359) * Update oss_catalog.json * fix conflicts * fix conflict oss * fix conflict oss * fix conflict oss * Merge branch 'yotpo' of https://github.com/btkcodedev/airbyte into yotpo * fix reviews expected records, add EOF * fix email_analytics schema * fix expected records * Update sample_config.json * Update source_definitions.yaml * auto-bump connector version --------- Signed-off-by: Henri Blancke <blanckehenri@gmail.com> Co-authored-by: Visnu Pitiyanuvath <visnup@gmail.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Alexandre Girard <alexandre@airbyte.io> Co-authored-by: girarda <girarda@users.noreply.github.com> Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com> Co-authored-by: artem1205 <artem1205@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com> Co-authored-by: arsenlosenko <arsenlosenko@users.noreply.github.com> Co-authored-by: Ali Hassan <alihassan01264@gmail.com> Co-authored-by: Ali Hassan <62539376+Alihassanc5@users.noreply.github.com> Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Boluwatife Victor <95125924+BirdboyBolu@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> Co-authored-by: Philip Corr <PhilipCorr@users.noreply.github.com> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: evantahler <evantahler@users.noreply.github.com> Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: Cole Snodgrass <cole@airbyte.io> Co-authored-by: colesnodgrass <colesnodgrass@users.noreply.github.com> Co-authored-by: timroes <timroes@users.noreply.github.com> Co-authored-by: Sajarin <sajarindider@gmail.com> Co-authored-by: Rachel RIZK <rachel.rizk@backmarket.com> Co-authored-by: sajarin <sajarin@users.noreply.github.com> Co-authored-by: Denys Davydov <davydov.den18@gmail.com> Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: Topher Lubaway <asimplechris@gmail.com> Co-authored-by: Jimmy Ma <gosusnp@users.noreply.github.com> Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com> Co-authored-by: Ivica Taseski <ivica.taseski94@gmail.com> Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com> Co-authored-by: Ryan Fu <ryan.fu@airbyte.io> Co-authored-by: richardlam-quotapath <113922446+richardlam-quotapath@users.noreply.github.com> Co-authored-by: Mal Hancock <mhancock@archangelic.space> Co-authored-by: akashkulk <akashkulk@users.noreply.github.com> Co-authored-by: Joe Reuter <joe@airbyte.io> Co-authored-by: bnchrch <bnchrch@users.noreply.github.com> Co-authored-by: Vladimir remar <vladimir.remar@gmail.com> Co-authored-by: Rachel Rizk <69161287+rach-r@users.noreply.github.com> Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com> Co-authored-by: Juan <80164312+jnr0790@users.noreply.github.com> Co-authored-by: Henri Blancke <blanckehenri@gmail.com> Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> Co-authored-by: erohmensing <erohmensing@users.noreply.github.com> Co-authored-by: Haithem Souala <haithem.souala@woopit.fr> Co-authored-by: josephkmh <josephkmh@users.noreply.github.com> Co-authored-by: midavadim <midavadim@yahoo.com> Co-authored-by: Sergio Ropero <42538006+sergio-ropero@users.noreply.github.com> Co-authored-by: subodh <subodh1810@gmail.com> Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com> Co-authored-by: Conor <cpdeethree@users.noreply.github.com> Co-authored-by: Dominik Wagner <dominik.r.mall2@gmail.com> Co-authored-by: Yevhen Sukhomud <suhomud@gmail.com> Co-authored-by: Pedro S. Lopez <pedroslopez@me.com> Co-authored-by: Riley Brook <riley@airbyte.io> Co-authored-by: Cynthia Yin <cynthia@airbyte.io> Co-authored-by: Joost Stessens <77833265+jooststessens@users.noreply.github.com> Co-authored-by: Ana Loureiro <37600275+anamargaridarl@users.noreply.github.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: Chris Wu <chris@faros.ai> Co-authored-by: iberchid <93732608+iberchid@users.noreply.github.com> Co-authored-by: Pekka Enberg <penberg@iki.fi> Co-authored-by: Mayank Vadariya <48036907+mayankvadariya@users.noreply.github.com> Co-authored-by: Tobias Macey <tmacey@mit.edu> Co-authored-by: brianjlai <brian.lai@airbyte.io> Co-authored-by: JJ Nilbodee <pornwipa.nil@gmail.com> Co-authored-by: Mal Hancock <mallory@archangelic.space>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Developing new connector for the source: Yotpo
Resolves:
Yotpo
How
Developed using (Configuration Based Source) low-code CDK
Recommended reading order
spec.yamlmanifest.yamlschemas/*Tests
Integration & Acceptance
Full Test Results🚨 User Impact 🚨
New Connector: Source Yotpo
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.mddocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampledocs/integrations/README.mdairbyte-integrations/builds.md