Skip to content

Commit

Permalink
[source-monday] [source-square] [source-greenhouse] [source-tempo] [s…
Browse files Browse the repository at this point in the history
…ource-zenloop] fix a few connectors with failing custom component unit tests + release some connectors w/ beta (#23231)

* fix a few connectors with failing custom component unit tests

* fix outdated expected records

* make square orders stream incremental

* Updated square SAT and delete custom extractor for monday.

* Added allowedHosts

* Delete DpathStringExtractor from init

* Updated abnormal records

* bump versions of greenhouse, tempo, woocommerce, and zenloop to use beta low code version

* remove woocommerce because it's already published with beta

* add allowedHosts for tempo and zenloop

* fix abnormal state file to use created_at for some streams

* auto-bump connector version

* bump versions for square and monday

* auto-bump connector version

* auto-bump connector version

* manually update square definition generation due to failure

* auto-bump connector version

---------

Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
  • Loading branch information
3 people authored Mar 8, 2023
1 parent cfeabd3 commit 52f83d4
Show file tree
Hide file tree
Showing 25 changed files with 182 additions and 1,577 deletions.
4 changes: 3 additions & 1 deletion airbyte-cdk/python/bin/low-code-unit-tests.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
#!/usr/bin/env bash

set -e
# Ideally we'd like to have set -e, but when set -e is used, the `python -m pytest unit_tests` command can return a
# non-zero exit code and end the whole script prematurely
# set -e

[ -z "$ROOT_DIR" ] && exit 1

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -803,7 +803,7 @@
- name: Greenhouse
sourceDefinitionId: 59f1e50a-331f-4f09-b3e8-2e8d4d355f44
dockerRepository: airbyte/source-greenhouse
dockerImageTag: 0.3.0
dockerImageTag: 0.3.1
documentationUrl: https://docs.airbyte.com/integrations/sources/greenhouse
icon: greenhouse.svg
sourceType: api
Expand Down Expand Up @@ -1171,11 +1171,14 @@
- name: Monday
sourceDefinitionId: 80a54ea2-9959-4040-aac1-eee42423ec9b
dockerRepository: airbyte/source-monday
dockerImageTag: 0.2.2
dockerImageTag: 0.2.3
documentationUrl: https://docs.airbyte.com/integrations/sources/monday
icon: monday.svg
sourceType: api
releaseStage: beta
allowedHosts:
hosts:
- "api.monday.com"
- name: MongoDb
sourceDefinitionId: b2e713cd-cc36-4c0a-b5bd-b47cb8a0561e
dockerRepository: airbyte/source-mongodb-v2
Expand Down Expand Up @@ -1851,11 +1854,15 @@
- name: Square
sourceDefinitionId: 77225a51-cd15-4a13-af02-65816bd0ecf4
dockerRepository: airbyte/source-square
dockerImageTag: 0.2.0
dockerImageTag: 0.2.1
documentationUrl: https://docs.airbyte.com/integrations/sources/square
icon: square.svg
sourceType: api
releaseStage: beta
allowedHosts:
hosts:
- "connect.squareupsandbox.com"
- "connect.squareup.com"
- sourceDefinitionId: 7a4327c4-315a-11ec-8d3d-0242ac130003
name: Strava
dockerRepository: airbyte/source-strava
Expand Down Expand Up @@ -1921,11 +1928,14 @@
- name: Tempo
sourceDefinitionId: d1aa448b-7c54-498e-ad95-263cbebcd2db
dockerRepository: airbyte/source-tempo
dockerImageTag: 0.3.0
dockerImageTag: 0.3.1
documentationUrl: https://docs.airbyte.com/integrations/sources/tempo
icon: tempo.svg
sourceType: api
releaseStage: beta
allowedHosts:
hosts:
- api.tempo.io
- name: TiDB
sourceDefinitionId: 0dad1a35-ccf8-4d03-b73e-6788c00b13ae
dockerRepository: airbyte/source-tidb
Expand Down Expand Up @@ -2248,11 +2258,14 @@
- name: Zenloop
sourceDefinitionId: f1e4c7f6-db5c-4035-981f-d35ab4998794
dockerRepository: airbyte/source-zenloop
dockerImageTag: 0.1.5
dockerImageTag: 0.1.6
documentationUrl: https://docs.airbyte.com/integrations/sources/zenloop
icon: zenloop.svg
sourceType: api
releaseStage: beta
allowedHosts:
hosts:
- api.zenloop.com
- sourceDefinitionId: cdaf146a-9b75-49fd-9dd2-9d64a0bb4781
name: Sentry
dockerRepository: airbyte/source-sentry
Expand Down
10 changes: 5 additions & 5 deletions airbyte-config/init/src/main/resources/seed/source_specs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6297,7 +6297,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-greenhouse:0.3.0"
- dockerImage: "airbyte/source-greenhouse:0.3.1"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/greenhouse"
connectionSpecification:
Expand Down Expand Up @@ -8937,7 +8937,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-monday:0.2.2"
- dockerImage: "airbyte/source-monday:0.2.3"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/monday"
connectionSpecification:
Expand Down Expand Up @@ -14374,7 +14374,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-square:0.2.0"
- dockerImage: "airbyte/source-square:0.2.1"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/square"
connectionSpecification:
Expand Down Expand Up @@ -14955,7 +14955,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-tempo:0.3.0"
- dockerImage: "airbyte/source-tempo:0.3.1"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/"
connectionSpecification:
Expand Down Expand Up @@ -16919,7 +16919,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-zenloop:0.1.5"
- dockerImage: "airbyte/source-zenloop:0.1.6"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/zenloop"
connectionSpecification:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@ COPY main.py ./
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.version=0.3.1
LABEL io.airbyte.name=airbyte/source-greenhouse
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-monday/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY source_monday ./source_monday
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.2.2
LABEL io.airbyte.version=0.2.3
LABEL io.airbyte.name=airbyte/source-monday

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@
# Copyright (c) 2021 Airbyte, Inc., all rights reserved.
#

from .dpath_string_extractor import DpathStringExtractor
from .graphql_requester import MondayGraphqlRequester
from .source import SourceMonday

__all__ = ["SourceMonday", "MondayGraphqlRequester", "DpathStringExtractor"]
__all__ = ["SourceMonday", "MondayGraphqlRequester"]

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,7 @@ definitions:
record_selector:
type: RecordSelector
extractor:
type: CustomRecordExtractor
field_path: "/data/boards/*/items/*"
class_name: "source_monday.DpathStringExtractor"
field_path: [ "data", "boards", "*", "items", "*" ]
paginator:
$ref: "#/definitions/default_paginator"
pagination_strategy:
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-square/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY source_square ./source_square
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.2.0
LABEL io.airbyte.version=0.2.1
LABEL io.airbyte.name=airbyte/source-square
8 changes: 4 additions & 4 deletions airbyte-integrations/connectors/source-square/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ We split dependencies between two groups, dependencies that are:
### Publishing a new version of the connector
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing unit and integration tests.
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
1. Create a Pull Request.
1. Pat yourself on the back for being an awesome contributor.
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
2. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
3. Create a Pull Request.
4. Pat yourself on the back for being an awesome contributor.
5. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
Original file line number Diff line number Diff line change
@@ -1,32 +1,42 @@
connector_image: airbyte/source-square:dev
tests:
test_strictness_level: high
acceptance_tests:
spec:
tests:
- spec_path: "source_square/spec.yaml"
connection:
tests:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "secrets/config_oauth.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"
discovery:
tests:
- config_path: "secrets/config.json"
- config_path: "secrets/config_oauth.json"
basic_read:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
empty_streams: ["payments"]
expect_records:
path: "integration_tests/expected_records.jsonl"
tests:
- config_path: "secrets/config_oauth.json"
configured_catalog_path: "integration_tests/configured_catalog_oauth.json"
empty_streams: ["shifts", "orders", "payments", "refunds"]
empty_streams:
- name: shifts
bypass_reason: "Not able to fill stream"
- name: orders
bypass_reason: "Not able to fill stream"
- name: payments
bypass_reason: "Not able to fill stream"
- name: refunds
bypass_reason: "Not able to fill stream"
expect_records:
path: "integration_tests/expected_records_oauth.jsonl"
incremental:
tests:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
future_state_path: "integration_tests/abnormal_state.json"
future_state:
future_state_path: "integration_tests/abnormal_state.json"
full_refresh:
tests:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
Original file line number Diff line number Diff line change
@@ -1,23 +1,58 @@
{
"items": {
"updated_at": "2200-01-01T00:00:00.0Z"
[
{
"type": "STREAM",
"stream": {
"stream_state": { "updated_at": "2200-01-01T00:00:00.0Z" },
"stream_descriptor": { "name": "items" }
}
},
"categories": {
"updated_at": "2200-01-01T00:00:00.0Z"
{
"type": "STREAM",
"stream": {
"stream_state": { "updated_at": "2200-01-01T00:00:00.0Z" },
"stream_descriptor": { "name": "categories" }
}
},
"discounts": {
"updated_at": "2200-01-01T00:00:00.0Z"
{
"type": "STREAM",
"stream": {
"stream_state": { "updated_at": "2200-01-01T00:00:00.0Z" },
"stream_descriptor": { "name": "discounts" }
}
},
"taxes": {
"updated_at": "2200-01-01T00:00:00.0Z"
{
"type": "STREAM",
"stream": {
"stream_state": { "updated_at": "2200-01-01T00:00:00.0Z" },
"stream_descriptor": { "name": "taxes" }
}
},
"modifier_list": {
"updated_at": "2200-01-01T00:00:00.0Z"
{
"type": "STREAM",
"stream": {
"stream_state": { "updated_at": "2200-01-01T00:00:00.0Z" },
"stream_descriptor": { "name": "modifier_list" }
}
},
"refunds": {
"created_at": "2200-01-01T00:00:00.0Z"
{
"type": "STREAM",
"stream": {
"stream_state": { "created_at": "2200-01-01T00:00:00.0Z" },
"stream_descriptor": { "name": "refunds" }
}
},
"payments": {
"created_at": "2200-01-01T00:00:00.0Z"
{
"type": "STREAM",
"stream": {
"stream_state": { "created_at": "2200-01-01T00:00:00.0Z" },
"stream_descriptor": { "name": "payments" }
}
},
{
"type": "STREAM",
"stream": {
"stream_state": { "updated_at": "2200-01-01T00:00:00.0Z" },
"stream_descriptor": { "name": "orders" }
}
}
}
]
Original file line number Diff line number Diff line change
Expand Up @@ -148,12 +148,12 @@
"stream": {
"name": "orders",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"supported_sync_modes": ["incremental", "full_refresh"],
"source_defined_cursor": true,
"default_cursor_field": ["id"]
"default_cursor_field": ["updated_at"]
},
"sync_mode": "full_refresh",
"cursor_field": ["id"],
"sync_mode": "incremental",
"cursor_field": ["updated_at"],
"destination_sync_mode": "overwrite"
}
]
Expand Down
Loading

0 comments on commit 52f83d4

Please sign in to comment.