Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎉 New Source: Yotpo [Low code CDK] #25532

Merged
merged 201 commits into from
May 1, 2023
Merged
Show file tree
Hide file tree
Changes from 191 commits
Commits
Show all changes
201 commits
Select commit Hold shift + click to select a range
506daaf
Initial commit
btkcodedev Apr 13, 2023
a4aa251
Add stream top_reviews
btkcodedev Apr 13, 2023
b6f0a3d
Add new stream: privacy_data
btkcodedev Apr 13, 2023
0ca3b33
add new stream: privacy user, refactor
btkcodedev Apr 13, 2023
cb6a1ef
Add new streams, update docs, update definitions
btkcodedev Apr 13, 2023
562f0e7
Merge branch 'master' into yotpo
btkcodedev Apr 13, 2023
a1c5cd0
EOF
btkcodedev Apr 13, 2023
03170a3
Merge branch 'yotpo' of https://github.com/btkcodedev/airbyte into yotpo
btkcodedev Apr 13, 2023
db5c947
Update spec.yaml
btkcodedev Apr 15, 2023
5e13b85
Merge branch 'master' into yotpo
btkcodedev Apr 15, 2023
7e32be8
fix date_from, optimize manifest, update docs
btkcodedev Apr 17, 2023
2a93869
Merge branch 'yotpo' of https://github.com/btkcodedev/airbyte into yotpo
btkcodedev Apr 17, 2023
5f00233
delete privacy streams, refactor webhooks
btkcodedev Apr 17, 2023
8ee1957
add pagination
btkcodedev Apr 17, 2023
38e2fe1
add streams raw_data, email_analytics
btkcodedev Apr 18, 2023
976b1c5
add streams unsubscribers, webhook_events
btkcodedev Apr 18, 2023
7e2a453
Fix pagination
btkcodedev Apr 22, 2023
9a2691a
revert doc
btkcodedev Apr 22, 2023
11769ae
Merge branch 'yotpo' of https://github.com/btkcodedev/airbyte into yotpo
btkcodedev Apr 22, 2023
c1005f1
Revert "remove top_reviews and reviews_bottomline, Make common pagina…
btkcodedev Apr 22, 2023
29ee857
remove streams top_reviews, reviews_bottomline
btkcodedev Apr 22, 2023
8dfcadb
make common base and extractor
btkcodedev Apr 23, 2023
9295a46
refactor, Checkpoint for full test passed
btkcodedev Apr 23, 2023
4e738c7
refactor base, requester, paginator
btkcodedev Apr 23, 2023
2464cf9
remove extra paginator property
btkcodedev Apr 23, 2023
deffecd
Merge branch 'master' into yotpo
btkcodedev Apr 23, 2023
dab3d79
Update oss_catalog.json
btkcodedev Apr 23, 2023
823ac77
fix conflicts
btkcodedev Apr 26, 2023
67aa645
fix conflict oss
btkcodedev Apr 26, 2023
0579925
fix conflict oss
btkcodedev Apr 26, 2023
f2bdcaa
Add stream top_reviews
btkcodedev Apr 13, 2023
5a5b80a
Add new stream: privacy_data
btkcodedev Apr 13, 2023
eab3d5a
add new stream: privacy user, refactor
btkcodedev Apr 13, 2023
d40445c
Add new streams, update docs, update definitions
btkcodedev Apr 13, 2023
1cab657
EOF
btkcodedev Apr 13, 2023
dbf309a
Source Stripe: expand tiers when syncing plans (#19906)
visnup Apr 12, 2023
1ffe136
airbyte-ci: improve metadata validation perfs (#25107)
alafanechere Apr 12, 2023
941b5d9
metadata service: upload metadata files to gcs (#25115)
alafanechere Apr 12, 2023
f8cdd57
metadata: cleanup workflows (#25105)
alafanechere Apr 12, 2023
b7a1b72
Source Stripe: generate seed source file (#25113)
marcosmarxm Apr 12, 2023
c5ff52a
Connector builder: set pages and slices limits (#25121)
girarda Apr 12, 2023
b9a3c7c
🤖 Bump patch version of Airbyte CDK
girarda Apr 12, 2023
5c42200
Bump Airbyte version from 0.43.1 to 0.43.2
girarda Apr 13, 2023
5d5efd0
source-postgres works with bigger datasets (#25138)
evantahler Apr 13, 2023
03572a3
create metadata files for all connectors (#24964)
alafanechere Apr 13, 2023
2f0e8b1
fix date_from, optimize manifest, update docs
btkcodedev Apr 17, 2023
a5f9e55
Update spec.yaml
btkcodedev Apr 15, 2023
fc8ddf8
🚨🚨 Source Amplitude: migrate to YAML (#22362)
artem1205 Apr 13, 2023
d209b60
Source Zendesk Support: update expected records (#25058)
arsenlosenko Apr 13, 2023
25850fa
Source Surveymonkey: Fix spec.json required fields and update schema…
marcosmarxm Apr 13, 2023
d261ab7
Add additional destination-bigquery datacenters (#25122)
evantahler Apr 13, 2023
c4e3c2f
Bump versions in atelier dockerfile (#25125)
girarda Apr 13, 2023
d21d131
Source: Amazon Ads - Add pk validator for reports when expected pk is…
lazebnyi Apr 13, 2023
c222af6
🎉Source Twilio: New Stream Conversation Participants (#23995)
btkcodedev Apr 13, 2023
940d210
Source Iterable: update expected records (#25144)
arsenlosenko Apr 13, 2023
b10e585
Source Facebook Pages: update insight metrics request params (#25143)
arsenlosenko Apr 13, 2023
8d39855
fix strict-encrypt metadata dockerRepository (#25159)
alafanechere Apr 13, 2023
a4bfb2c
Source Slack: update expected records (#25150)
arsenlosenko Apr 13, 2023
7ebf1e6
dagger: upgrade to 0.5.0 (#25171)
alafanechere Apr 13, 2023
c379849
Fixed missing punctuations (#25151)
BirdboyBolu Apr 13, 2023
b089ce3
refactor source harness and GitHub action to add source-mysql (#25036)
rodireich Apr 13, 2023
2e77287
change google analytics dockerfile order to pip install source (#23082)
PhilipCorr Apr 13, 2023
3a741c1
Always show client cert and key (#25131)
lmossman Apr 13, 2023
c1dccdc
:bug: Source File: fix column data types for numerical values (#23759)
Alihassanc5 Apr 13, 2023
acb99aa
`connectors.md` gains Github labels for connector issues (#25182)
evantahler Apr 13, 2023
d1b72d5
Housekeeping: Remove unused workflows. (#25191)
davinchia Apr 13, 2023
77e60be
increase sentry heartbeat - 43200->64800 (#25185)
colesnodgrass Apr 13, 2023
abf2c19
Bump Airbyte version from 0.43.2 to 0.44.0
timroes Apr 14, 2023
961fe99
dagger CI: auth to dockerhub (#25196)
alafanechere Apr 14, 2023
3b911bc
airbyte-ci: orchestrate the upload of modified metadata.yaml files to…
alafanechere Apr 14, 2023
4b4598f
Source Hubspot: update expected records (#25152)
arsenlosenko Apr 14, 2023
0c53367
Source Github: update expected records (#25148)
arsenlosenko Apr 14, 2023
4b2376b
Source Pipedrive: update expected records (#25199)
arsenlosenko Apr 14, 2023
0eea77f
Source Zendesk Support: update expected records (#25206)
arsenlosenko Apr 14, 2023
3360cd4
Source Twilio: update expected records (#25200)
arsenlosenko Apr 14, 2023
1fdf3b8
connector builder: Emit message at start of slice (#25180)
girarda Apr 14, 2023
774e2eb
🤖 Bump patch version of Airbyte CDK
girarda Apr 14, 2023
fcaaf45
CI checks for #23868 (#24717)
sajarin Apr 14, 2023
3d0fb85
Source Facebook Marketing: fix data retention period validation (#25204)
davydov-d Apr 14, 2023
b59c047
Add OSS getting started redirect (#25213)
timroes Apr 14, 2023
6a8b35b
Add is_optional flag to connector tests (#25211)
bnchrch Apr 14, 2023
ca0f960
Run doc deploys on docosaurus changes (#25215)
timroes Apr 14, 2023
650ca35
Fix manifest_declarative_source + add unit tests (#25217)
girarda Apr 14, 2023
b3a25a7
🤖 Bump patch version of Airbyte CDK
girarda Apr 14, 2023
e2ad907
Document the config for docker workspace cleanup (#25210)
gosusnp Apr 14, 2023
eab748f
delete privacy streams, refactor webhooks
btkcodedev Apr 17, 2023
8c98707
add pagination
btkcodedev Apr 17, 2023
e9026bf
add streams raw_data, email_analytics
btkcodedev Apr 18, 2023
1fd54c2
add streams unsubscribers, webhook_events
btkcodedev Apr 18, 2023
0c663b5
Fix pagination
btkcodedev Apr 22, 2023
0797e1c
revert doc
btkcodedev Apr 22, 2023
c7f8d96
Merge branch 'yotpo' of https://github.com/btkcodedev/airbyte into yotpo
btkcodedev Apr 22, 2023
b6db4d6
Revert "remove top_reviews and reviews_bottomline, Make common pagina…
btkcodedev Apr 22, 2023
a737e7d
remove streams top_reviews, reviews_bottomline
btkcodedev Apr 22, 2023
479504e
make common base and extractor
btkcodedev Apr 23, 2023
b8a0e49
refactor, Checkpoint for full test passed
btkcodedev Apr 23, 2023
65ad3b1
refactor base, requester, paginator
btkcodedev Apr 23, 2023
57a681c
remove extra paginator property
btkcodedev Apr 23, 2023
9f05a27
Cleanup some logging in connectors (#25227)
akashkulk Apr 16, 2023
7fc4bd8
🎉 New Source: Teradata (#24221)
itaseskii Apr 17, 2023
e20d844
Add metadata gcs sensor to Dagster (#25055)
bnchrch Apr 17, 2023
85da76c
🎉Source Stripe: New Stream - Subscription Schedule (#23624)
btkcodedev Apr 17, 2023
0d9a79e
Add compatibility guide to sidebar (#25224)
girarda Apr 17, 2023
69fa458
Shuffled `destination-jdbc` into `bases` folder (#25120)
ryankfu Apr 17, 2023
8757e4f
Run ./gradlew :airbyte-config:init:processResources (#25250)
bnchrch Apr 17, 2023
9f533f8
add note on security self-check to security.md (#25235)
sh4sh Apr 17, 2023
004def7
Source Copper: Add opportunities to streams (#24824)
richardlam-quotapath Apr 17, 2023
76e90d7
low-code: Clean up SessionTokenAuthenticator interface (#25086)
girarda Apr 17, 2023
045c3c9
🤖 Bump minor version of Airbyte CDK
girarda Apr 17, 2023
141a6fd
Add logging functionality to understand Xmin wraparound (#25220)
akashkulk Apr 17, 2023
a372f70
metadata: create specs secret mask (#25201)
alafanechere Apr 18, 2023
0a83e62
Connector builder: Authentication documentation (#25112)
Apr 18, 2023
00e87db
Source S3: Add start date filter for files (#25010)
artem1205 Apr 18, 2023
bca85c7
Source Zendesk Chat: update expected records (#25271)
arsenlosenko Apr 18, 2023
8607d9d
Source Sentry: update expected records (#25270)
arsenlosenko Apr 18, 2023
09d8005
Source S3: handle block size related errors (#25067)
artem1205 Apr 18, 2023
615caa6
Source Zendesk Support: update expected records (#25272)
arsenlosenko Apr 18, 2023
e1101a6
Add Dagster Deployment Action skeleton (#24929)
bnchrch Apr 18, 2023
6899207
:tada: Source Facebook Marketing: Increase pause interval time after …
vladimir-remar Apr 18, 2023
3867129
🎉 New Source: Apple Search Ads [low-code CDK] (#19557)
rach-r Apr 18, 2023
f5b644c
🎉 New Source: Aircall [Low code CDK] (#25240)
btkcodedev Apr 18, 2023
5d20259
Increase memory_limit for Amplitude (#25282)
clnoll Apr 18, 2023
754e8ff
Adds 400 errors and reauth suggestion to Airtable doc (#25255)
jnr0790 Apr 18, 2023
f87f435
🐛 Source Jira: Add missing types to issues json schema (#25275)
henriblancke Apr 18, 2023
8945c25
CAT tools/acceptance_test_config_migration: Make relatively configura…
erohmensing Apr 18, 2023
59acd19
Dagster: rename catalog to registry (#25254)
bnchrch Apr 18, 2023
2d77e2b
Source Amplitude: Reduce request_time_range (#25281)
artem1205 Apr 18, 2023
439d0e3
increase postgres heartbeat from 3600->7200 (#25304)
colesnodgrass Apr 18, 2023
1bf47a7
connectors-ci: fix java connectors testing (#25268)
alafanechere Apr 19, 2023
4341b7d
Source Stripe: add account stream (#25136)
marcosmarxm Apr 19, 2023
b7b2bf5
remove session token auth (#25142)
Apr 19, 2023
8d1acad
Connector builder: Tutorial (#25153)
Apr 19, 2023
8c4ff39
Source Hubspot: update expected records (#25309)
arsenlosenko Apr 19, 2023
f1f3bfb
Bump Airbyte version from 0.44.0 to 0.44.1
josephkmh Apr 19, 2023
468fb51
:tada: Source Google Ads - fixed schema type for "data_type: ENUM" an…
midavadim Apr 19, 2023
bd03b3a
source-amplitude: Only fetch date_time_fields once per request (#25315)
clnoll Apr 19, 2023
822957a
postgres-cdc checkpointing: fix LSN parsing bug + refactor for effici…
sergio-ropero Apr 19, 2023
92eae00
use actual release tag (#25311)
josephkmh Apr 19, 2023
1e14f05
Source Twilio: update expected records (#25320)
arsenlosenko Apr 19, 2023
78cebca
Builder documentation: Pagination documentation (#25261)
lmossman Apr 19, 2023
919a1bc
🐛 Destination AWS Datalake: fix airbyte type in complex data types (#…
henriblancke Apr 19, 2023
5b397f7
Dagger: Return true on valid exit in Dagger Commands (#25333)
bnchrch Apr 19, 2023
34c44e8
update to new version of dagger cli (#25340)
Apr 19, 2023
62d417e
🎉Source Mailjet-Mail: Add `listrecipient` stream (#24689)
domzae Apr 19, 2023
4ecc6ac
24153 Destination Databricks: Fix support for external tables on S3 (…
suhomud Apr 19, 2023
c74e96f
Add version info to publish comment (#25343)
pedroslopez Apr 19, 2023
b83eb86
Update managing-airbyte-cloud.md (#25341)
rileybrook Apr 19, 2023
dcb7b66
Destination BigQuery + BigQuery denormalized: add configurable file b…
cynthiaxyin Apr 20, 2023
ce4fed1
[JDBC sources] : Log database indexes per stream (#25345)
akashkulk Apr 20, 2023
257b5b1
update link (#25349)
erohmensing Apr 20, 2023
d442c2f
connectors-ci: build multi platform connector images (#25286)
alafanechere Apr 20, 2023
f4a2cbc
🎉 New Destination: Cumul.io (#23153)
jooststessens Apr 20, 2023
8a39275
Source SFTP-Bulk: Support custom CSV separators (#19224)
anamargaridarl Apr 20, 2023
fd2f442
Source Amplitude: refactor Events stream (#25317)
artem1205 Apr 20, 2023
9a0115e
low-code: Do not apply transforms on AirbyteLogMessages and AirbyteTr…
girarda Apr 20, 2023
48749af
connectors-ci: use shared pip cache instead of locked (#25370)
alafanechere Apr 20, 2023
dd1ed17
fix typo in GHA workflow (#25372)
alafanechere Apr 20, 2023
75947fe
Dagster registry generation and persist (#25260)
bnchrch Apr 21, 2023
2ad5603
Destination Databricks: use hive_metastore if catalog not set (#25366)
edgao Apr 21, 2023
71f1088
Source Salesforce: update fields in Account expected_records (#25385)
arsenlosenko Apr 21, 2023
8a42731
Source Faker: update expected records (#25387)
arsenlosenko Apr 21, 2023
f6433d1
Source Freshsales: update expected records (#25388)
arsenlosenko Apr 21, 2023
9679e3f
Source Amplitude: update changelog (#25392)
artem1205 Apr 21, 2023
5c63fb3
Update Issues Templates (#25365)
marcosmarxm Apr 21, 2023
9f42a63
Docs: Add AWS_SECRET_MANAGER to list of available options for storing…
cjwooo Apr 21, 2023
53d1c14
Source Snapchat Marketing: Add missing story metrics to basic stats (…
iberchid Apr 21, 2023
98ba192
Fix Java destination template (#25390)
penberg Apr 21, 2023
e871381
Connector builder: Incremental sync documentation (#25238)
Apr 21, 2023
546013b
Conenctor builder: Record processing page (#25267)
Apr 21, 2023
a8a0030
🎉 New Destination: Starburst Galaxy (#25399)
marcosmarxm Apr 21, 2023
fc443cc
Add git checkout (#25376)
bnchrch Apr 21, 2023
3108cbd
Destination S3 Glue: Fix precision and scale of decimal type (#25397)
marcosmarxm Apr 21, 2023
2a72dbf
small docs fixes (#25377)
cynthiaxyin Apr 21, 2023
178d6c0
Add descriptions and examples to component schema (#25117)
girarda Apr 21, 2023
8bd3991
Fix and document macros and interpolation variables (#25305)
girarda Apr 21, 2023
0b6efe3
🤖 Bump patch version of Airbyte CDK
girarda Apr 21, 2023
32b1e3c
Source Twilio: update messages stream expected records (#25384)
arsenlosenko Apr 21, 2023
4899080
🎉 Source Stripe: add SetupIntents stream (#23963)
jj-nilbodee Apr 21, 2023
9f6b9aa
connectors-ci: document the CLI (#25391)
alafanechere Apr 22, 2023
3800290
connectors-ci: package normalization inside destinations images suppo…
alafanechere Apr 22, 2023
138088a
Update oss_catalog.json
btkcodedev Apr 23, 2023
b9b033c
fix conflicts
btkcodedev Apr 26, 2023
d6cf0ae
fix conflict oss
btkcodedev Apr 26, 2023
d65485c
fix conflict oss
btkcodedev Apr 26, 2023
54d3298
fix conflict oss
btkcodedev Apr 26, 2023
d341aa8
fix conflict oss
btkcodedev Apr 26, 2023
a5be2e3
Merge branch 'yotpo' of https://github.com/btkcodedev/airbyte into yotpo
btkcodedev Apr 26, 2023
675dd0d
Merge branch 'master' into yotpo
btkcodedev Apr 26, 2023
b1534b3
Merge branch 'master' into yotpo
btkcodedev Apr 26, 2023
9222b2a
fix reviews expected records, add EOF
btkcodedev Apr 26, 2023
95befd7
Merge branch 'yotpo' of https://github.com/btkcodedev/airbyte into yotpo
btkcodedev Apr 26, 2023
6d6c356
Merge branch 'airbytehq:master' into yotpo
btkcodedev Apr 27, 2023
00bf746
fix email_analytics schema
btkcodedev Apr 27, 2023
dcb06ce
Merge branch 'master' into yotpo
btkcodedev Apr 27, 2023
863bd43
fix expected records
btkcodedev Apr 27, 2023
cb87d81
Merge branch 'master' into yotpo
btkcodedev Apr 27, 2023
2f08c75
Merge branch 'yotpo' of https://github.com/btkcodedev/airbyte into yotpo
btkcodedev Apr 27, 2023
226d075
Merge branch 'master' into yotpo
btkcodedev May 1, 2023
aae23ed
Update sample_config.json
marcosmarxm May 1, 2023
8fa2cc9
Update source_definitions.yaml
marcosmarxm May 1, 2023
42a93a5
auto-bump connector version
octavia-squidington-iii May 1, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions airbyte-config-oss/init-oss/src/main/resources/icons/yotpo.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -2549,6 +2549,14 @@
allowedHosts:
hosts:
- api-metrica.yandex.net
- name: Yotpo
sourceDefinitionId: <To-be-given>
dockerRepository: airbyte/source-yotpo
dockerImageTag: 0.1.0
documentationUrl: https://docs.airbyte.com/integrations/sources/yotpo
icon: yotpo.svg
sourceType: api
releaseStage: alpha
- name: Younium
sourceDefinitionId: 9c74c2d7-531a-4ebf-b6d8-6181f805ecdc
dockerRepository: airbyte/source-younium
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18619,6 +18619,45 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-yotpo:0.1.0"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/yotpo"
connectionSpecification:
$schema: "http://json-schema.org/draft-07/schema#"
title: Yotpo Spec
type: object
required:
- access_token
- app_key
- start_date
- email
additionalProperties: true
properties:
access_token:
title: Access Token
type: string
description: Access token recieved as a result of API call to https://api.yotpo.com/oauth/token (Ref- https://apidocs.yotpo.com/reference/yotpo-authentication)
airbyte_secret: true
app_key:
title: App Key
type: string
description: App key found at settings (Ref- https://settings.yotpo.com/#/general_settings)
start_date:
title: Date-From Filter
type: string
description: Date time filter for incremental filter, Specify which date to extract from.
pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}Z$"
examples:
- "2022-03-01T00:00:00.000Z"
format: "date-time"
email:
title: Registered email address
type: string
description: Email address registered with yotpo.
default: example@gmail.com
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-younium:0.1.0"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/younium"
Expand Down
6 changes: 6 additions & 0 deletions airbyte-integrations/connectors/source-yotpo/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
*
!Dockerfile
!main.py
!source_yotpo
!setup.py
!secrets
38 changes: 38 additions & 0 deletions airbyte-integrations/connectors/source-yotpo/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
FROM python:3.9.11-alpine3.15 as base

# build and load all requirements
FROM base as builder
WORKDIR /airbyte/integration_code

# upgrade pip to the latest version
RUN apk --no-cache upgrade \
&& pip install --upgrade pip \
&& apk --no-cache add tzdata build-base


COPY setup.py ./
# install necessary packages to a temporary folder
RUN pip install --prefix=/install .

# build a clean environment
FROM base
WORKDIR /airbyte/integration_code

# copy all loaded and built libraries to a pure basic image
COPY --from=builder /install /usr/local
# add default timezone settings
COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
RUN echo "Etc/UTC" > /etc/timezone

# bash is installed for more convenient debugging.
RUN apk --no-cache add bash

# copy payload code only
COPY main.py ./
COPY source_yotpo ./source_yotpo

ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.0
LABEL io.airbyte.name=airbyte/source-yotpo
82 changes: 82 additions & 0 deletions airbyte-integrations/connectors/source-yotpo/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Yotpo Source

This is the repository for the Yotpo configuration based source connector.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/yotpo).

## Local development

#### Building via Gradle
You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.

To build using Gradle, from the Airbyte repository root, run:
```
./gradlew :airbyte-integrations:connectors:source-yotpo:build
```

#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/yotpo)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_yotpo/spec.yaml` file.
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
See `integration_tests/sample_config.json` for a sample config file.

**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source yotpo test creds`
and place them into `secrets/config.json`.

### Locally running the connector docker image

#### Build
First, make sure you build the latest Docker image:
```
docker build . -t airbyte/source-yotpo:dev
```

You can also build the connector image via Gradle:
```
./gradlew :airbyte-integrations:connectors:source-yotpo:airbyteDocker
```
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
the Dockerfile.

#### Run
Then run any of the connector commands as follows:
```
docker run --rm airbyte/source-yotpo:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-yotpo:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-yotpo:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-yotpo:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```
## Testing

#### Acceptance Tests
Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.

To run your integration tests with Docker, run:
```
./acceptance-test-docker.sh
```

### Using gradle to run tests
All commands should be run from airbyte project root.
To run unit tests:
```
./gradlew :airbyte-integrations:connectors:source-yotpo:unitTest
```
To run acceptance and custom integration tests:
```
./gradlew :airbyte-integrations:connectors:source-yotpo:integrationTest
```

## Dependency Management
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
We split dependencies between two groups, dependencies that are:
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
* required for the testing need to go to `TEST_REQUIREMENTS` list

### Publishing a new version of the connector
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing unit and integration tests.
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
1. Create a Pull Request.
1. Pat yourself on the back for being an awesome contributor.
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
3 changes: 3 additions & 0 deletions airbyte-integrations/connectors/source-yotpo/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference)
# for more information about how to configure these tests
connector_image: airbyte/source-yotpo:dev
acceptance_tests:
spec:
tests:
- spec_path: "source_yotpo/spec.yaml"
connection:
tests:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"
discovery:
tests:
- config_path: "secrets/config.json"
basic_read:
tests:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
empty_streams:
- name: "raw_data"
bypass_reason: "Sandbox account cannot seed the endpoint"
- name: "unsubscribers"
bypass_reason: "Sandbox account cannot seed the endpoint"
- name: "webhooks"
bypass_reason: "Sandbox account cannot seed the endpoint"
- name: "webhook_events"
bypass_reason: "Sandbox account cannot seed the endpoint"
expect_records:
path: "integration_tests/expected_records.jsonl"
extra_fields: no
exact_order: no
extra_records: yes
incremental:
tests:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
future_state:
future_state_path: "integration_tests/abnormal_state.json"
full_refresh:
tests:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/usr/bin/env sh

source "$(git rev-parse --show-toplevel)/airbyte-integrations/bases/connector-acceptance-test/acceptance-test-docker.sh"
9 changes: 9 additions & 0 deletions airbyte-integrations/connectors/source-yotpo/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
plugins {
id 'airbyte-python'
id 'airbyte-docker'
id 'airbyte-connector-acceptance-test'
}

airbytePython {
moduleDirectory 'source_yotpo'
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[
{
"type": "STREAM",
"stream": {
"stream_state": { "created_at": "2099-04-12T18:13:36.000Z" },
"stream_descriptor": { "name": "reviews" }
}
}
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import pytest

pytest_plugins = ("connector_acceptance_test.plugin",)


@pytest.fixture(scope="session", autouse=True)
def connector_setup():
"""This fixture is a placeholder for external resources that acceptance test might require."""
# TODO: setup test dependencies if needed. otherwise remove the TODO comments
yield
# TODO: clean up test dependencies
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
{
"streams": [
{
"stream": {
"name": "email_analytics",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "raw_data",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "reviews",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"]
},
"sync_mode": "incremental",
"destination_sync_mode": "append"
},
{
"stream": {
"name": "unsubscribers",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "webhooks",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "webhook_events",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"stream": "reviews", "data":{"id":465001595,"title":"Great Phone","content":"It’s really good","score":5,"votes_up":0,"votes_down":0,"created_at":"2023-04-26T21:46:31.000Z","updated_at":"2023-04-26T21:46:42.000Z","sentiment":null,"sku":"100","name":"John S.","email":"moles57397@snowlash.com","reviewer_type":"verified_reviewer","deleted":false,"archived":false,"escalated":false,"is_incentivized":false}, "emitted_at": 1679051186823}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"app_key": "<invalid>",
"access_token": "<invalid>",
"start_date": "2099-04-12T18:42:50.000Z",
"email": "<invalid>"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"app_key": "XXXXXXXXXXXXXXXXXXX",
"access_token": "YYYYYYYYYYYYYYYY",
"start_date": "2020-04-12T18:42:50.000Z",
"email": "example@gmail.com"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[
{
"type": "STREAM",
"stream": {
"stream_state": { "created_at": "2023-04-12T18:13:36.000Z" },
"stream_descriptor": { "name": "reviews" }
}
}
]
13 changes: 13 additions & 0 deletions airbyte-integrations/connectors/source-yotpo/main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk.entrypoint import launch
from source_yotpo import SourceYotpo

if __name__ == "__main__":
source = SourceYotpo()
launch(source, sys.argv[1:])
2 changes: 2 additions & 0 deletions airbyte-integrations/connectors/source-yotpo/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
-e ../../bases/connector-acceptance-test
-e .
Loading