Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎉 Source Zendesk: Migration from Singer to CDK #4861

Merged
merged 174 commits into from
Aug 2, 2021
Merged
Show file tree
Hide file tree
Changes from 166 commits
Commits
Show all changes
174 commits
Select commit Hold shift + click to select a range
ab16590
init the new connector source-zendesk-support
gl-pix Jul 15, 2021
aeb73d6
Finished a development of ZenDesk streams
gl-pix Jul 19, 2021
964382d
Source ZenDesk: finished
gl-pix Jul 20, 2021
7ed6cb2
Source ZenDesk: remove unused test files
gl-pix Jul 20, 2021
46aeb90
Merge branch 'master' of github.com:airbytehq/airbyte into antixar/44…
gl-pix Jul 20, 2021
62c2cd4
Source ZenDesk: format and validate code
gl-pix Jul 20, 2021
001be7f
Merge branch 'master' of github.com:airbytehq/airbyte into antixar/44…
gl-pix Jul 21, 2021
d148beb
Source Zendesk: update docs
gl-pix Jul 21, 2021
f7bd7d6
Remove unused files
gl-pix Jul 21, 2021
4d19040
add a stream_slices logic for ticket_comments stream
gl-pix Jul 22, 2021
f02d74a
🎉 Python CDK: Allow setting network adapter args on outgoing HTTP req…
sherifnada Jul 7, 2021
a9ae0eb
🎉 Destination S3: support `anyOf` `allOf` and `oneOf` (#4613)
tuliren Jul 7, 2021
1cdf2aa
SAT: verify `AIRBYTE_ENTRYPOINT` is defined (#4478)
vovavovavovavova Jul 7, 2021
198f4db
Migrate Quickstart to use PokeAPI (#4615)
avaidyanatha Jul 8, 2021
6704488
Left isn't right (#4616)
avaidyanatha Jul 8, 2021
abae392
Create on on-oci-vm.md (#4468)
shadabshaukat Jul 8, 2021
dcefea5
🐛 platform: Fix silent failures in sources (#4617)
sherifnada Jul 8, 2021
8c471ba
add oracle dpeloyment guide to summary.md (#4619)
sherifnada Jul 8, 2021
fad4b75
Mailchimp fix url-base (#4621)
vovavovavovavova Jul 8, 2021
83e26b0
🎉 New Source: Paypal Transaction (#4240)
midavadim Jul 8, 2021
f63066a
set db version after full import is complete (#4626)
subodh1810 Jul 8, 2021
95c8c05
Fix docs formatting
sherifnada Jul 8, 2021
dfcd1e1
Redirect old link to upgrading tutorial (#4635)
avaidyanatha Jul 8, 2021
a96bbad
Fix broken link in SUMMARY.md
sherifnada Jul 8, 2021
8d27adf
Airflow Demo: Remove superset in down.sh (#4638)
avaidyanatha Jul 8, 2021
00ffe20
Airflow demo: Clean up scripts and more clearly describe actions (#4639)
avaidyanatha Jul 8, 2021
52593f7
:tada: Add documentation for configuring Kube GCS logging. (#4622)
davinchia Jul 9, 2021
c7c26a0
Bump version: 0.27.0-alpha → 0.27.1-alpha (#4640)
davinchia Jul 9, 2021
8c96b5d
0.27.1 Platform Patch Notes (#4644)
avaidyanatha Jul 9, 2021
47fe87c
🎉 New Source: Zendesk Sunshine (#4359)
vovavovavovavova Jul 9, 2021
8d949bb
0.27.1 Connector Patch Notes (#4646)
avaidyanatha Jul 9, 2021
3dfe5cb
Update connector certification table. (#4647)
avaidyanatha Jul 9, 2021
80c3dcd
:bug: Stub out the GCP Env Var in Docker to prevent noisy and harmles…
davinchia Jul 9, 2021
19c03de
Add hint to Airflow guide about local example (#4656)
avaidyanatha Jul 9, 2021
86f05e4
fix version for kube automatic migration support (#4649)
subodh1810 Jul 9, 2021
f1304fa
format zendesk sunshine connector (#4658)
subodh1810 Jul 9, 2021
9d3dd2f
🎉 New source: Dixa (#4358)
olivermeyer Jul 9, 2021
bb0f808
Turn on MYSQL normalization flag. (#4651)
davinchia Jul 9, 2021
d7eafe5
Combine admin and settings (#4525)
jamakase Jul 9, 2021
61d597d
Job history purging (#4575)
airbyte-jenny Jul 9, 2021
551038b
0.27.1 Announcement Summary (#4678)
avaidyanatha Jul 9, 2021
f19f5d7
🐛 Source Sendgrid: add start_time config and correct primary_key (#4682)
marcosmarxm Jul 10, 2021
fa2ea50
test to show how automatic migration handles deprecated definitions (…
subodh1810 Jul 12, 2021
5a8b185
Source dixa: fix unit tests (#4690)
olivermeyer Jul 12, 2021
6bf9028
introduce common abstraction for CDC via debezium (#4580)
subodh1810 Jul 12, 2021
c59c1da
Source Dixa: Pin tz in ConversationExport.ms_timestamp_to_datetime (#…
olivermeyer Jul 12, 2021
c074c8e
Source Dixa: add to connector index (#4701)
sherifnada Jul 12, 2021
9e1575a
allow injecting filters for server (#4677)
jrhizor Jul 12, 2021
4b03782
upgrade postgres version for new cdc abstraction (#4702)
subodh1810 Jul 12, 2021
4cefe23
Fix dependencies for Superset demo (#4705)
avaidyanatha Jul 12, 2021
eacd887
📚 add SSH instructions for OCI VM setup (#4684)
shadabshaukat Jul 12, 2021
c085949
upgrade mysql version for new cdc abstraction (#4703)
subodh1810 Jul 12, 2021
0ca57e8
Update with ALTER TABLE statements (#4707)
avaidyanatha Jul 12, 2021
0150c03
remove unused deps (#4512)
cgardens Jul 12, 2021
af1b9d5
fix config init race condition (#4679)
jrhizor Jul 12, 2021
8ac0588
🐛 Destination S3: fix minio output for parquet format
varunbpatil Jul 13, 2021
2875270
Bump destination s3 version (#4718)
tuliren Jul 13, 2021
8e11a09
Fix scheduler race condition. (#4691)
davinchia Jul 13, 2021
d845536
Periodic connector tests workflow: add `Accept` header per github doc…
sherifnada Jul 13, 2021
36d85e8
allow launching integration tests from workflow dispatch (#4723)
sherifnada Jul 13, 2021
98eb547
Bump version: 0.27.1-alpha → 0.27.2-alpha (#4724)
davinchia Jul 13, 2021
44f07ca
🐛 Source Square: Update _send_request method due to changes in Airbyt…
TymoshokDmytro Jul 13, 2021
9441ff9
🎉 Destination Snowflake: tag snowflake traffic with airbyte ID to ena…
sherifnada Jul 13, 2021
710ab30
🎉 New source: Typeform (#4541)
yaroslav-dudar Jul 13, 2021
df01bb1
Upgrade postgres and redshift destination to remove basic_normalizati…
subodh1810 Jul 13, 2021
0b17e6b
fix broken assertions for automatic migration tests (#4732)
subodh1810 Jul 13, 2021
af9db0b
Slightly improve sed-based yaml parsing (#4721)
bkrausz Jul 13, 2021
a5bc4a9
throw exception if we close engine before snapshot is complete + incr…
subodh1810 Jul 13, 2021
fad9432
allow publishing airbyte-server to local maven repo (#4717)
jrhizor Jul 13, 2021
9284b13
CDK: Add initial Destination abstraction and tests (#4719)
sherifnada Jul 13, 2021
460b63a
Update docs on GitHub connector now that its Airbyte native (#4739)
avaidyanatha Jul 14, 2021
b3f25ef
Remove statement about Postgres connector being based on Singer (#4740)
avaidyanatha Jul 14, 2021
2074c61
fix flaky migration acceptance test (#4743)
subodh1810 Jul 14, 2021
9f98d42
upgrade fabric8 client (#4738)
jrhizor Jul 14, 2021
81467bb
🎉 Source MSSQL: implementation for CDC (#4689)
subodh1810 Jul 14, 2021
ab140b1
bump up MSSQL version for cdc (#4694)
subodh1810 Jul 14, 2021
2f31260
fixed broken links and styling (#4747)
Phlair Jul 14, 2021
e933641
Fix enabling connection in refresh catalog mode (#4527)
jamakase Jul 14, 2021
6bf15bf
Do not update deprecated connectors (#4674)
jamakase Jul 14, 2021
3383d2b
🐛 Source Slack: add float_ts field (#4683)
marcosmarxm Jul 14, 2021
48ea0f8
copy docs to webapp docker image (#4522)
cgardens Jul 14, 2021
5a9210a
use kube service user for pod sweeper (#4737)
jrhizor Jul 14, 2021
15f980d
🐛 Source GitHub: fix bug with `IssueEvents` stream and add handling f…
Zirochkaa Jul 15, 2021
daf5fab
:bug: Fix some api-spec errors. (#4742)
davinchia Jul 15, 2021
61889fa
Source PostHog: Use account information for checking the connection (…
vovavovavovavova Jul 15, 2021
da173b1
SAT: Improve error message when data mismatches schema (#4753)
keu Jul 15, 2021
029a5f4
increase sleep duration + show logs in CI (#4756)
subodh1810 Jul 15, 2021
45efe29
Fixed cockroachdb repo image (#4758)
etsybaev Jul 15, 2021
dc8ed7e
Bump version: 0.27.2-alpha → 0.27.3-alpha (#4761)
davinchia Jul 15, 2021
3142732
update kube docs (#4749)
jrhizor Jul 15, 2021
778818f
fix kube overlay version (#4765)
jrhizor Jul 15, 2021
44c3b62
Split Platform and Connector Builds (#4514)
cgardens Jul 15, 2021
5d77867
remove second docs check in build(#4766)
cgardens Jul 15, 2021
48a3085
Restore template generator and fix formatting. (#4768)
cgardens Jul 15, 2021
bdebcdd
connector generate: fix chown logic (#4774)
sherifnada Jul 15, 2021
0db91db
Remove example use cases from docs (#4775)
avaidyanatha Jul 16, 2021
9644175
Update README.md
johnlafleur Jul 16, 2021
bc2be6d
🎉 All java connectors: Added configValidator to check, discover, read…
etsybaev Jul 16, 2021
d15a4f2
🎉 Stripe Source: Fix subscriptions stream to return all kinds of subs…
annalvova05 Jul 16, 2021
7f87b56
Add note about orphaned Airbyte configs preventing automatic upgrades…
avaidyanatha Jul 16, 2021
9733df2
Patch 0.27.2 and 0.27.3 platform notes (#4792)
avaidyanatha Jul 16, 2021
6b15f32
Connector notes for 0.27.3 (#4794)
avaidyanatha Jul 16, 2021
e2d4b4f
Add new logo to GitHub page (#4796)
avaidyanatha Jul 16, 2021
9d7ab35
🎉 New Destination: Google Cloud Storage (#4784)
tuliren Jul 17, 2021
104f940
🐛 CDK: Fix logging of initial state value (#4795)
keu Jul 18, 2021
6eb98b2
bug fix: use register api (#4811)
cgardens Jul 18, 2021
d44d401
🐛 Add missing dependencies for acceptance tests to run. (#4808)
davinchia Jul 19, 2021
3d33387
🎉 Add Python Destination Template (#4771)
sherifnada Jul 19, 2021
4a2204c
Format. (#4814)
davinchia Jul 19, 2021
9f3fcf2
🎉 Migrate config persistence to database (#4670)
tuliren Jul 19, 2021
19e0ddf
🎉 Source intercom: migration to CDK (#4676)
lazebnyi Jul 19, 2021
5658797
🎉 New source: Pipedrive connector (#4686)
htrueman Jul 19, 2021
b57a303
Make number of Concurrent Jobs configurable. (#4687)
davinchia Jul 19, 2021
7a56b53
Explicitly pin ec2 runner version to 2.2.1. (#4823)
davinchia Jul 19, 2021
2b077ec
🐛 Source Facebook: Improve rate limit management (#4820)
vladimir-remar Jul 19, 2021
307b0f2
format and fix
eugene-kulak Jul 19, 2021
e413ed6
Source Facebook: fix formatting and publish new version (#4826)
keu Jul 19, 2021
52953b6
Code generator: Update generator to chown docs and config definition …
TymoshokDmytro Jul 19, 2021
89edad1
Python Demo Destination: KVDB (#4786)
sherifnada Jul 19, 2021
cdd3a7d
📚 CDK: Add python destination tutorial (#4800)
sherifnada Jul 19, 2021
d5ff70c
📚 Source Shopify: migrate to new sandbox, update API version to 2021-…
bazarnov Jul 19, 2021
09baa5b
🐛 Source Instagram: Read previous state format and upgrade it (#4805)
keu Jul 19, 2021
0ebb913
Add placeholder (#4816)
jamakase Jul 19, 2021
8930074
Add update button (#4809)
jamakase Jul 19, 2021
fb739b0
Point to new location for connector build status history (#4840)
sherifnada Jul 19, 2021
a92554d
Update GAds docs to indicate incremental support
sherifnada Jul 19, 2021
4eecba3
Add openreplay (#4685)
jamakase Jul 19, 2021
bc22519
🎉 Source mixpanel: migration to CDK (#4566)
midavadim Jul 19, 2021
8ada997
Add openreplay variable (#4844)
jamakase Jul 20, 2021
0cdb485
🐛 Sendgrid source: Gracefully handle malformed responses from sendgr…
sherifnada Jul 20, 2021
d96c781
Update job description (#4848)
johnlafleur Jul 20, 2021
77d880c
Add py destination tutorial to summary.md (#4853)
sherifnada Jul 20, 2021
ff29d2f
Update CHANGELOG.md
sherifnada Jul 20, 2021
37052be
🐛 Kube: Fix Source Ports not releasing. (#4822)
davinchia Jul 20, 2021
6a8c9a0
use new AMI ID for connector builds (#4855)
sherifnada Jul 20, 2021
31c8f9a
Wait for config volume to be ready (#4835)
tuliren Jul 20, 2021
4065fa6
🎉 New source: US census (#4228)
dmateusp Jul 20, 2021
eccf1c0
publish US Census (connector) (#4857)
sherifnada Jul 20, 2021
21b2e77
🐛 Source JIRA: Fix DBT failing normalization on `Labels` schema. (#4817)
bazarnov Jul 20, 2021
6af1e5b
Rename founding-account-executive to founding-account-executive.md
johnlafleur Jul 20, 2021
1ebc9ba
Tweak ConfigNotFoundException class (#4821)
ChristopheDuong Jul 20, 2021
feb3d3c
Source ZenDesk: format and validate code
gl-pix Jul 20, 2021
43013b9
refactor import / export endpoints to use the same code path as auto …
cgardens Jul 20, 2021
435b3f2
fix build (#4865)
cgardens Jul 20, 2021
657b804
📝 Add server version requirement for mysql normalization (#4856)
tuliren Jul 20, 2021
aa62bac
🐛 Destination MySQL: fix problem if source has a column with json (#4…
etsybaev Jul 20, 2021
ed35fe5
hotfix: rename senior PM file to add .md
jrhizor Jul 20, 2021
ada4ddc
📚 improve mongo docs and param descriptions (#4870)
sherifnada Jul 20, 2021
32a9e60
Remove duplicated seed repository (#4869)
tuliren Jul 20, 2021
768aad0
add workspace helper (#4868)
jrhizor Jul 20, 2021
09566d9
🐛 Fix Oracle spec to declare `sid` instead of `database` param, Redsh…
sherifnada Jul 21, 2021
45fb8da
Kube: Better Port Abstraction. (#4829)
davinchia Jul 21, 2021
88898e8
Source Zendesk: update docs
gl-pix Jul 21, 2021
366d2c1
Remove unused files
gl-pix Jul 21, 2021
4b4a5c0
add a stream_slices logic for ticket_comments stream
gl-pix Jul 22, 2021
71c542e
Merge branch 'antixar/4422-source-zendesk-support' of github.com:airb…
gl-pix Jul 22, 2021
382806f
Merge branch 'master' of github.com:airbytehq/airbyte into antixar/44…
gl-pix Jul 22, 2021
76cad71
remove changes of other connections
gl-pix Jul 22, 2021
5c72e3f
Merge branch 'master' of github.com:airbytehq/airbyte into antixar/44…
gl-pix Jul 22, 2021
0f0c0a1
add secret Zendesk keys to command configs
gl-pix Jul 22, 2021
2ac7666
:bug: Source Zendesk Support: add dummy unit test
gl-pix Jul 22, 2021
25eab39
add dummy integration test
gl-pix Jul 22, 2021
e814042
fix Zendesk not loading username and facebook/twitter id #4373
gl-pix Jul 23, 2021
d0710ee
sort streams alphabetically
gl-pix Jul 23, 2021
7fd842b
fix test issue with the unsupport field validate_output_from_all_streams
gl-pix Jul 23, 2021
bb22bee
add info to source_definitions.yaml
gl-pix Jul 27, 2021
28e899a
remove json_schema from configured_catalog.json
gl-pix Jul 27, 2021
bd75114
add backoff logic
gl-pix Jul 29, 2021
41209e8
Merge remote-tracking branch 'origin/master' into antixar/4422-source…
gl-pix Jul 29, 2021
324d7c9
add unit tests
gl-pix Jul 30, 2021
4d9156f
Merge remote-tracking branch 'origin/master' into antixar/4422-source…
gl-pix Aug 2, 2021
d597161
move part of unit tests to integration tests
gl-pix Aug 2, 2021
b93620e
fix test dependencies
gl-pix Aug 2, 2021
2a461a9
add a build status
gl-pix Aug 2, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/publish-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@ jobs:
ZENDESK_SECRETS_CREDS: ${{ secrets.ZENDESK_SECRETS_CREDS }}
ZENDESK_SUNSHINE_TEST_CREDS: ${{ secrets.ZENDESK_SUNSHINE_TEST_CREDS }}
ZENDESK_TALK_TEST_CREDS: ${{ secrets.ZENDESK_TALK_TEST_CREDS }}
ZENDESK_SUPPORT_TEST_CREDS: ${{ secrets.ZENDESK_SUPPORT_TEST_CREDS }}
ZOOM_INTEGRATION_TEST_CREDS: ${{ secrets.ZOOM_INTEGRATION_TEST_CREDS }}
PLAID_INTEGRATION_TEST_CREDS: ${{ secrets.PLAID_INTEGRATION_TEST_CREDS }}
DESTINATION_S3_INTEGRATION_TEST_CREDS: ${{ secrets.DESTINATION_S3_INTEGRATION_TEST_CREDS }}
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/test-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ jobs:
ZENDESK_SECRETS_CREDS: ${{ secrets.ZENDESK_SECRETS_CREDS }}
antixar marked this conversation as resolved.
Show resolved Hide resolved
ZENDESK_SUNSHINE_TEST_CREDS: ${{ secrets.ZENDESK_SUNSHINE_TEST_CREDS }}
ZENDESK_TALK_TEST_CREDS: ${{ secrets.ZENDESK_TALK_TEST_CREDS }}
ZENDESK_SUPPORT_TEST_CREDS: ${{ secrets.ZENDESK_SUPPORT_TEST_CREDS }}
ZOOM_INTEGRATION_TEST_CREDS: ${{ secrets.ZOOM_INTEGRATION_TEST_CREDS }}
PLAID_INTEGRATION_TEST_CREDS: ${{ secrets.PLAID_INTEGRATION_TEST_CREDS }}
DESTINATION_S3_INTEGRATION_TEST_CREDS: ${{ secrets.DESTINATION_S3_INTEGRATION_TEST_CREDS }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -182,11 +182,18 @@
documentationUrl: https://hub.docker.com/r/airbyte/source-zendesk-chat
icon: zendesk.svg
- sourceDefinitionId: d29764f8-80d7-4dd7-acbe-1a42005ee5aa
name: Zendesk Support
name: Zendesk Support Singer
antixar marked this conversation as resolved.
Show resolved Hide resolved
dockerRepository: airbyte/source-zendesk-support-singer
dockerImageTag: 0.2.3
documentationUrl: https://hub.docker.com/r/airbyte/source-zendesk-support-singer
icon: zendesk.svg
- sourceDefinitionId: 79c1aa37-dae3-42ae-b333-d1c105477715
name: Zendesk Support
dockerRepository: airbyte/source-zendesk-support
dockerImageTag: 0.1.0
documentationUrl: https://hub.docker.com/r/airbyte/source-zendesk-support
icon: zendesk.svg

- sourceDefinitionId: d8313939-3782-41b0-be29-b3ca20d8dd3a
name: Intercom
dockerRepository: airbyte/source-intercom
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
*
!Dockerfile
!Dockerfile.test
!main.py
!source_zendesk_support
!setup.py
!secrets
25 changes: 25 additions & 0 deletions airbyte-integrations/connectors/source-zendesk-support/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
FROM python:3.7.11-alpine3.14 as base
FROM base as builder


RUN apk --no-cache upgrade \
&& pip install --upgrade pip

WORKDIR /airbyte/integration_code
COPY setup.py ./
RUN pip install --prefix=/install .


FROM base
COPY --from=builder /install /usr/local

WORKDIR /airbyte/integration_code
COPY main.py ./
COPY source_zendesk_support ./source_zendesk_support


ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.0
LABEL io.airbyte.name=airbyte/source-zendesk-support
131 changes: 131 additions & 0 deletions airbyte-integrations/connectors/source-zendesk-support/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# Source Zendesk Support Source

This is the repository for the Source Zendesk Support source connector, written in Python.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/source-zendesk-support).

## Local development

### Prerequisites
**To iterate on this connector, make sure to complete this prerequisites section.**

#### Minimum Python version required `= 3.7.0`

#### Build & Activate Virtual Environment and install dependencies
From this connector directory, create a virtual environment:
```
python -m venv .venv
```

This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
development environment of choice. To activate it from the terminal, run:
```
source .venv/bin/activate
pip install -r requirements.txt
```
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.

Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
should work as you expect.

#### Building via Gradle
You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.

To build using Gradle, from the Airbyte repository root, run:
```
./gradlew :airbyte-integrations:connectors:source-zendesk-support:build
```

#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/source-zendesk-support)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_zendesk_support/spec.json` file.
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
See `integration_tests/sample_config.json` for a sample config file.

**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source source-zendesk-support test creds`
and place them into `secrets/config.json`.

### Locally running the connector
```
python main.py spec
python main.py check --config secrets/config.json
python main.py discover --config secrets/config.json
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
```

### Locally running the connector docker image

#### Build
First, make sure you build the latest Docker image:
```
docker build . -t airbyte/source-zendesk-support:dev
```

You can also build the connector image via Gradle:
```
./gradlew :airbyte-integrations:connectors:source-zendesk-support:airbyteDocker
```
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
the Dockerfile.

#### Run
Then run any of the connector commands as follows:
```
docker run --rm airbyte/source-zendesk-support:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-zendesk-support:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-zendesk-support:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-zendesk-support:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```
## Testing
Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
First install test dependencies into your virtual environment:
```
pip install .[tests]
```
### Unit Tests
To run unit tests locally, from the connector directory run:
```
python -m pytest unit_tests
```

### Integration Tests
There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector).
#### Custom Integration tests
Place custom tests inside `integration_tests/` folder, then, from the connector root, run
```
python -m pytest integration_tests
```
#### Acceptance Tests
Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](source-acceptance-tests.md) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
To run your integration tests with acceptance tests, from the connector root, run
```
python -m pytest integration_tests -p integration_tests.acceptance
```
To run your integration tests with docker

### Using gradle to run tests
All commands should be run from airbyte project root.
To run unit tests:
```
./gradlew :airbyte-integrations:connectors:source-zendesk-support:unitTest
```
To run acceptance and custom integration tests:
```
./gradlew :airbyte-integrations:connectors:source-zendesk-support:integrationTest
```

## Dependency Management
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
We split dependencies between two groups, dependencies that are:
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
* required for the testing need to go to `TEST_REQUIREMENTS` list

### Publishing a new version of the connector
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing unit and integration tests.
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
1. Create a Pull Request.
1. Pat yourself on the back for being an awesome contributor.
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# See [Source Acceptance Tests](https://docs.airbyte.io/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md)
# for more information about how to configure these tests
connector_image: airbyte/source-zendesk-support:dev
tests:
spec:
- spec_path: "source_zendesk_support/spec.json"
connection:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"
discovery:
- config_path: "secrets/config.json"
basic_read:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
incremental:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
future_state_path: "integration_tests/abnormal_state.json"
antixar marked this conversation as resolved.
Show resolved Hide resolved
cursor_paths:
ticket_comments: ["created_at"]
full_refresh:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/usr/bin/env sh
../../../../discover2catalog.sh main.py ./secrets/config.json ./integration_tests/configured_catalog.json
docker build . -t airbyte/source-zendesk-support:dev
antixar marked this conversation as resolved.
Show resolved Hide resolved

docker run --rm -it \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp:/tmp \
-v $(pwd):/test_input \
airbyte/source-acceptance-test \
--acceptance-test-config /test_input
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
plugins {
id 'airbyte-python'
id 'airbyte-docker'
id 'airbyte-source-acceptance-test'
}

airbytePython {
moduleDirectory 'source_zendesk_support'
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"users": {
"updated_at": "2022-07-19T22:21:37Z"
},
"groups": {
"updated_at": "2022-07-15T22:19:01Z"
},
"organizations": {
"updated_at": "2022-07-15T19:29:14Z"
},
"satisfaction_ratings": {
"updated_at": "2022-07-20T10:05:18Z"
},
"tickets": {
"updated_at": "2022-07-19T22:21:26Z"
},
"group_memberships": {
"updated_at": "2022-04-23T15:34:20Z"
},
"ticket_fields": {
"updated_at": "2022-12-11T19:34:05Z"
},
"ticket_forms": {
"updated_at": "2022-12-11T20:34:37Z"
},
"ticket_metrics": {
"updated_at": "2022-07-19T22:21:26Z"
},
"macros": {
"updated_at": "2022-12-11T19:34:06Z"
},
"ticket_comments": {
"created_at": "2022-07-19T22:21:26Z"
},
"ticket_audits": {
"created_at": "2022-07-19T22:21:26Z"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#
# MIT License
#
# Copyright (c) 2020 Airbyte
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
#


import pytest

pytest_plugins = ("source_acceptance_test.plugin",)


@pytest.fixture(scope="session", autouse=True)
def connector_setup():
""" This fixture is a placeholder for external resources that acceptance test might require."""
yield
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#
# MIT License
#
# Copyright (c) 2020 Airbyte
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
#


def test_dummy_test():
"""This test added for successful passing customIntegrationTests"""
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"email": "broken.email@invalid.config",
"api_token": "<broken token>",
"subdomain": "test-failure-airbyte",
"start_date": "2030-01-01T00:00:00Z"
}
Loading