Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix affected connectors #19865

Merged
merged 21 commits into from
Nov 30, 2022
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,14 @@
icon: appfollow.svg
sourceType: api
releaseStage: alpha
- name: AppsFlyer
sourceDefinitionId: 16447954-e6a8-4593-b140-43dea13bc457
dockerRepository: airbyte/source-appsflyer
dockerImageTag: 0.1.0
documentationUrl: https://docs.airbyte.com/integrations/sources/appsflyer
icon: appsflyer.svg
sourceType: api
releaseStage: alpha
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AppsFlyer was missing from the catalog

- name: Appstore
sourceDefinitionId: 2af123bf-0aaf-4e0d-9784-cb497f23741a
dockerRepository: airbyte/source-appstore-singer
Expand Down Expand Up @@ -202,7 +210,7 @@
- name: Chargebee
sourceDefinitionId: 686473f1-76d9-4994-9cc7-9b13da46147c
dockerRepository: airbyte/source-chargebee
dockerImageTag: 0.1.15
dockerImageTag: 0.1.16
documentationUrl: https://docs.airbyte.com/integrations/sources/chargebee
icon: chargebee.svg
sourceType: api
Expand Down Expand Up @@ -814,7 +822,7 @@
- name: Lemlist
sourceDefinitionId: 789f8e7a-2d28-11ec-8d3d-0242ac130003
dockerRepository: airbyte/source-lemlist
dockerImageTag: 0.1.0
dockerImageTag: 0.1.1
documentationUrl: https://docs.airbyte.com/integrations/sources/lemlist
sourceType: api
releaseStage: alpha
Expand Down Expand Up @@ -1109,6 +1117,14 @@
icon: outreach.svg
sourceType: api
releaseStage: alpha
- name: Pardot
sourceDefinitionId: ad15c7ba-72a7-440b-af15-b9a963dc1a8a
dockerRepository: airbyte/source-pardot
dockerImageTag: 0.1.0
documentationUrl: https://docs.airbyte.com/integrations/sources/pardot
icon: pardot.svg
sourceType: api
releaseStage: alpha
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pardot was missing from the catalog

- name: PagerDuty
sourceDefinitionId: 2817b3f0-04e4-4c7a-9f32-7a5e8a83db95
dockerRepository: farosai/airbyte-pagerduty-source
Expand Down Expand Up @@ -1237,15 +1253,15 @@
sourceDefinitionId: d60a46d4-709f-4092-a6b7-2457f7d455f5
dockerRepository: airbyte/source-prestashop
dockerImageTag: 0.3.0
documentationUrl: https://docs.airbyte.com/integrations/sources/presta-shop
documentationUrl: https://docs.airbyte.com/integrations/sources/prestashop
Comment on lines -1247 to +1261
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

documentation URLs need to match the connector name exactly

icon: prestashop.svg
sourceType: api
releaseStage: beta
- name: Primetric
sourceDefinitionId: f636c3c6-4077-45ac-b109-19fc62a283c1
dockerRepository: airbyte/source-primetric
dockerImageTag: 0.1.0
documentationUrl: https://docs.airbyte.com/integrations/sources/presta-shop
documentationUrl: https://docs.airbyte.com/integrations/sources/primetric
Comment on lines -1255 to +1269
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong doc url!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch! There could be more of those in there... Might be worth a [optional, hypothetical] script to validate that 🤔

icon: primetric.svg
sourceType: api
releaseStage: alpha
Expand Down Expand Up @@ -1289,7 +1305,7 @@
sourceDefinitionId: 29b409d9-30a5-4cc8-ad50-886eb846fea3
dockerRepository: airbyte/source-quickbooks-singer
dockerImageTag: 0.1.5
documentationUrl: https://docs.airbyte.com/integrations/sources/quickbooks
documentationUrl: https://docs.airbyte.com/integrations/sources/quickbooks-singer
Comment on lines -1299 to +1313
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

documentation URLs need to match the connector name exactly

icon: qb.svg
sourceType: api
releaseStage: alpha
Expand Down Expand Up @@ -1626,6 +1642,13 @@
documentationUrl: https://docs.airbyte.com/integrations/sources/the-guardian-api
sourceType: api
releaseStage: alpha
- name: TPLcentral
sourceDefinitionId: f9b6c538-ee12-42fe-8d4b-0c10f5955417
dockerRepository: airbyte/source-tplcentral
dockerImageTag: 0.1.1
documentationUrl: https://docs.airbyte.com/integrations/sources/tplcentral
sourceType: api
releaseStage: alpha
- name: Trello
sourceDefinitionId: 8da67652-004c-11ec-9a03-0242ac130003
dockerRepository: airbyte/source-trello
Expand Down
4 changes: 2 additions & 2 deletions airbyte-config/init/src/main/resources/seed/source_specs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11561,7 +11561,7 @@
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-prestashop:0.3.0"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/presta-shop"
documentationUrl: "https://docs.airbyte.com/integrations/sources/prestashop"
connectionSpecification:
$schema: "http://json-schema.org/draft-07/schema#"
title: "PrestaShop Spec"
Expand Down Expand Up @@ -11804,7 +11804,7 @@
- - "key"
- dockerImage: "airbyte/source-quickbooks-singer:0.1.5"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/quickbooks"
documentationUrl: "https://docs.airbyte.com/integrations/sources/quickbooks-singer"
connectionSpecification:
$schema: "http://json-schema.org/draft-07/schema#"
title: "Source QuickBooks Singer Spec"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11208,7 +11208,7 @@
"name": "Prestashop",
"dockerRepository": "airbyte/source-prestashop",
"dockerImageTag": "0.1.0",
"documentationUrl": "https://docs.airbyte.io/integrations/sources/presta-shop",
"documentationUrl": "https://docs.airbyte.io/integrations/sources/prestashop",
"icon": "prestashop.svg",
"sourceType": "api",
"spec": {
Expand Down
45 changes: 41 additions & 4 deletions airbyte-integrations/connectors/source-prestashop/README.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,32 @@
# PrestaShop Source

This is the repository for the Presta Shop source connector, written in Python.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/presta-shop).
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/prestashop).

## Local development

### Prerequisites

**To iterate on this connector, make sure to complete this prerequisites section.**

#### Minimum Python version required `= 3.7.0`

#### Build & Activate Virtual Environment and install dependencies

From this connector directory, create a virtual environment:

```
python -m venv .venv
```

This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
development environment of choice. To activate it from the terminal, run:

```
source .venv/bin/activate
pip install -r requirements.txt
```

If you are in an IDE, follow your IDE's instructions to activate the virtualenv.

Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
Expand All @@ -30,15 +35,18 @@ If this is mumbo jumbo to you, don't worry about it, just put your deps in `setu
should work as you expect.

#### Building via Gradle

You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.

To build using Gradle, from the Airbyte repository root, run:

```
./gradlew :airbyte-integrations:connectors:source-prestashop:build
```

#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/presta-shop)

**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/prestashop)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_prestashop/spec.json` file.
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
See `integration_tests/sample_config.json` for a sample config file.
Expand All @@ -47,6 +55,7 @@ See `integration_tests/sample_config.json` for a sample config file.
and place them into `secrets/config.json`.

### Locally running the connector

```
python main.py spec
python main.py check --config secrets/config.json
Expand All @@ -57,73 +66,101 @@ python main.py read --config secrets/config.json --catalog integration_tests/con
### Locally running the connector docker image

#### Build

First, make sure you build the latest Docker image:

```
docker build . -t airbyte/source-prestashop:dev
```

You can also build the connector image via Gradle:

```
./gradlew :airbyte-integrations:connectors:source-prestashop:airbyteDocker
```

When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
the Dockerfile.

#### Run

Then run any of the connector commands as follows:

```
docker run --rm airbyte/source-prestashop:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-prestashop:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-prestashop:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-prestashop:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```

## Testing

Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
First install test dependencies into your virtual environment:

```
pip install .[tests]
```

### Unit Tests

To run unit tests locally, from the connector directory run:

```
python -m pytest unit_tests
```

### Integration Tests

There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector).

#### Custom Integration tests

Place custom tests inside `integration_tests/` folder, then, from the connector root, run

```
python -m pytest integration_tests
```

#### Acceptance Tests

Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/source-acceptance-tests-reference) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
To run your integration tests with acceptance tests, from the connector root, run

```
python -m pytest integration_tests -p integration_tests.acceptance
```

To run your integration tests with docker

### Using gradle to run tests

All commands should be run from airbyte project root.
To run unit tests:

```
./gradlew :airbyte-integrations:connectors:source-prestashop:unitTest
```

To run acceptance and custom integration tests:

```
./gradlew :airbyte-integrations:connectors:source-prestashop:integrationTest
```

## Dependency Management

All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
We split dependencies between two groups, dependencies that are:
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
* required for the testing need to go to `TEST_REQUIREMENTS` list

- required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
- required for the testing need to go to `TEST_REQUIREMENTS` list

### Publishing a new version of the connector

You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?

1. Make sure your changes are passing unit and integration tests.
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
1. Create a Pull Request.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
documentationUrl: https://docs.airbyte.com/integrations/sources/presta-shop
documentationUrl: https://docs.airbyte.com/integrations/sources/prestashop
connectionSpecification:
$schema: http://json-schema.org/draft-07/schema#
title: PrestaShop Spec
Expand Down
Loading