Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Datadog: Update CDK version and dependencies, remove parameters and migrate to inline schemas #44371

Merged
merged 11 commits into from
Aug 20, 2024
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-datadog/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,15 @@ poetry install --with dev
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/datadog)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_datadog/spec.yaml` file.
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
See `sample_files/sample_config.json` for a sample config file.
See `integration_tests/sample_config.json` for a sample config file.


### Locally running the connector
```
poetry run source-datadog spec
poetry run source-datadog check --config secrets/config.json
poetry run source-datadog discover --config secrets/config.json
poetry run source-datadog read --config secrets/config.json --catalog sample_files/configured_catalog.json
poetry run source-datadog read --config secrets/config.json --catalog integration_tests/configured_catalog.json
```

### Running unit tests
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ acceptance_tests:
spec:
tests:
- spec_path: "source_datadog/spec.yaml"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You removed the spec file, so the spec path here should lead to the manifest, I think. /cc @ChristoGrab

backward_compatibility_tests_config:
disable_for_version: "0.4.15" # Set default start and end date for incremental sync
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory, you only want to disable the backwards compatibility tests if the new version will not work, but the old one worked.

If the new version provides defaults and works in a scenario when a new one does not, why would it be broken?

Also, usually if you're skipping backwards compat checks, this should signal that the change is breaking, which this is not.

BUT, your build step in CI is failing on something something date can't be parsed something, so seems relevant.

connection:
tests:
- config_path: "secrets/config.json"
Expand All @@ -25,6 +27,8 @@ acceptance_tests:
bypass_reason: Sandbox account can't seed this stream
- name: service_level_objectives
bypass_reason: Sandbox account can't seed this stream
- name: logs
bypass_reason: Sandbox account can't seed this stream
incremental:
bypass_reason: "This connector does not implement incremental sync"
full_refresh:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"site": "datadoghq.com",
"api_key": "<api_key>",
"application_key": "<api_key>",
"application_key": "<application_key>",
"query": "sample query",
"limit": 100,
"start_date": "2022-10-10T00:00:00Z",
Expand Down
7 changes: 6 additions & 1 deletion airbyte-integrations/connectors/source-datadog/metadata.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,15 @@ data:
enabled: true
cloud:
enabled: true
releases:
breakingChanges:
1.0.0:
message: "Spec and schema are inline now, and default start and end date is setup for incremental sync."
upgradeDeadline: "2024-09-18"
connectorSubtype: api
connectorType: source
definitionId: 1cfc30c7-82db-43f4-9fd7-ac1b42312cda
dockerImageTag: 0.4.15
dockerImageTag: 1.0.0
dockerRepository: airbyte/source-datadog
githubIssueLabel: source-datadog
icon: datadog.svg
Expand Down
398 changes: 303 additions & 95 deletions airbyte-integrations/connectors/source-datadog/poetry.lock

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions airbyte-integrations/connectors/source-datadog/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
version = "0.4.15"
version = "1.0.0"
name = "source-datadog"
description = "Source implementation for Datadog."
authors = [ "Airbyte <contact@airbyte.io>",]
Expand All @@ -16,13 +16,13 @@ repository = "https://github.com/airbytehq/airbyte"
include = "source_datadog"

[tool.poetry.dependencies]
python = "^3.9,<3.12"
airbyte-cdk = "^1"
python = "^3.10,<3.12"
airbyte-cdk = "^4"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feeling adventurous I see


[tool.poetry.scripts]
source-datadog = "source_datadog.run:run"

[tool.poetry.group.dev.dependencies]
requests-mock = "^1.9.3"
pytest = "^6.2"
pytest-mock = "^3.6.1"
requests-mock = "*"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

pytest = "*"
pytest-mock = "*"
Loading
Loading