Source Datadog: Update CDK version and dependencies, remove parameters and migrate to inline schemas #44371

btkcodedev · 2024-08-18T17:39:59Z

What

Closes #41965

How

…mas and spec

vercel · 2024-08-18T17:40:03Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Aug 20, 2024 3:00pm

natikgadzhi

This is great work, but it's broken ;) I think the fix is very easy.

Again, strong signal for me that we need to make the test reports public.

Here's the failure that CI shows at connector build step:

process "python /airbyte/integration_code/main.py spec" did not complete successfully: exit code: 1
Stdout:
{"type":"LOG","log":{"level":"FATAL","message":"Object of type datetime is not JSON serializable\nTraceback (most recent call last):\n  File \"/airbyte/integration_code/main.py\", line 8, in <module>\n    run()\n  File \"/airbyte/integration_code/source_datadog/run.py\", line 14, in run\n    launch(source, sys.argv[1:])\n  File \"/usr/local/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 234, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/usr/local/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 97, in run\n    source_spec: ConnectorSpecification = self.source.spec(self.logger)\n  File \"/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py\", line 142, in spec\n    self._emit_manifest_debug_message(extra_args={\"source_name\": self.name, \"parsed_config\": json.dumps(self._source_config)})\n  File \"/usr/local/lib/python3.10/json/__init__.py\", line 231, in dumps\n    return _default_encoder.encode(obj)\n  File \"/usr/local/lib/python3.10/json/encoder.py\", line 199, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n  File \"/usr/local/lib/python3.10/json/encoder.py\", line 257, in iterencode\n    return _iterencode(o, 0)\n  File \"/usr/local/lib/python3.10/json/encoder.py\", line 179, in default\n    raise TypeError(f'Object of type {o.__class__.__name__} '\nTypeError: Object of type datetime is not JSON serializable"}}
{"type":"TRACE","trace":{"type":"ERROR","emitted_at":1724003967406.657,"error":{"message":"Something went wrong in the connector. See the logs for more details.","internal_message":"Object of type datetime is not JSON serializable","stack_trace":"Traceback (most recent call last):\n  File \"/airbyte/integration_code/main.py\", line 8, in <module>\n    run()\n  File \"/airbyte/integration_code/source_datadog/run.py\", line 14, in run\n    launch(source, sys.argv[1:])\n  File \"/usr/local/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 234, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/usr/local/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 97, in run\n    source_spec: ConnectorSpecification = self.source.spec(self.logger)\n  File \"/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py\", line 142, in spec\n    self._emit_manifest_debug_message(extra_args={\"source_name\": self.name, \"parsed_config\": json.dumps(self._source_config)})\n  File \"/usr/local/lib/python3.10/json/__init__.py\", line 231, in dumps\n    return _default_encoder.encode(obj)\n  File \"/usr/local/lib/python3.10/json/encoder.py\", line 199, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n  File \"/usr/local/lib/python3.10/json/encoder.py\", line 257, in iterencode\n    return _iterencode(o, 0)\n  File \"/usr/local/lib/python3.10/json/encoder.py\", line 179, in default\n    raise TypeError(f'Object of type {o.__class__.__name__} '\nTypeError: Object of type datetime is not JSON serializable\n","failure_type":"system_error","stream_descriptor":null}}}
Stderr:

natikgadzhi · 2024-08-18T21:29:33Z

airbyte-integrations/connectors/source-datadog/pyproject.toml

-requests-mock = "^1.9.3"
-pytest = "^6.2"
-pytest-mock = "^3.6.1"
+requests-mock = "*"


natikgadzhi · 2024-08-18T21:29:40Z

airbyte-integrations/connectors/source-datadog/pyproject.toml

-python = "^3.9,<3.12"
-airbyte-cdk = "^1"
+python = "^3.10,<3.12"
+airbyte-cdk = "^4"


Feeling adventurous I see

natikgadzhi · 2024-08-18T21:30:16Z

airbyte-integrations/connectors/source-datadog/acceptance-test-config.yml

@@ -5,6 +5,8 @@ acceptance_tests:
  spec:
    tests:
      - spec_path: "source_datadog/spec.yaml"


You removed the spec file, so the spec path here should lead to the manifest, I think. /cc @ChristoGrab

natikgadzhi · 2024-08-18T21:31:46Z

airbyte-integrations/connectors/source-datadog/acceptance-test-config.yml

@@ -5,6 +5,8 @@ acceptance_tests:
  spec:
    tests:
      - spec_path: "source_datadog/spec.yaml"
+        backward_compatibility_tests_config:
+          disable_for_version: "0.4.15" # Set default start and end date for incremental sync


In theory, you only want to disable the backwards compatibility tests if the new version will not work, but the old one worked.

If the new version provides defaults and works in a scenario when a new one does not, why would it be broken?

Also, usually if you're skipping backwards compat checks, this should signal that the change is breaking, which this is not.

BUT, your build step in CI is failing on something something date can't be parsed something, so seems relevant.

natikgadzhi · 2024-08-18T21:33:05Z

/cc @jnr0790

btkcodedev · 2024-08-19T02:42:09Z

Oops, I've now converted the default start and end date to string, it should work now.

btkcodedev · 2024-08-19T11:43:42Z

@natikgadzhi The breaking change is -> Added default start date and end date to the spec
all other tests are passing in ci, I guess
Please continue review 🙇‍♂️

natikgadzhi · 2024-08-19T21:49:08Z

airbyte-integrations/connectors/source-datadog/source_datadog/manifest.yaml

+        pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$
+        examples:
+          - "2022-10-01T00:00:00Z"
+        default: "2024-12-01T00:00:00Z"


Hmmm, that would mean that the start date by default is in the future.

What if instead of setting the default, we make this field optional, and use a srtring interpolation to default to 1 year ago, but dynamically?

Similar to this:

airbyte/airbyte-integrations/connectors/source-linkedin-pages/manifest.yaml

Line 649 in 526bdee

datetime:

@girarda what's a better practice?

Changed my mind: let's perhaps default to a specific start date, but in the past — smaller change. 2024-01-01 maybe?

natikgadzhi

Left some comments on start date default value, and suggestions on breaking change messaging.

Leaving as open so @btkcodedev you can catch up with these and ask questions / apply / figure out start date default values.

docs/integrations/sources/datadog-migrations.md

natikgadzhi · 2024-08-19T21:53:01Z

airbyte-integrations/connectors/source-datadog/source_datadog/manifest.yaml

+        pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$
+        examples:
+          - "2022-10-01T00:00:00Z"
+        default: "2024-12-01T00:00:00Z"


Changed my mind: let's perhaps default to a specific start date, but in the past — smaller change. 2024-01-01 maybe?

Co-authored-by: Natik Gadzhi <natik@respawn.io>

btkcodedev · 2024-08-20T15:00:37Z

Done

natikgadzhi · 2024-08-20T22:32:40Z

👏🏼

natikgadzhi · 2024-08-20T22:33:09Z

@jnr0790 this is merged! It's a breaking change, so once it lands, follow up with the customer and tell them they'd need to opt-in.

btkcodedev added 2 commits August 18, 2024 23:03

updat cdk & yaml version, dependency versions, migrate to inline sche…

3ee0af5

…mas and spec

add error handler

ad30408

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/datadog labels Aug 18, 2024

update poetry, format files, update docs

8a34a6b

btkcodedev self-assigned this Aug 18, 2024

btkcodedev added 2 commits August 18, 2024 23:13

update docs

dae8502

fix pagination

7304233

natikgadzhi reviewed Aug 18, 2024

View reviewed changes

Resolve date bug, make breaking change

94a4dd1

octavia-squidington-iv requested a review from a team August 19, 2024 02:07

vercel bot deployed to Preview August 19, 2024 02:11 View deployment

update docs

a743965

vercel bot deployed to Preview August 19, 2024 02:18 View deployment

update docs, update empty_streams

fb94e1a

vercel bot deployed to Preview August 19, 2024 02:45 View deployment

btkcodedev requested a review from natikgadzhi August 19, 2024 04:13

natikgadzhi reviewed Aug 19, 2024

View reviewed changes

btkcodedev and others added 3 commits August 20, 2024 19:45

Update docs/integrations/sources/datadog-migrations.md

7b42d8a

Co-authored-by: Natik Gadzhi <natik@respawn.io>

fix default start date and tests

2d3056f

revert tests

3950a95

btkcodedev requested a review from natikgadzhi August 20, 2024 15:00

natikgadzhi approved these changes Aug 20, 2024

View reviewed changes

natikgadzhi merged commit f0b9c5b into master Aug 20, 2024
36 of 37 checks passed

natikgadzhi deleted the btkcodedev/datadogApiFix branch August 20, 2024 22:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Datadog: Update CDK version and dependencies, remove parameters and migrate to inline schemas #44371

Source Datadog: Update CDK version and dependencies, remove parameters and migrate to inline schemas #44371

btkcodedev commented Aug 18, 2024 •

edited

Loading

vercel bot commented Aug 18, 2024 •

edited

Loading

natikgadzhi left a comment

natikgadzhi Aug 18, 2024

natikgadzhi Aug 18, 2024

natikgadzhi Aug 18, 2024

natikgadzhi Aug 18, 2024

natikgadzhi commented Aug 18, 2024

btkcodedev commented Aug 19, 2024

btkcodedev commented Aug 19, 2024

natikgadzhi Aug 19, 2024

natikgadzhi Aug 19, 2024

natikgadzhi left a comment

natikgadzhi Aug 19, 2024

btkcodedev commented Aug 20, 2024

natikgadzhi commented Aug 20, 2024

natikgadzhi commented Aug 20, 2024

Source Datadog: Update CDK version and dependencies, remove parameters and migrate to inline schemas #44371

Source Datadog: Update CDK version and dependencies, remove parameters and migrate to inline schemas #44371

Conversation

btkcodedev commented Aug 18, 2024 • edited Loading

What

How

vercel bot commented Aug 18, 2024 • edited Loading

natikgadzhi left a comment

Choose a reason for hiding this comment

natikgadzhi Aug 18, 2024

Choose a reason for hiding this comment

natikgadzhi Aug 18, 2024

Choose a reason for hiding this comment

natikgadzhi Aug 18, 2024

Choose a reason for hiding this comment

natikgadzhi Aug 18, 2024

Choose a reason for hiding this comment

natikgadzhi commented Aug 18, 2024

btkcodedev commented Aug 19, 2024

btkcodedev commented Aug 19, 2024

natikgadzhi Aug 19, 2024

Choose a reason for hiding this comment

natikgadzhi Aug 19, 2024

Choose a reason for hiding this comment

natikgadzhi left a comment

Choose a reason for hiding this comment

natikgadzhi Aug 19, 2024

Choose a reason for hiding this comment

btkcodedev commented Aug 20, 2024

natikgadzhi commented Aug 20, 2024

natikgadzhi commented Aug 20, 2024

btkcodedev commented Aug 18, 2024 •

edited

Loading

vercel bot commented Aug 18, 2024 •

edited

Loading