[source-datadog] Connector attempts to query unnecessary endpoint

[et-dynamic]

Connector Name

source-datadog

Connector Version

0.4.8

What step the error happened?

During the sync

Relevant information

After this connector was automatically updated in Airbyte Cloud (see note on versions at the bottom), I started getting these errors around the beginning of my syncs (see log output).

The issue seems to be that the connector is attempting to query audit_logs, which I do not have access to. The source shouldn't be hitting this endpoint because I've configured the connector to only query logs. Specifically, I have attempted to use the "top-level" parameters in the UI (where you find Query, Start date, Site, End date, and Max records per requests) in addition to the Add button next to Queries (where you find Query name, Data Source, and Query). In both cases, the sync stops because it tries to access audit_logs where I only want logs.

In both cases, it seems like there is just no way to have the connector not hit audit_logs.

Note on versions: this error did not occur on 0.2.2 (for some reason when I set this connector up in the Airbyte Cloud UI a few weeks ago, it installed 0.2.2). My connector first updated around 0.4.8, but it could have been 0.4.7 or 0.4.6. I would have tested each version but I can't choose the version to use in the Airbyte Cloud UI so I'm stuck just giving you guesswork, sadly. However, since I basically leapfrogged from 0.2.2 to the high 0.4's, this issue could have been introduced any time in between those.

Relevant log output

2024-07-10 16:37:20 ERROR i.a.w.i.VersionedAirbyteStreamFactory(internalLog):304 - Encountered an error trying to connect to stream audit_logs. Error: 
 Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/checks/check_stream.py", line 42, in check_connection
    stream_is_available, reason = availability_strategy.check_availability(stream, logger, source)
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 50, in check_availability
    get_first_record_for_slice(stream, stream_slice)
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/streams/utils/stream_helper.py", line 40, in get_first_record_for_slice
    return next(records_for_slice)
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 128, in read_records
    yield from self.retriever.read_records(self.get_json_schema(), stream_slice)
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 370, in read_records
    yield from self._read_page(record_generator, stream_state, _slice)
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 317, in _read_page
    response = self._fetch_next_page(stream_state, stream_slice)
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 273, in _fetch_next_page
    return self.requester.send_request(
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 294, in send_request
    request, response = self._http_client.send_request(
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 331, in send_request
    response: requests.Response = self._send_with_retry(request=request, request_kwargs=request_kwargs, log_formatter=log_formatter)
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 209, in _send_with_retry
    response = backoff_handler(user_backoff_handler)(request, request_kwargs, log_formatter=log_formatter)  # type: ignore # mypy can't infer that backoff_handler wraps _send
  File "/usr/local/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 260, in _send
    raise MessageRepresentationAirbyteTracedErrors(
airbyte_cdk.sources.streams.http.http_client.MessageRepresentationAirbyteTracedErrors: Forbidden. You don't have permission to access this resource.

Contribute

• Yes, I want to contribute

[marcosmarxm]

Thanks for reporting the issue @et-dynamic Datadog is a community connector and it isn’t in the current roadmap for improvements. If you want to contribute fixing the issue please reach me out in Slack so I can provide you instructions to make the contribution.

[natikgadzhi]

Hey! Not ignoring. Datadog seems to be pretty lightweight manifest connector, we'll try to bump it to the latest CDK version, then see what's going on. I hope that this was just a new stream added that got automatically enabled in your connection, and something like a connection reset would heal this.

[natikgadzhi]

@jnr0790 keep an eye on this. @btkcodedev or other devs will take a look here soon.

[btkcodedev]

I'll take a look at this

[btkcodedev]

Update: Got the free tier API keys