Sourse OneSignal: Authentication of OneSignal API changed to not allow User Auth key for notification requests

[andnig]

Environment

• Airbyte version: all
• OS Version / Instance: all
• Deployment: all
• Source Connector and version: OneSignal 0.1.2
• Step where error happened: Sync

Current Behavior

The OneSignal 0.1.2 connector uses the User Auth Key (which is sort of a "global" key) to authenticate the API. This worked fine until 2023-02-21. However, OneSignal changed their authentication schema to only allow app-specific REST API Keys to authenticate. This is one key per OneSignal app - meaning we can't use this "global" key to authenticate ALL operations anymore.

This also means, we can't simply configure one key and sync ALL app notifications. We need to configure one key per app.

This change in authentication was confirmed by OneSignal support.

Expected Behavior

We'd need to have configuration options to set the app we want to sync as well as the respective rest api key. As there are also API operations / airbyte streams which still require the user auth key (like listing all apps), I propose having the following configuration options:

• user auth key
• app to sync
• rest api key

Then we'd need to change the behavior of the connector itself.

• For app-specific streams: Instead of iterating through all apps and syncing all apps, we'd only need to sync the configured app and use the rest-api-key
• For "global" streams like listing apps, we keep the behavior as is

Logs

I'll not add the whole logs here, as the issue was confirmed by OneSignal support and the logs simply tell us that we make a bad request to OneSignl.

The main portion of the logs is:

2023-02-27 12:02:18 source > Encountered an exception while reading stream SourceOnesignal
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 108, in read
    internal_config=internal_config,
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 141, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 213, in _read_full_refresh
    for record in records:
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py", line 352, in read_records
    response = self._send_request(request, request_kwargs)
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py", line 319, in _send_request
    return backoff_handler(user_backoff_handler)(request, request_kwargs)
  File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py", line 285, in _send
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://onesignal.com/api/v1/notifications?app_id=2c1d5775-8fd6-4c1c-bdad-7bfc9cf9fd64&limit=50

Steps to Reproduce

1. Use the OneSignal source connector
2. Attempt to sync the notifications stream

Are you willing to submit a PR?

I'm happy to submit a PR - but I'd need the following question answered:

• As I need to dive deep into the OneSignal connector and remodel most of the more complex app-specific streams would it make sense to create a new connector using the lowcode CDK? This would most probably take less time and make it more robust to maintain in the future.

[artem1205]

Fixed in #24076