Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎉 Source Snapchat Marketing - enable default availability strategy #22808

Merged
merged 19 commits into from
Feb 24, 2023
Merged

🎉 Source Snapchat Marketing - enable default availability strategy #22808

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
9437a85
Enable default availability trategy
midavadim Feb 10, 2023
a560200
added PR number
midavadim Feb 10, 2023
0f9f8b8
Merge branch 'master' into midavadim/21787-snapchat-marketing-availab…
midavadim Feb 11, 2023
13c2a9a
remove custom error handling
erohmensing Feb 13, 2023
b7c3045
Merge branch 'master' into midavadim/21787-snapchat-marketing-availab…
lazebnyi Feb 15, 2023
ff992ac
Add allowedHosts
lazebnyi Feb 15, 2023
73a62f7
Merge master to branch
lazebnyi Feb 15, 2023
02d585e
Merge branch 'master' into midavadim/21787-snapchat-marketing-availab…
lazebnyi Feb 15, 2023
9db1eae
Merge branch 'master' into midavadim/21787-snapchat-marketing-availab…
lazebnyi Feb 16, 2023
9d3dfe0
Merge branch 'master' into midavadim/21787-snapchat-marketing-availab…
midavadim Feb 20, 2023
64e1fa0
Merge remote-tracking branch 'origin/midavadim/21787-snapchat-marketi…
midavadim Feb 20, 2023
31ab477
Merge branch 'master' into midavadim/21787-snapchat-marketing-availab…
lazebnyi Feb 22, 2023
82c2f11
Merge branch 'master' into midavadim/21787-snapchat-marketing-availab…
lazebnyi Feb 22, 2023
92963f8
reverted customized should_retry
midavadim Feb 23, 2023
e3ce8b4
Merge remote-tracking branch 'origin/midavadim/21787-snapchat-marketi…
midavadim Feb 23, 2023
9eeda6a
Merge branch 'master' into midavadim/21787-snapchat-marketing-availab…
midavadim Feb 23, 2023
22e35ff
updated version in source_definitions.yaml
midavadim Feb 23, 2023
5f08d2c
Merge branch 'master' into midavadim/21787-snapchat-marketing-availab…
midavadim Feb 23, 2023
f6f1431
auto-bump connector version
octavia-squidington-iii Feb 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1690,6 +1690,10 @@
icon: snapchat.svg
sourceType: api
releaseStage: generally_available
allowedHosts:
hosts:
- accounts.snapchat.com
- adsapi.snapchat.com
- name: Snowflake
sourceDefinitionId: e2d65910-8c8b-40a1-ae7d-ee2416b2bfa2
dockerRepository: airbyte/source-snowflake
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-snapchat-marketing/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,5 @@ COPY source_snapchat_marketing ./source_snapchat_marketing
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.13
LABEL io.airbyte.version=0.1.14
LABEL io.airbyte.name=airbyte/source-snapchat-marketing
12 changes: 0 additions & 12 deletions ...yte-integrations/connectors/source-snapchat-marketing/source_snapchat_marketing/source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources import AbstractSource
from airbyte_cdk.sources.streams import Stream
from airbyte_cdk.sources.streams.availability_strategy import AvailabilityStrategy
from airbyte_cdk.sources.streams.core import IncrementalMixin, package_name_from_class
from airbyte_cdk.sources.streams.http import HttpStream
from airbyte_cdk.sources.streams.http.auth import Oauth2Authenticator
Expand Down Expand Up @@ -184,10 +183,6 @@ def response_root_name(self):
"""Using the class name in lower to set the root node for response parsing"""
return self.name

@property
def availability_strategy(self) -> Optional["AvailabilityStrategy"]:
return None

@property
def response_item_name(self):
"""Remove last 's' from response_root_name, see example in parse_response function"""
Expand Down Expand Up @@ -219,13 +214,6 @@ def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapp
raise SnapchatMarketingException(error_text)
yield resp.get(self.response_item_name)

def should_retry(self, response: requests.Response) -> bool:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I investigated the reasons for this custom error handling. There was a stream which was failing for one specifc slice, becuase of permissions error. that's why it handles 403 error. So in order to not fail the whole sync, we just skip one particular slice.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know why it was failing for one specific slice? That would indeed not be covered by the availability strategy, but I'm curious as to why it happened

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erohmensing

related PR: #20865
related issue: https://github.com/airbytehq/oncall/issues/1067

The problem happened with 'segments' stream for one of 'adaccount' in its slice.

As far as I understand snapchat credentials could have access to a couple of snapchat organization/adaccounts and for one of such accounts there was no permission access to 'segments' data.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will leave this up to you as per my comment in the PR itself!

if response.status_code == 403:
setattr(self, "raise_on_http_errors", False)
self.logger.warning(f"Got permission error when accessing URL {response.request.url}. " f"Skipping {self.name} stream.")
return False
return super().should_retry(response)


class IncrementalSnapchatMarketingStream(SnapchatMarketingStream, ABC):
cursor_field = "updated_at"
Expand Down
31 changes: 16 additions & 15 deletions docs/integrations/sources/snapchat-marketing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,18 +111,19 @@ Snapchat Marketing API has limitations to 1000 items per page.

## Changelog

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:------------------------------------------------------|
| 0.1.13 | 2023-01-27 | [22023](https://github.com/airbytehq/airbyte/pull/22023) | Set `AvailabilityStrategy` for streams explicitly to `None` |
| 0.1.12 | 2023-01-11 | [21267](https://github.com/airbytehq/airbyte/pull/21267) | Fix parse empty error response |
| 0.1.11 | 2022-12-23 | [20865](https://github.com/airbytehq/airbyte/pull/20865) | Handle 403 permission error |
| 0.1.10 | 2022-12-15 | [20537](https://github.com/airbytehq/airbyte/pull/20537) | Run on CDK 0.15.0 |
| 0.1.9 | 2022-12-14 | [20498](https://github.com/airbytehq/airbyte/pull/20498) | Fix output state when no records are read |
| 0.1.8 | 2022-10-05 | [17596](https://github.com/airbytehq/airbyte/pull/17596) | Retry 429 and 5xx errors when refreshing access token |
| 0.1.6 | 2022-07-21 | [14924](https://github.com/airbytehq/airbyte/pull/14924) | Remove `additionalProperties` field from specs |
| 0.1.5 | 2022-07-13 | [14577](https://github.com/airbytehq/airbyte/pull/14577) | Added stats streams hourly, daily, lifetime |
| 0.1.4 | 2021-12-07 | [8429](https://github.com/airbytehq/airbyte/pull/8429) | Update titles and descriptions |
| 0.1.3 | 2021-11-10 | [7811](https://github.com/airbytehq/airbyte/pull/7811) | Add oauth2.0, fix stream_state |
| 0.1.2 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies |
| 0.1.1 | 2021-07-29 | [5072](https://github.com/airbytehq/airbyte/pull/5072) | Fix bug with incorrect stream\_state value |
| 0.1.0 | 2021-07-26 | [4843](https://github.com/airbytehq/airbyte/pull/4843) | Initial release supporting the Snapchat Marketing API |
| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:------------------------------------------------------------|
| 0.1.14 | 2023-02-10 | [22808](https://github.com/airbytehq/airbyte/pull/22808) | Enable default `AvailabilityStrategy` |
| 0.1.13 | 2023-01-27 | [22023](https://github.com/airbytehq/airbyte/pull/22023) | Set `AvailabilityStrategy` for streams explicitly to `None` |
| 0.1.12 | 2023-01-11 | [21267](https://github.com/airbytehq/airbyte/pull/21267) | Fix parse empty error response |
| 0.1.11 | 2022-12-23 | [20865](https://github.com/airbytehq/airbyte/pull/20865) | Handle 403 permission error |
| 0.1.10 | 2022-12-15 | [20537](https://github.com/airbytehq/airbyte/pull/20537) | Run on CDK 0.15.0 |
| 0.1.9 | 2022-12-14 | [20498](https://github.com/airbytehq/airbyte/pull/20498) | Fix output state when no records are read |
| 0.1.8 | 2022-10-05 | [17596](https://github.com/airbytehq/airbyte/pull/17596) | Retry 429 and 5xx errors when refreshing access token |
| 0.1.6 | 2022-07-21 | [14924](https://github.com/airbytehq/airbyte/pull/14924) | Remove `additionalProperties` field from specs |
| 0.1.5 | 2022-07-13 | [14577](https://github.com/airbytehq/airbyte/pull/14577) | Added stats streams hourly, daily, lifetime |
| 0.1.4 | 2021-12-07 | [8429](https://github.com/airbytehq/airbyte/pull/8429) | Update titles and descriptions |
| 0.1.3 | 2021-11-10 | [7811](https://github.com/airbytehq/airbyte/pull/7811) | Add oauth2.0, fix stream_state |
| 0.1.2 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies |
| 0.1.1 | 2021-07-29 | [5072](https://github.com/airbytehq/airbyte/pull/5072) | Fix bug with incorrect stream\_state value |
| 0.1.0 | 2021-07-26 | [4843](https://github.com/airbytehq/airbyte/pull/4843) | Initial release supporting the Snapchat Marketing API |