Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source: Facebook Marketing - New default for action_breakdowns, improve "check" speed #19803

Merged
merged 11 commits into from
Nov 28, 2022
Merged

Source: Facebook Marketing - New default for action_breakdowns, improve "check" speed #19803

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a1e77e3
bump 0.2.74
grubberr Nov 25, 2022
0c737aa
check_breakdowns added
grubberr Nov 25, 2022
0e99dba
facebook-marketing.md updated
grubberr Nov 25, 2022
c93b49e
code format
grubberr Nov 25, 2022
5939971
spec.json updated
grubberr Nov 25, 2022
ab9c03c
Merge branch 'master' into grubberr/oncall-1083-source-facebook-marke…
grubberr Nov 25, 2022
a9481b5
lint code
grubberr Nov 26, 2022
cff526b
airbyte-cdk~=0.2 -> airbyte-cdk~=0.9
grubberr Nov 26, 2022
eca4e09
check_breakdowns updated
grubberr Nov 26, 2022
2ef8833
Merge branch 'master' into grubberr/oncall-1083-source-facebook-marke…
grubberr Nov 28, 2022
fd876c4
auto-bump connector version
octavia-squidington-iii Nov 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,7 +437,7 @@
- name: Facebook Marketing
sourceDefinitionId: e7778cfc-e97c-4458-9ecb-b4f2bba8946c
dockerRepository: airbyte/source-facebook-marketing
dockerImageTag: 0.2.73
dockerImageTag: 0.2.74
documentationUrl: https://docs.airbyte.com/integrations/sources/facebook-marketing
icon: facebook.svg
sourceType: api
Expand Down
7 changes: 5 additions & 2 deletions airbyte-config/init/src/main/resources/seed/source_specs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3396,7 +3396,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-facebook-marketing:0.2.73"
- dockerImage: "airbyte/source-facebook-marketing:0.2.74"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/facebook-marketing"
changelogUrl: "https://docs.airbyte.com/integrations/sources/facebook-marketing"
Expand Down Expand Up @@ -3651,7 +3651,10 @@
action_breakdowns:
title: "Action Breakdowns"
description: "A list of chosen action_breakdowns for action_breakdowns"
default: []
default:
- "action_type"
- "action_target_id"
- "action_destination"
type: "array"
items:
title: "ValidActionBreakdowns"
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-facebook-marketing/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]


LABEL io.airbyte.version=0.2.73
LABEL io.airbyte.version=0.2.74
LABEL io.airbyte.name=airbyte/source-facebook-marketing
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-facebook-marketing/integration_tests/spec.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@
"action_breakdowns": {
"title": "Action Breakdowns",
"description": "A list of chosen action_breakdowns for action_breakdowns",
"default": [],
"default": ["action_type", "action_target_id", "action_destination"],
"type": "array",
"items": {
"title": "ValidActionBreakdowns",
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-facebook-marketing/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from setuptools import find_packages, setup

MAIN_REQUIREMENTS = [
"airbyte-cdk~=0.2",
"airbyte-cdk~=0.9",
"cached_property==1.5.2",
"facebook_business==15.0.0",
"pendulum>=2,<3",
Expand Down
7 changes: 3 additions & 4 deletions ...yte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
Videos,
)

from .utils import read_full_refresh, validate_end_date, validate_start_date
from .utils import validate_end_date, validate_start_date

logger = logging.getLogger("airbyte")

Expand Down Expand Up @@ -63,11 +63,10 @@ def check_connection(self, logger: logging.Logger, config: Mapping[str, Any]) ->
except requests.exceptions.RequestException as e:
return False, e

# read one record from all custom insights streams
# to ensure that we have valid combination of "action_breakdowns" and "breakdowns" parameters
# make sure that we have valid combination of "action_breakdowns" and "breakdowns" parameters
for stream in self.get_custom_insights_streams(api, config):
try:
next(read_full_refresh(stream), None)
stream.check_breakdowns()
except facebook_business.exceptions.FacebookRequestError as e:
return False, e._api_error_message
return True, None
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/spec.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ class Config:
action_breakdowns: Optional[List[ValidActionBreakdowns]] = Field(
title="Action Breakdowns",
description="A list of chosen action_breakdowns for action_breakdowns",
default=[],
default=["action_type", "action_target_id", "action_destination"],
)

time_increment: Optional[PositiveInt] = Field(
Expand Down
19 changes: 14 additions & 5 deletions ...ctors/source-facebook-marketing/source_facebook_marketing/streams/base_insight_streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,10 @@ class AdsInsights(FBMarketingIncrementalStream):
# https://developers.facebook.com/docs/marketing-api/reference/ad-account/insights/#overview
INSIGHTS_RETENTION_PERIOD = pendulum.duration(months=37)

action_breakdowns = ALL_ACTION_BREAKDOWNS
level = "ad"
action_attribution_windows = ALL_ACTION_ATTRIBUTION_WINDOWS
time_increment = 1

breakdowns = []

def __init__(
self,
name: str = None,
Expand All @@ -70,8 +67,8 @@ def __init__(
self._start_date = self._start_date.date()
self._end_date = self._end_date.date()
self._fields = fields
self.action_breakdowns = action_breakdowns or self.action_breakdowns
self.breakdowns = breakdowns or self.breakdowns
self.action_breakdowns = action_breakdowns or self.ALL_ACTION_BREAKDOWNS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this also be cleared? (set to [])

This comment seems to imply that it's still an issue https://github.com/airbytehq/oncall/issues/1083#issuecomment-1329630450

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pedroslopez

The main problem here is backward compatibility.
I cannot change action_breakdowns = []
because it will break about 20 existing syncs on the prod cloud.

We need to find a good way to distinguish old created sync from new ones for such cases.

self.breakdowns = breakdowns or []
self.time_increment = time_increment or self.time_increment
self._new_class_name = name
self._insights_lookback_window = insights_lookback_window
Expand Down Expand Up @@ -198,6 +195,18 @@ def _generate_async_jobs(self, params: Mapping) -> Iterator[AsyncJob]:
interval = pendulum.Period(ts_start, ts_end)
yield InsightAsyncJob(api=self._api.api, edge_object=self._api.account, interval=interval, params=params)

def check_breakdowns(self):
"""
Making call to check "action_breakdowns" and "breakdowns" combinations
https://developers.facebook.com/docs/marketing-api/insights/breakdowns#combiningbreakdowns
"""
params = {
"action_breakdowns": self.action_breakdowns,
"breakdowns": self.breakdowns,
"fields": ["account_id"],
}
self._api.account.get_insights(params=params, is_async=False)

def stream_slices(
self, sync_mode: SyncMode, cursor_field: List[str] = None, stream_state: Mapping[str, Any] = None
) -> Iterable[Optional[Mapping[str, Any]]]:
Expand Down
10 changes: 0 additions & 10 deletions airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@
import logging

import pendulum
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources.streams import Stream
from pendulum import DateTime

logger = logging.getLogger("airbyte")
Expand Down Expand Up @@ -42,11 +40,3 @@ def validate_end_date(start_date: DateTime, end_date: DateTime) -> DateTime:
logger.warning(message)
return start_date
return end_date


def read_full_refresh(stream_instance: Stream):
slices = stream_instance.stream_slices(sync_mode=SyncMode.full_refresh)
for _slice in slices:
records = stream_instance.read_records(stream_slice=_slice, sync_mode=SyncMode.full_refresh)
for record in records:
yield record
1 change: 1 addition & 0 deletions docs/integrations/sources/facebook-marketing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ Please be informed that the connector uses the `lookback_window` parameter to pe

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0.2.74 | 2022-11-25 | [19803](https://github.com/airbytehq/airbyte/pull/19803) | New default for `action_breakdowns`, improve "check" command speed |
| 0.2.73 | 2022-11-21 | [19645](https://github.com/airbytehq/airbyte/pull/19645) | Check "breakdowns" combinations |
| 0.2.72 | 2022-11-04 | [18971](https://github.com/airbytehq/airbyte/pull/18971) | handle FacebookBadObjectError for empty results on async jobs |
| 0.2.71 | 2022-10-31 | [18734](https://github.com/airbytehq/airbyte/pull/18734) | Reduce request record limit on retry |
Expand Down