Skip to content

Commit

Permalink
Source: Amazon Ads - Add pk validator for reports when expected pk is…
Browse files Browse the repository at this point in the history
… not returned (airbytehq#25146)

* Add pk validator for reports when expected pk is not returned

* Update PR number

* Fix formating

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
  • Loading branch information
2 people authored and btkcodedev committed Apr 26, 2023
1 parent c4e3c2f commit d21d131
Show file tree
Hide file tree
Showing 8 changed files with 34 additions and 6 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -8481,7 +8481,7 @@
"sourceDefinitionId": "c6b0a29e-1da9-4512-9002-7bfd0cba2246",
"name": "Amazon Ads",
"dockerRepository": "airbyte/source-amazon-ads",
"dockerImageTag": "1.0.2",
"dockerImageTag": "1.0.3",
"documentationUrl": "https://docs.airbyte.com/integrations/sources/amazon-ads",
"icon": "amazonads.svg",
"sourceType": "api",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
- name: Amazon Ads
sourceDefinitionId: c6b0a29e-1da9-4512-9002-7bfd0cba2246
dockerRepository: airbyte/source-amazon-ads
dockerImageTag: 1.0.2
dockerImageTag: 1.0.3
documentationUrl: https://docs.airbyte.com/integrations/sources/amazon-ads
icon: amazonads.svg
sourceType: api
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -822,7 +822,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-amazon-ads:1.0.2"
- dockerImage: "airbyte/source-amazon-ads:1.0.3"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/amazon-ads"
connectionSpecification:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]



LABEL io.airbyte.version=1.0.2
LABEL io.airbyte.version=1.0.3
LABEL io.airbyte.name=airbyte/source-amazon-ads
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

import json
import re
import uuid
from abc import ABC, abstractmethod
from copy import deepcopy
from dataclasses import dataclass
Expand Down Expand Up @@ -165,7 +166,7 @@ def read_records(
profileId=report_info.profile_id,
recordType=report_info.record_type,
reportDate=report_date,
recordId=metric_object[self.metrics_type_to_id_map[report_info.record_type]],
recordId=metric_object.get(self.metrics_type_to_id_map[report_info.record_type], str(uuid.uuid4())),
metric=metric_object,
).dict()

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,15 @@
)
METRICS_COUNT = 5

METRIC_RESPONSE_WITHOUT_ASIN_PK = b64decode(
"""
H4sIAAAAAAAAClXMvQrCMBQF4L1PETIbyM1v4+giLr6AOAQTStH+ECsi4r
t7aZOKwx3u+TjnVBHyxiOEXnw3+rbpD4FuSe205IYrvvnHo+8iMr3jf4us
xKzHnK0lmls+7NPwGOdFapTjINXPcmy0FByELjD51MRpQesAUSooeI2v55
DCrp1ZAwipnF1HMy9lZ7StJRdAET/V+Qv1VRgd7AAAAA==
"""
)


def setup_responses(init_response=None, init_response_products=None, init_response_brands=None, status_response=None, metric_response=None):
if init_response:
Expand Down Expand Up @@ -206,6 +215,23 @@ def test_products_report_stream(config):
assert len(metrics) == METRICS_COUNT * len(stream.metrics_map)


@responses.activate
def test_products_report_stream_without_pk(config):
setup_responses(
init_response_products=REPORT_INIT_RESPONSE,
status_response=REPORT_STATUS_RESPONSE,
metric_response=METRIC_RESPONSE_WITHOUT_ASIN_PK,
)

profiles = make_profiles(profile_type="vendor")

stream = SponsoredProductsReportStream(config, profiles, authenticator=mock.MagicMock())
stream_slice = {"profile": profiles[0], "reportDate": "2021-07-25", "retry_count": 3}
metrics = [m for m in stream.read_records(SyncMode.incremental, stream_slice=stream_slice)]

assert len(metrics) == len(stream.metrics_map)


@responses.activate
def test_brands_report_stream(config):
setup_responses(
Expand Down
2 changes: 1 addition & 1 deletion connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
| **Airtable** | <img alt="Airtable icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config-oss/init-oss/src/main/resources/icons/airtable.svg" height="30" height="30"/> | Source | airbyte/source-airtable:3.0.0 | generally_available | [link](https://docs.airbyte.com/integrations/sources/airtable) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-airtable) | <small>`14c6e7ea-97ed-4f5e-a7b5-25e9a80b8212`</small> |
| **AlloyDB for PostgreSQL** | <img alt="AlloyDB for PostgreSQL icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config-oss/init-oss/src/main/resources/icons/alloydb.svg" height="30" height="30"/> | Source | airbyte/source-alloydb:2.0.19 | generally_available | [link](https://docs.airbyte.com/integrations/sources/alloydb) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-alloydb) | <small>`1fa90628-2b9e-11ed-a261-0242ac120002`</small> |
| **Alpha Vantage** | <img alt="Alpha Vantage icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config-oss/init-oss/src/main/resources/icons/alpha-vantage.svg" height="30" height="30"/> | Source | airbyte/source-alpha-vantage:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/alpha-vantage) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-alpha-vantage) | <small>`db385323-9333-4fec-bec3-9e0ca9326c90`</small> |
| **Amazon Ads** | <img alt="Amazon Ads icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config-oss/init-oss/src/main/resources/icons/amazonads.svg" height="30" height="30"/> | Source | airbyte/source-amazon-ads:1.0.2 | generally_available | [link](https://docs.airbyte.com/integrations/sources/amazon-ads) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-amazon-ads) | <small>`c6b0a29e-1da9-4512-9002-7bfd0cba2246`</small> |
| **Amazon Ads** | <img alt="Amazon Ads icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config-oss/init-oss/src/main/resources/icons/amazonads.svg" height="30" height="30"/> | Source | airbyte/source-amazon-ads:1.0.3 | generally_available | [link](https://docs.airbyte.com/integrations/sources/amazon-ads) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-amazon-ads) | <small>`c6b0a29e-1da9-4512-9002-7bfd0cba2246`</small> |
| **Amazon SQS** | <img alt="Amazon SQS icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config-oss/init-oss/src/main/resources/icons/awssqs.svg" height="30" height="30"/> | Source | airbyte/source-amazon-sqs:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/amazon-sqs) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-amazon-sqs) | <small>`983fd355-6bf3-4709-91b5-37afa391eeb6`</small> |
| **Amazon Seller Partner** | <img alt="Amazon Seller Partner icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config-oss/init-oss/src/main/resources/icons/amazonsellerpartner.svg" height="30" height="30"/> | Source | airbyte/source-amazon-seller-partner:1.0.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/amazon-seller-partner) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-amazon-seller-partner) | <small>`e55879a8-0ef8-4557-abcf-ab34c53ec460`</small> |
| **Amplitude** | <img alt="Amplitude icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config-oss/init-oss/src/main/resources/icons/amplitude.svg" height="30" height="30"/> | Source | airbyte/source-amplitude:0.2.0 | generally_available | [link](https://docs.airbyte.com/integrations/sources/amplitude) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-amplitude) | <small>`fa9f58c6-2d03-4237-aaa4-07d75e0c1396`</small> |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/amazon-ads.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ Information about expected report generation waiting time you may find [here](ht

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------|
| 1.0.3 | 2023-04-13 | [25146](https://github.com/airbytehq/airbyte/pull/25146) | Validate pk for reports when expected pk is not returned |
| 1.0.2 | 2023-02-03 | [22355](https://github.com/airbytehq/airbyte/pull/22355) | Migrate `products_report` stream to API v3 |
| 1.0.1 | 2022-11-01 | [18677](https://github.com/airbytehq/airbyte/pull/18677) | Add optional config report_record_types |
| 1.0.0 | 2023-01-30 | [21677](https://github.com/airbytehq/airbyte/pull/21677) | Fix bug with non-unique primary keys in report streams. Add asins_keywords and asins_targets |
Expand Down

0 comments on commit d21d131

Please sign in to comment.