Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Salesforce: include primary key in every chunk #23190

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1598,7 +1598,7 @@
- name: Salesforce
sourceDefinitionId: b117307c-14b6-41aa-9422-947e34922962
dockerRepository: airbyte/source-salesforce
dockerImageTag: 2.0.2
dockerImageTag: 2.0.3
documentationUrl: https://docs.airbyte.com/integrations/sources/salesforce
icon: salesforce.svg
sourceType: api
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13109,7 +13109,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-salesforce:2.0.2"
- dockerImage: "airbyte/source-salesforce:2.0.3"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/salesforce"
connectionSpecification:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ RUN pip install .

ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=2.0.2
LABEL io.airbyte.version=2.0.3
LABEL io.airbyte.name=airbyte/source-salesforce
Original file line number Diff line number Diff line change
Expand Up @@ -136,13 +136,16 @@ def request_params(
def chunk_properties(self) -> Iterable[Mapping[str, Any]]:
selected_properties = self.get_json_schema().get("properties", {})

def empty_props_with_pk_if_present():
return {self.primary_key: selected_properties[self.primary_key]} if self.primary_key else {}

summary_length = 0
local_properties = {}
local_properties = empty_props_with_pk_if_present()
for property_name, value in selected_properties.items():
current_property_length = len(urllib.parse.quote(f"{property_name},"))
if current_property_length + summary_length >= self.max_properties_length:
yield local_properties
local_properties = {}
local_properties = empty_props_with_pk_if_present()
summary_length = 0

local_properties[property_name] = value
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -627,7 +627,10 @@ def test_rest_stream_init_with_too_many_properties(stream_config, stream_api_v2_

def test_too_many_properties(stream_config, stream_api_v2_pk_too_many_properties, requests_mock):
stream = generate_stream("Account", stream_config, stream_api_v2_pk_too_many_properties)
chunks = len(list(stream.chunk_properties()))
chunks = list(stream.chunk_properties())
for chunk in chunks:
assert stream.primary_key in chunk
chunks_len = len(chunks)
assert stream.too_many_properties
assert stream.primary_key
assert type(stream) == RestSalesforceStream
Expand All @@ -648,7 +651,7 @@ def test_too_many_properties(stream_config, stream_api_v2_pk_too_many_properties
}
},
# 2 for 2 chunks above and 1 for a chunk below
*[{"json": {"records": [{"Id": 1}, {"Id": 2}], "nextRecordsUrl": next_page_url}} for _ in range(chunks - 3)],
*[{"json": {"records": [{"Id": 1}, {"Id": 2}], "nextRecordsUrl": next_page_url}} for _ in range(chunks_len - 3)],
{
"json": {
"records": [{"Id": 1}, {"Id": 2}]
Expand All @@ -665,7 +668,7 @@ def test_too_many_properties(stream_config, stream_api_v2_pk_too_many_properties
}
},
# 2 for 2 chunks above and 1 for a chunk below
*[{"json": {"records": [{"Id": 3}, {"Id": 4}]}} for _ in range(chunks - 3)],
*[{"json": {"records": [{"Id": 3}, {"Id": 4}]}} for _ in range(chunks_len - 3)],
{
"json": {
"records": [{"Id": 3}, {"Id": 4}]
Expand Down
2 changes: 1 addition & 1 deletion connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@
| **SFTP** | <img alt="SFTP icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/sftp.svg" height="30" height="30"/> | Source | airbyte/source-sftp:0.1.2 | alpha | [link](https://docs.airbyte.com/integrations/sources/sftp) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-sftp) | <small>`a827c52e-791c-4135-a245-e233c5255199`</small> |
| **SFTP Bulk** | <img alt="SFTP Bulk icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/sftp.svg" height="30" height="30"/> | Source | airbyte/source-sftp-bulk:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/sftp-bulk) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-sftp-bulk) | <small>`31e3242f-dee7-4cdc-a4b8-8e06c5458517`</small> |
| **SalesLoft** | <img alt="SalesLoft icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/salesloft.svg" height="30" height="30"/> | Source | airbyte/source-salesloft:0.1.3 | alpha | [link](https://docs.airbyte.com/integrations/sources/salesloft) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-salesloft) | <small>`41991d12-d4b5-439e-afd0-260a31d4c53f`</small> |
| **Salesforce** | <img alt="Salesforce icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/salesforce.svg" height="30" height="30"/> | Source | airbyte/source-salesforce:2.0.2 | generally_available | [link](https://docs.airbyte.com/integrations/sources/salesforce) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-salesforce) | <small>`b117307c-14b6-41aa-9422-947e34922962`</small> |
| **Salesforce** | <img alt="Salesforce icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/salesforce.svg" height="30" height="30"/> | Source | airbyte/source-salesforce:2.0.3 | generally_available | [link](https://docs.airbyte.com/integrations/sources/salesforce) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-salesforce) | <small>`b117307c-14b6-41aa-9422-947e34922962`</small> |
| **Sample Data (Faker)** | <img alt="Sample Data (Faker) icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/faker.svg" height="30" height="30"/> | Source | airbyte/source-faker:2.0.3 | beta | [link](https://docs.airbyte.com/integrations/sources/faker) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-faker) | <small>`dfd88b22-b603-4c3d-aad7-3701784586b1`</small> |
| **SearchMetrics** | <img alt="SearchMetrics icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/searchmetrics.svg" height="30" height="30"/> | Source | airbyte/source-search-metrics:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/search-metrics) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-search-metrics) | <small>`8d7ef552-2c0f-11ec-8d3d-0242ac130003`</small> |
| **Secoda** | <img alt="Secoda icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/secoda.svg" height="30" height="30"/> | Source | airbyte/source-secoda:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/secoda) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-secoda) | <small>`da9fc6b9-8059-4be0-b204-f56e22e4d52d`</small> |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/salesforce.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ Now that you have set up the Salesforce source connector, check out the followin

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------|
| 2.0.3 | 2023-02-17 | [23190](https://github.com/airbytehq/airbyte/pull/23190) | In case properties are chunked, fetch primary key in every chunk |
| 2.0.2 | 2023-02-13 | [22896](https://github.com/airbytehq/airbyte/pull/22896) | Count the URL length based on encoded params |
| 2.0.1 | 2023-02-08 | [22597](https://github.com/airbytehq/airbyte/pull/22597) | Make multiple requests if a REST stream has too many properties |
| 2.0.0 | 2023-02-02 | [22322](https://github.com/airbytehq/airbyte/pull/22322) | Remove `ActivityMetricRollup` stream |
Expand Down