Source Salesforce: include primary key in every chunk (#23190)

* #20703 source salesforce: include primary key in every chunk

* #20703 source salesforce: upd changelog

* #20703 source salesforce: review fix

* auto-bump connector version

* #20703 source salesforce: upd connectors.md

---------

Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

@@ -1598,7 +1598,7 @@
 - name: Salesforce
   sourceDefinitionId: b117307c-14b6-41aa-9422-947e34922962
   dockerRepository: airbyte/source-salesforce
-  dockerImageTag: 2.0.2
+  dockerImageTag: 2.0.3
   documentationUrl: https://docs.airbyte.com/integrations/sources/salesforce
   icon: salesforce.svg
   sourceType: api

airbyte-config/init/src/main/resources/seed/source_specs.yaml

@@ -13109,7 +13109,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-salesforce:2.0.2"
+- dockerImage: "airbyte/source-salesforce:2.0.3"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/sources/salesforce"
     connectionSpecification:

airbyte-integrations/connectors/source-salesforce/Dockerfile

@@ -13,5 +13,5 @@ RUN pip install .
 
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=2.0.2
+LABEL io.airbyte.version=2.0.3
 LABEL io.airbyte.name=airbyte/source-salesforce

airbyte-integrations/connectors/source-salesforce/source_salesforce/streams.py

@@ -136,13 +136,16 @@ def request_params(
     def chunk_properties(self) -> Iterable[Mapping[str, Any]]:
         selected_properties = self.get_json_schema().get("properties", {})
 
+        def empty_props_with_pk_if_present():
+            return {self.primary_key: selected_properties[self.primary_key]} if self.primary_key else {}
+
         summary_length = 0
-        local_properties = {}
+        local_properties = empty_props_with_pk_if_present()
         for property_name, value in selected_properties.items():
             current_property_length = len(urllib.parse.quote(f"{property_name},"))
             if current_property_length + summary_length >= self.max_properties_length:
                 yield local_properties
-                local_properties = {}
+                local_properties = empty_props_with_pk_if_present()
                 summary_length = 0
 
             local_properties[property_name] = value

airbyte-integrations/connectors/source-salesforce/unit_tests/api_test.py

@@ -627,7 +627,10 @@ def test_rest_stream_init_with_too_many_properties(stream_config, stream_api_v2_
 
 def test_too_many_properties(stream_config, stream_api_v2_pk_too_many_properties, requests_mock):
     stream = generate_stream("Account", stream_config, stream_api_v2_pk_too_many_properties)
-    chunks = len(list(stream.chunk_properties()))
+    chunks = list(stream.chunk_properties())
+    for chunk in chunks:
+        assert stream.primary_key in chunk
+    chunks_len = len(chunks)
     assert stream.too_many_properties
     assert stream.primary_key
     assert type(stream) == RestSalesforceStream
@@ -648,7 +651,7 @@ def test_too_many_properties(stream_config, stream_api_v2_pk_too_many_properties
                 }
             },
             # 2 for 2 chunks above and 1 for a chunk below
-            *[{"json": {"records": [{"Id": 1}, {"Id": 2}], "nextRecordsUrl": next_page_url}} for _ in range(chunks - 3)],
+            *[{"json": {"records": [{"Id": 1}, {"Id": 2}], "nextRecordsUrl": next_page_url}} for _ in range(chunks_len - 3)],
             {
                 "json": {
                     "records": [{"Id": 1}, {"Id": 2}]
@@ -665,7 +668,7 @@ def test_too_many_properties(stream_config, stream_api_v2_pk_too_many_properties
                 }
             },
             # 2 for 2 chunks above and 1 for a chunk below
-            *[{"json": {"records": [{"Id": 3}, {"Id": 4}]}} for _ in range(chunks - 3)],
+            *[{"json": {"records": [{"Id": 3}, {"Id": 4}]}} for _ in range(chunks_len - 3)],
             {
                 "json": {
                     "records": [{"Id": 3}, {"Id": 4}]

connectors.md

@@ -200,7 +200,7 @@
 | **SFTP** | <img alt="SFTP icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/sftp.svg" height="30" height="30"/> | Source | airbyte/source-sftp:0.1.2 | alpha | [link](https://docs.airbyte.com/integrations/sources/sftp) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-sftp) | <small>`a827c52e-791c-4135-a245-e233c5255199`</small> |
 | **SFTP Bulk** | <img alt="SFTP Bulk icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/sftp.svg" height="30" height="30"/> | Source | airbyte/source-sftp-bulk:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/sftp-bulk) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-sftp-bulk) | <small>`31e3242f-dee7-4cdc-a4b8-8e06c5458517`</small> |
 | **SalesLoft** | <img alt="SalesLoft icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/salesloft.svg" height="30" height="30"/> | Source | airbyte/source-salesloft:0.1.3 | alpha | [link](https://docs.airbyte.com/integrations/sources/salesloft) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-salesloft) | <small>`41991d12-d4b5-439e-afd0-260a31d4c53f`</small> |
-| **Salesforce** | <img alt="Salesforce icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/salesforce.svg" height="30" height="30"/> | Source | airbyte/source-salesforce:2.0.2 | generally_available | [link](https://docs.airbyte.com/integrations/sources/salesforce) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-salesforce) | <small>`b117307c-14b6-41aa-9422-947e34922962`</small> |
+| **Salesforce** | <img alt="Salesforce icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/salesforce.svg" height="30" height="30"/> | Source | airbyte/source-salesforce:2.0.3 | generally_available | [link](https://docs.airbyte.com/integrations/sources/salesforce) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-salesforce) | <small>`b117307c-14b6-41aa-9422-947e34922962`</small> |
 | **Sample Data (Faker)** | <img alt="Sample Data (Faker) icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/faker.svg" height="30" height="30"/> | Source | airbyte/source-faker:2.0.3 | beta | [link](https://docs.airbyte.com/integrations/sources/faker) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-faker) | <small>`dfd88b22-b603-4c3d-aad7-3701784586b1`</small> |
 | **SearchMetrics** | <img alt="SearchMetrics icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/searchmetrics.svg" height="30" height="30"/> | Source | airbyte/source-search-metrics:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/search-metrics) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-search-metrics) | <small>`8d7ef552-2c0f-11ec-8d3d-0242ac130003`</small> |
 | **Secoda** | <img alt="Secoda icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/secoda.svg" height="30" height="30"/> | Source | airbyte/source-secoda:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/secoda) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-secoda) | <small>`da9fc6b9-8059-4be0-b204-f56e22e4d52d`</small> |

docs/integrations/sources/salesforce.md

@@ -129,6 +129,7 @@ Now that you have set up the Salesforce source connector, check out the followin
 
 | Version | Date       | Pull Request                                             | Subject                                                                                                                          |
 |:--------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------|
+| 2.0.3   | 2023-02-17 | [23190](https://github.com/airbytehq/airbyte/pull/23190) | In case properties are chunked, fetch primary key in every chunk                                                                 |
 | 2.0.2   | 2023-02-13 | [22896](https://github.com/airbytehq/airbyte/pull/22896) | Count the URL length based on encoded params                                                                                     |
 | 2.0.1   | 2023-02-08 | [22597](https://github.com/airbytehq/airbyte/pull/22597) | Make multiple requests if a REST stream has too many properties                                                                  |
 | 2.0.0   | 2023-02-02 | [22322](https://github.com/airbytehq/airbyte/pull/22322) | Remove `ActivityMetricRollup` stream                                                                                             |