🐛 Source Microsoft Dataverse: Fixed deduped not working correctly (#2…

…2805) * Fixed dedupeot working correctly Due to delete only containing id, the deduped dbt could not work properly, so we also fill the cursor field with the current date so it does have a properly end * bump connector version * auto-bump connector version --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
airbytehq · Mar 22, 2023 · 1aaa221 · 1aaa221
1 parent 28a5f8e
commit 1aaa221
Show file tree

Hide file tree

Showing 6 changed files with 8 additions and 5 deletions.
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -1199,7 +1199,7 @@
 - name: Microsoft Dataverse
   sourceDefinitionId: 9220e3de-3b60-4bb2-a46f-046d59ea235a
   dockerRepository: airbyte/source-microsoft-dataverse
-  dockerImageTag: 0.1.0
+  dockerImageTag: 0.1.1
   documentationUrl: https://docs.airbyte.com/integrations/sources/microsoft-dataverse
   icon: microsoftdataverse.svg
   sourceType: api

diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml
@@ -8835,7 +8835,7 @@
               path_in_connector_config:
               - "credentials"
               - "client_secret"
-- dockerImage: "airbyte/source-microsoft-dataverse:0.1.0"
+- dockerImage: "airbyte/source-microsoft-dataverse:0.1.1"
   spec:
     documentationUrl: "https://docs.airbyte.io/integrations/sources/microsoft-dataverse"
     connectionSpecification:

diff --git a/airbyte-integrations/connectors/source-microsoft-dataverse/Dockerfile b/airbyte-integrations/connectors/source-microsoft-dataverse/Dockerfile
@@ -34,5 +34,5 @@ COPY source_microsoft_dataverse ./source_microsoft_dataverse
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.0
+LABEL io.airbyte.version=0.1.1
 LABEL io.airbyte.name=airbyte/source-microsoft-dataverse
diff --git a/...-integrations/connectors/source-microsoft-dataverse/source_microsoft_dataverse/streams.py b/...-integrations/connectors/source-microsoft-dataverse/source_microsoft_dataverse/streams.py
@@ -143,7 +143,9 @@ def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapp
                 result.pop("@odata.context", None)
                 result.pop("id", None)
                 result.pop("reason", None)
-                result.update({"_ab_cdc_deleted_at": datetime.now().isoformat()})
+                now = datetime.now().isoformat()
+                result.update({self.cursor_field[0]: now})
+                result.update({"_ab_cdc_deleted_at": now})
             else:
                 result.update({"_ab_cdc_updated_at": result[self.cursor_field[0]]})
 

diff --git a/connectors.md b/connectors.md
@@ -136,7 +136,7 @@
 | **Mailjet SMS** | <img alt="Mailjet SMS icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/mailjetsms.svg" height="30" height="30"/> | Source | airbyte/source-mailjet-sms:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/mailjet-sms) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-mailjet-sms) | <small>`6ec2acea-7fd1-4378-b403-41a666e0c028`</small> |
 | **Marketo** | <img alt="Marketo icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/marketo.svg" height="30" height="30"/> | Source | airbyte/source-marketo:1.0.2 | generally_available | [link](https://docs.airbyte.com/integrations/sources/marketo) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-marketo) | <small>`9e0556f4-69df-4522-a3fb-03264d36b348`</small> |
 | **Metabase** | <img alt="Metabase icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/metabase.svg" height="30" height="30"/> | Source | airbyte/source-metabase:0.3.1 | beta | [link](https://docs.airbyte.com/integrations/sources/metabase) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-metabase) | <small>`c7cb421b-942e-4468-99ee-e369bcabaec5`</small> |
-| **Microsoft Dataverse** | <img alt="Microsoft Dataverse icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/microsoftdataverse.svg" height="30" height="30"/> | Source | airbyte/source-microsoft-dataverse:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/microsoft-dataverse) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-microsoft-dataverse) | <small>`9220e3de-3b60-4bb2-a46f-046d59ea235a`</small> |
+| **Microsoft Dataverse** | <img alt="Microsoft Dataverse icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/microsoftdataverse.svg" height="30" height="30"/> | Source | airbyte/source-microsoft-dataverse:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/microsoft-dataverse) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-microsoft-dataverse) | <small>`9220e3de-3b60-4bb2-a46f-046d59ea235a`</small> |
 | **Microsoft SQL Server (MSSQL)** | <img alt="Microsoft SQL Server (MSSQL) icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/mssql.svg" height="30" height="30"/> | Source | airbyte/source-mssql:1.0.3 | alpha | [link](https://docs.airbyte.com/integrations/sources/mssql) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-mssql) | <small>`b5ea17b1-f170-46dc-bc31-cc744ca984c1`</small> |
 | **Microsoft teams** | <img alt="Microsoft teams icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/microsoft-teams.svg" height="30" height="30"/> | Source | airbyte/source-microsoft-teams:0.2.5 | alpha | [link](https://docs.airbyte.com/integrations/sources/microsoft-teams) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-microsoft-teams) | <small>`eaf50f04-21dd-4620-913b-2a83f5635227`</small> |
 | **Mixpanel** | <img alt="Mixpanel icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/mixpanel.svg" height="30" height="30"/> | Source | airbyte/source-mixpanel:0.1.30 | generally_available | [link](https://docs.airbyte.com/integrations/sources/mixpanel) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-mixpanel) | <small>`12928b32-bf0a-4f1e-964f-07e12e37153a`</small> |

diff --git a/docs/integrations/sources/microsoft-dataverse.md b/docs/integrations/sources/microsoft-dataverse.md
@@ -61,4 +61,5 @@ https://blog.magnetismsolutions.com/blog/paulnieuwelaar/2021/9/21/setting-up-an-
 
 | Version | Date       | Pull Request                                             | Subject                                         |
 | :------ | :--------- | :------------------------------------------------------- | :---------------------------------------------- |
+| 0.1.1   | 2023-03-16 | [22805](https://github.com/airbytehq/airbyte/pull/22805) | Fixed deduped cursor field value update |
 | 0.1.0   | 2022-11-14 | [18646](https://github.com/airbytehq/airbyte/pull/18646) | 🎉 New Source: Microsoft Dataverse [python cdk] |