🐛 Source Salesforce: Fix getting `anyType` fields using BULK API #7592

yevhenii-ldv · 2021-11-03T13:04:46Z

What

resolves #6886.

How

Describe the solution

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

yevhenii-ldv · 2021-11-03T13:05:08Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1416834253
❌ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1416834253
🐛 https://gradle.com/s/xhkikcbk2tp2u

yevhenii-ldv · 2021-11-03T13:22:17Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1416907540
❌ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1416907540
🐛 https://gradle.com/s/4bmzjjt4fzwqo

yevhenii-ldv · 2021-11-03T14:15:19Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1417128751
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1417128751
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        75      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       33     22    33%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  891    435    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                 Stmts   Miss  Cover
	 --------------------------------------------------------
	 source_salesforce/__init__.py            2      2     0%
	 source_salesforce/api.py               114    114     0%
	 source_salesforce/exceptions.py          1      1     0%
	 source_salesforce/rate_limiting.py      22     22     0%
	 source_salesforce/source.py             52     52     0%
	 source_salesforce/streams.py           201    201     0%
	 source_salesforce/api.py               114    114     0%/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-salesforce/.venv/lib/python3.8/site-packages/coverage/control.py:761: CoverageWarning: No data was collected. (no-data-collected)
	 --------------------------------------------------------
	   self._warn("No data was collected.", slug="no-data-collected")
	 TOTAL                                  392    392     0%

yevhenii-ldv · 2021-11-03T17:40:25Z

/publish connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1417954009
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1417954009

sherifnada

let's close the PR after testing the new version with the user reporting the issue

sherifnada · 2021-11-03T16:26:02Z

airbyte-integrations/connectors/source-salesforce/BOOTSTRAP.md

@@ -10,6 +10,19 @@ There are two types of objects:
 To query an object, one must use [SOQL](https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/dome_query.htm), Salesforce’s proprietary SQL language. 
 An example might be `SELECT * FROM <sobject.name> WHERE SystemModstamp > 2122-01-18T21:18:20.000Z`.

+Because the `Salesforce` connector pulls all objects from `Salesforce` dynamically, then all streams are dynamically generated accordingly. 


sherifnada · 2021-11-03T16:28:01Z

airbyte-integrations/connectors/source-salesforce/source_salesforce/spec.json

@@ -28,7 +28,7 @@
        "airbyte_secret": true
      },
      "start_date": {
-        "description": "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.",
+        "description": "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. Priority for filtering by `updated` fields, and only then by `created` fields if they are available for stream.",


Suggested change

"description": "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. Priority for filtering by `updated` fields, and only then by `created` fields if they are available for stream.",

"description": "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. This field uses the "updated" field if available, otherwise the "created" fields if they are available for a stream.",

yevhenii-ldv · 2021-11-05T10:09:57Z

@sherifnada Could you provide information from the user if it appears, since I can't see the message for the ticket author in the github?
If everything goes well, I will update the specification for the specification and close the issue.

sherifnada · 2021-11-05T16:21:38Z

@yevhenii-ldv the fix worked great! you're good to move forward :)

…bytehq#7592) * Source Salesforce: Fix getting anyType fields using BULK API

yevhenii-ldv added 2 commits November 3, 2021 14:48

Source Salesforce: Fix getting anyType fields using BULK API

87e74a9

update Changelogs, Bootstrap.md and spec.json

4c2711b

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Nov 3, 2021

yevhenii-ldv temporarily deployed to more-secrets November 3, 2021 13:05 Inactive

jrhizor temporarily deployed to more-secrets November 3, 2021 13:07 Inactive

remove dynamic stream from tests

0c8ba01

yevhenii-ldv requested review from avida and htrueman November 3, 2021 13:22

yevhenii-ldv temporarily deployed to more-secrets November 3, 2021 13:23 Inactive

jrhizor temporarily deployed to more-secrets November 3, 2021 13:24 Inactive

remove dynamic stream from tests 2

3178a54

yevhenii-ldv temporarily deployed to more-secrets November 3, 2021 14:16 Inactive

jrhizor temporarily deployed to more-secrets November 3, 2021 14:17 Inactive

jrhizor temporarily deployed to more-secrets November 3, 2021 17:42 Inactive

htrueman approved these changes Nov 3, 2021

View reviewed changes

sherifnada reviewed Nov 4, 2021

View reviewed changes

yevhenii-ldv mentioned this pull request Nov 5, 2021

Source Salesforce: NewValue and OldValue columns are not getting pulled for OpportunityFieldHistory #6886

Closed

bump version. update changelogs and specs

027aa5e

yevhenii-ldv merged commit 2efef3c into master Nov 5, 2021

yevhenii-ldv deleted the ykurochkin/salesforce-fix-bug-oldvalue-newvalue-nullable branch November 5, 2021 22:42

yevhenii-ldv temporarily deployed to more-secrets November 5, 2021 22:43 Inactive

jrhizor mentioned this pull request Nov 6, 2021

Bump Airbyte version from 0.30.31-alpha to 0.30.32-alpha #7700

Merged

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

🐛 Source Salesforce: Fix getting anyType fields using BULK API (air…

a9d84ac

…bytehq#7592) * Source Salesforce: Fix getting anyType fields using BULK API

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Salesforce: Fix getting `anyType` fields using BULK API #7592

🐛 Source Salesforce: Fix getting `anyType` fields using BULK API #7592

yevhenii-ldv commented Nov 3, 2021

yevhenii-ldv commented Nov 3, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Nov 3, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Nov 3, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Nov 3, 2021 •

edited by github-actions bot

Loading

sherifnada left a comment

sherifnada Nov 3, 2021

sherifnada Nov 3, 2021

yevhenii-ldv commented Nov 5, 2021

sherifnada commented Nov 5, 2021

	"description": "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. Priority for filtering by `updated` fields, and only then by `created` fields if they are available for stream.",
	"description": "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. This field uses the "updated" field if available, otherwise the "created" fields if they are available for a stream.",

🐛 Source Salesforce: Fix getting anyType fields using BULK API #7592

🐛 Source Salesforce: Fix getting anyType fields using BULK API #7592

Conversation

yevhenii-ldv commented Nov 3, 2021

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

yevhenii-ldv commented Nov 3, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Nov 3, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Nov 3, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Nov 3, 2021 • edited by github-actions bot Loading

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Nov 3, 2021

Choose a reason for hiding this comment

sherifnada Nov 3, 2021

Choose a reason for hiding this comment

yevhenii-ldv commented Nov 5, 2021

sherifnada commented Nov 5, 2021

🐛 Source Salesforce: Fix getting `anyType` fields using BULK API #7592

🐛 Source Salesforce: Fix getting `anyType` fields using BULK API #7592

yevhenii-ldv commented Nov 3, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Nov 3, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Nov 3, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Nov 3, 2021 •

edited by github-actions bot

Loading