Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Salesforce: include primary key in every chunk #23190

Conversation

davydov-d
Copy link
Collaborator

What

#20703
this is a followup for the on-call issue https://github.com/airbytehq/oncall/issues/1403

How

If the properties are chunked - we need to include the primary key in every chunk, otherwise we won't be able to stick response chunks

@octavia-squidington-iii octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/salesforce labels Feb 17, 2023
@davydov-d
Copy link
Collaborator Author

davydov-d commented Feb 17, 2023

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/4202648226
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/4202648226
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             96      6    94%
source_salesforce/streams.py           382     33    91%
source_salesforce/api.py               155     14    91%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
--------------------------------------------------------
TOTAL                                  673     57    92%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               155     21    86%
source_salesforce/streams.py           382     86    77%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             96     33    66%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  673    154    77%
	 Name                                                    Stmts   Miss  Cover   Missing
	 -------------------------------------------------------------------------------------
	 connector_acceptance_test/base.py                          12      4    67%   16-19
	 connector_acceptance_test/config.py                       142      5    96%   87, 93, 242, 246-247
	 connector_acceptance_test/conftest.py                     220    102    54%   37, 43-45, 50, 55, 60, 83, 89, 95-97, 116, 121-123, 129-131, 137-138, 143-144, 149, 160, 169-178, 184-189, 204, 228, 259, 265, 273-281, 289-302, 310-323, 328-334, 341-352, 359-375
	 connector_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 connector_acceptance_test/tests/test_core.py              476    117    75%   53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
	 connector_acceptance_test/tests/test_incremental.py       162     14    91%   58-65, 70-83, 252
	 connector_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 connector_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 connector_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 connector_acceptance_test/utils/connector_runner.py       134     33    75%   30-33, 53-54, 57-61, 64-65, 80-82, 85-87, 90-92, 95-97, 100-102, 132-133, 167-169, 216
	 connector_acceptance_test/utils/json_schema_helper.py     114     13    89%   31-32, 39, 42, 66-69, 97, 121, 203-205
	 -------------------------------------------------------------------------------------
	 TOTAL                                                    1716    348    80%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
================== 36 passed, 2 skipped in 528.28s (0:08:48) ===================

Copy link
Contributor

@maxi297 maxi297 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@davydov-d
Copy link
Collaborator Author

davydov-d commented Feb 23, 2023

/publish connector=connectors/source-salesforce

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@davydov-d
Copy link
Collaborator Author

davydov-d commented Feb 23, 2023

/publish connector=connectors/source-salesforce

🕑 Publishing the following connectors:
connectors/source-salesforce
https://github.com/airbytehq/airbyte/actions/runs/4250335050


Connector Did it publish? Were definitions generated?
connectors/source-salesforce

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@davydov-d davydov-d merged commit ca16502 into master Feb 23, 2023
@davydov-d davydov-d deleted the ddavydov/#20703-source-salesforce-include-pk-in-properties-chunks branch February 23, 2023 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/salesforce
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants