🎉 Source Salesforce: Add the ability to filter streams #8871

yevhenii-ldv · 2021-12-17T13:55:56Z

What

resolves #7031.

How

A function has been added to the specification that allows users to specify values for filtering streams, as well as specify the type of option for filtering.
It is Optional field

Pre-merge Checklist

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

yevhenii-ldv · 2021-12-17T14:01:13Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1592551166
❌ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1592551166
🐛 https://gradle.com/s/rpkcfatlt2j2k
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs1] - AssertionError:  Rec...
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs1]
=================== 2 failed, 19 passed in 373.02s (0:06:13) ===================

yevhenii-ldv · 2021-12-17T15:02:58Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1592797471
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1592797471
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py       101     65    36%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  972    459    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                 Stmts   Miss  Cover
	 --------------------------------------------------------
	 source_salesforce/__init__.py            2      0   100%
	 source_salesforce/api.py               120     63    48%
	 source_salesforce/exceptions.py          1      0   100%
	 source_salesforce/rate_limiting.py      22      6    73%
	 source_salesforce/source.py             57     24    58%
	 source_salesforce/streams.py           214     38    82%
	 source_salesforce/utils.py               8      7    12%
	 --------------------------------------------------------
	 TOTAL                                  424    138    67%

yevhenii-ldv · 2021-12-20T11:55:47Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1601884862
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1601884862
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py       101     65    36%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  972    459    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                 Stmts   Miss  Cover
	 --------------------------------------------------------
	 source_salesforce/__init__.py            2      0   100%
	 source_salesforce/api.py               115     43    63%
	 source_salesforce/exceptions.py          1      0   100%
	 source_salesforce/rate_limiting.py      22      6    73%
	 source_salesforce/source.py             57     24    58%
	 source_salesforce/streams.py           214     38    82%
	 source_salesforce/utils.py               8      0   100%
	 --------------------------------------------------------
	 TOTAL                                  419    111    74%

htrueman

LGTM. Have just one question for unit tests.

htrueman · 2021-12-22T14:12:02Z

airbyte-integrations/connectors/source-salesforce/unit_tests/unit_test.py

+    sf_object.instance_url = "https://fase-account.salesforce.com"
+    sf_object.describe = Mock(return_value={"sobjects": [{"name": f"FakeStream{i}"} for i in range(100)]})
+    filtered_streams = sf_object.get_validated_streams(config=updated_config)
+    assert len(filtered_streams) == number_of_filtered_streams


perhaps we may check the steams by name instead of just count them?

That is, you propose to use the same method to filter the names of streams, and in the end compare them with each other, it is strange?
Could you tell us in more detail how you see this implementation?

I just suggest to compare filtered_stream == expected_filtered_streams.
E.g. ['stream1', 'stream2'] == ['stream1']
Is it possible?

yevhenii-ldv · 2021-12-23T11:22:32Z

/publish connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1615342856
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1615342856

yevhenii-ldv · 2021-12-23T11:40:09Z

/publish connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1615397742
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1615397742

Source Salesforce: Add the ability to filter streams

404797e

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Dec 17, 2021

jrhizor temporarily deployed to more-secrets December 17, 2021 14:03 Inactive

update acceptance-test-config.yml

b3db366

yevhenii-ldv temporarily deployed to more-secrets December 17, 2021 14:55 Inactive

jrhizor temporarily deployed to more-secrets December 17, 2021 15:04 Inactive

add unit_test for new functionality

ec7eecb

yevhenii-ldv temporarily deployed to more-secrets December 20, 2021 11:57 Inactive

jrhizor temporarily deployed to more-secrets December 20, 2021 11:57 Inactive

yevhenii-ldv requested review from Zirochkaa, htrueman and sergei-solonitcyn December 20, 2021 12:10

VasylLazebnyk mentioned this pull request Dec 20, 2021

Salesforce: investigate selecting schemas/streams before the discovery step #7031

Closed

htrueman reviewed Dec 22, 2021

View reviewed changes

yevhenii-ldv added 2 commits December 23, 2021 12:50

update tests

620488a

merge with master

1e2093f

yevhenii-ldv temporarily deployed to more-secrets December 23, 2021 10:54 Inactive

htrueman approved these changes Dec 23, 2021

View reviewed changes

Zirochkaa approved these changes Dec 23, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets December 23, 2021 11:24 Inactive

Source Salesforce: fix examples for new field in specification

2bfea2f

yevhenii-ldv temporarily deployed to more-secrets December 23, 2021 11:41 Inactive

jrhizor temporarily deployed to more-secrets December 23, 2021 11:42 Inactive

bump version

2acab0d

yevhenii-ldv merged commit ea578e8 into master Dec 23, 2021

yevhenii-ldv deleted the ykurochkin/add-filter-params-for-schemas branch December 23, 2021 11:54

yevhenii-ldv temporarily deployed to more-secrets December 23, 2021 11:55 Inactive

jrhizor mentioned this pull request Dec 27, 2021

Bump Airbyte version from 0.35.1-alpha to 0.35.2-alpha #9119

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Salesforce: Add the ability to filter streams #8871

🎉 Source Salesforce: Add the ability to filter streams #8871

yevhenii-ldv commented Dec 17, 2021 •

edited

Loading

yevhenii-ldv commented Dec 17, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Dec 17, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Dec 20, 2021 •

edited by github-actions bot

Loading

htrueman left a comment

htrueman Dec 22, 2021

yevhenii-ldv Dec 22, 2021

htrueman Dec 23, 2021

yevhenii-ldv commented Dec 23, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Dec 23, 2021 •

edited by github-actions bot

Loading

🎉 Source Salesforce: Add the ability to filter streams #8871

🎉 Source Salesforce: Add the ability to filter streams #8871

Conversation

yevhenii-ldv commented Dec 17, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

yevhenii-ldv commented Dec 17, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Dec 17, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Dec 20, 2021 • edited by github-actions bot Loading

htrueman left a comment

Choose a reason for hiding this comment

htrueman Dec 22, 2021

Choose a reason for hiding this comment

yevhenii-ldv Dec 22, 2021

Choose a reason for hiding this comment

htrueman Dec 23, 2021

Choose a reason for hiding this comment

yevhenii-ldv commented Dec 23, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Dec 23, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Dec 17, 2021 •

edited

Loading

yevhenii-ldv commented Dec 17, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Dec 17, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Dec 20, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Dec 23, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Dec 23, 2021 •

edited by github-actions bot

Loading