Source Outreach - Feature more data from more streams #17385

FransDel · 2022-09-29T12:45:28Z

What

We want the following new streams
- mailboxes
- accounts
- maillings
- opportunities
- personas
- stages
- calls
In addition to the attributes that are already pulled in the tables, we need the relationship data between tables to be present as well.

How

Added parsing of relationship data in source.py
Added New streams with their schemas
Bonus: Added page size variable so user can decide what it should be instead of defaulting + sort by updatedAt for performance.

🚨 User Impact 🚨

New relationship fields in tables.

Bonus: Possibility to change page size (improved stability or faster). The value defaults to 100 so nothing more then the user being able to change it impacts the user.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Integration

Acceptance

Put your acceptance tests output here.

Tables added: * mailboxes * accounts * maillings * opportunities * personas * stages

CLAassistant · 2022-09-29T12:45:34Z

All committers have signed the CLA.

FransDel · 2022-09-29T12:46:32Z

Related task #15873

sajarin · 2022-09-29T14:05:58Z

Thanks for the PR @FransDel! This PR is part of Airbyte's Community Maintainer program and will be reviewed/merged by a member of the community. Someone will be assigned shortly, thanks for being patient!

tuanchris · 2022-10-25T12:13:29Z

Hi @FransDel, thank you for being patient. Can you upload screenshots of unit tests and integration test passing?

Can you also update the outreach.md file with a changelog?
Is there a reason for increasing the page size to 1000?

FransDel · 2022-10-27T12:06:25Z

Hi @FransDel, thank you for being patient. Can you upload screenshots of unit tests and integration test passing?

Can you also update the outreach.md file with a changelog?

Is there a reason for increasing the page size to 1000?

Hi @tuanchris thank you for reviewing this!

Updated the integration test and unit test sections with the screen shots.
Updated outreach.md with changelog.
For the 1000 page size. Indeed yes there was. It seemed to be faster to pull data with pages of 1000 elements. After a few weeks of testing this though, sometimes it isn't optimal we were getting some 500 errors from outreach api on one endpoint (more data there then the rest). After conversing with them they suggested to decrease page size to 500 or 100 and to sort by the updatedAt field. I've included these optimizations: page size is now variable depending on user's configuration (defaults to 100 so no breaking changes) and api call sorts by updatedAt (present in all endpoints).

tuanchris · 2022-10-27T12:28:29Z

Hi @FransDel, thank you for being patient. Can you upload screenshots of unit tests and integration test passing?

Can you also update the outreach.md file with a changelog?

Is there a reason for increasing the page size to 1000?

Hi @tuanchris thank you for reviewing this!

Updated the integration test and unit test sections with the screen shots.

Updated outreach.md with changelog.

For the 1000 page size. Indeed yes there was. It seemed to be faster to pull data with pages of 1000 elements. After a few weeks of testing this though, sometimes it isn't optimal we were getting some 500 errors from outreach api on one endpoint (more data there then the rest). After conversing with them they suggested to decrease page size to 500 or 100 and to sort by the updatedAt field. I've included these optimizations: page size is now variable depending on user's configuration (defaults to 100 so no breaking changes) and api call sorts by updatedAt (present in all endpoints).

Thanks @FransDel. Unit tests are looking good, but I don't see the results for integration tests. Can you upload another screenshot with the results (or raw log is also fine).

FransDel · 2022-10-28T13:22:10Z

Hi @FransDel, thank you for being patient. Can you upload screenshots of unit tests and integration test passing?

Can you also update the outreach.md file with a changelog?

Is there a reason for increasing the page size to 1000?

Hi @tuanchris thank you for reviewing this!

Updated the integration test and unit test sections with the screen shots.

Updated outreach.md with changelog.

For the 1000 page size. Indeed yes there was. It seemed to be faster to pull data with pages of 1000 elements. After a few weeks of testing this though, sometimes it isn't optimal we were getting some 500 errors from outreach api on one endpoint (more data there then the rest). After conversing with them they suggested to decrease page size to 500 or 100 and to sort by the updatedAt field. I've included these optimizations: page size is now variable depending on user's configuration (defaults to 100 so no breaking changes) and api call sorts by updatedAt (present in all endpoints).

Thanks @FransDel. Unit tests are looking good, but I don't see the results for integration tests. Can you upload another screenshot with the results (or raw log is also fine).

Hey @tuanchris. There isn't really much to show more actually here is a bigger image but thats all I get running python -m pytest integration_tests -p integration_tests:

tuanchris · 2022-10-30T13:49:38Z

Hi @FransDel, thank you for being patient. Can you upload screenshots of unit tests and integration test passing?

Can you also update the outreach.md file with a changelog?

Is there a reason for increasing the page size to 1000?

Hi @tuanchris thank you for reviewing this!

Updated the integration test and unit test sections with the screen shots.

Updated outreach.md with changelog.

For the 1000 page size. Indeed yes there was. It seemed to be faster to pull data with pages of 1000 elements. After a few weeks of testing this though, sometimes it isn't optimal we were getting some 500 errors from outreach api on one endpoint (more data there then the rest). After conversing with them they suggested to decrease page size to 500 or 100 and to sort by the updatedAt field. I've included these optimizations: page size is now variable depending on user's configuration (defaults to 100 so no breaking changes) and api call sorts by updatedAt (present in all endpoints).

Thanks @FransDel. Unit tests are looking good, but I don't see the results for integration tests. Can you upload another screenshot with the results (or raw log is also fine).

Hey @tuanchris. There isn't really much to show more actually here is a bigger image but thats all I get running python -m pytest integration_tests -p integration_tests:

I was able to run an integration test locally and confirm that everything is working. PR approved.

@sajarin please proceed with merging.

sajarin · 2022-10-31T14:20:14Z

/test connector=connectors/source-outreach

🕑 connectors/source-outreach https://github.com/airbytehq/airbyte/actions/runs/3362002993
❌ connectors/source-outreach https://github.com/airbytehq/airbyte/actions/runs/3362002993
🐛 https://gradle.com/s/todhoy7fpm7ji

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_backward_compatibility.py - pydantic.error_wrappers.Va...
	 FAILED unit_tests/test_core.py - pydantic.error_wrappers.ValidationError: 1 v...
	 !!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
	 
	 Results (1.94s):
	 /actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/.venv/lib/python3.9/site-packages/coverage/control.py:788: CoverageWarning: No data was collected. (no-data-collected)
	   self._warn("No data was collected.", slug="no-data-collected")

> Task :airbyte-integrations:connectors:source-outreach:installLocalReqs
	 Collecting wrapt<2,>=1.10
	   Using cached wrapt-1.14.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (77 kB)
	 Collecting pyrepl>=0.8.2
	   Using cached pyrepl-0.9.0-py3-none-any.whl
	 Collecting MarkupSafe>=2.0
	   Using cached MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
	 Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.venv/lib/python3.9/site-packages (from packaging->pytest~=6.1->source-acceptance-test==0.0.0->-r requirements.txt (line 1)) (3.0.9)
	 Collecting charset-normalizer<3,>=2
	   Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
	 Collecting certifi>=2017.4.17
	   Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
	 Collecting urllib3<1.27,>=1.21.1
	   Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
	 Collecting idna<4,>=2.5
	   Using cached idna-3.4-py3-none-any.whl (61 kB)
	 Collecting pytzdata>=2020.1
	   Using cached pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
	 Collecting yarl
	   Using cached yarl-1.8.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB)
	 Collecting multidict>=4.0
	   Using cached multidict-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)
	 Installing collected packages: six, pyrsistent, multidict, idna, yarl, wrapt, urllib3, sortedcontainers, PyYAML, pytzdata, python-dateutil, pyrepl, MarkupSafe, jsonschema, exceptiongroup, charset-normalizer, certifi, wmctrl, websocket-client, vcrpy, termcolor, requests, pygments, pydantic, pendulum, ordered-set, jsonref, Jinja2, hypothesis, fancycompleter, dpath, Deprecated, dataclasses-jsonschema, backoff, requests-mock, pytest-timeout, pytest-sugar, pytest-mock, pytest-cov, pprintpp, pdbpp, inflection, icdiff, hypothesis-jsonschema, docker, deepdiff, airbyte-cdk, source-outreach, source-acceptance-test
	   Running setup.py develop for source-outreach
	   Running setup.py develop for source-acceptance-test
	 Successfully installed Deprecated-1.2.13 Jinja2-3.1.2 MarkupSafe-2.1.1 PyYAML-5.4.1 airbyte-cdk-0.1.104 backoff-2.2.1 certifi-2022.9.24 charset-normalizer-2.1.1 dataclasses-jsonschema-2.15.1 deepdiff-5.8.1 docker-5.0.3 dpath-2.0.6 exceptiongroup-1.0.0 fancycompleter-0.9.1 hypothesis-6.54.6 hypothesis-jsonschema-0.20.1 icdiff-1.9.1 idna-3.4 inflection-0.5.1 jsonref-0.2 jsonschema-3.2.0 multidict-6.0.2 ordered-set-4.1.0 pdbpp-0.10.3 pendulum-2.1.2 pprintpp-0.4.0 pydantic-1.9.2 pygments-2.13.0 pyrepl-0.9.0 pyrsistent-0.19.1 pytest-cov-3.0.0 pytest-mock-3.6.1 pytest-sugar-0.9.5 pytest-timeout-1.4.2 python-dateutil-2.8.2 pytzdata-2020.1 requests-2.28.1 requests-mock-1.9.3 six-1.16.0 sortedcontainers-2.4.0 source-acceptance-test-0.0.0 source-outreach-0.0.0 termcolor-2.1.0 urllib3-1.26.12 vcrpy-4.2.1 websocket-client-1.4.1 wmctrl-0.4 wrapt-1.14.1 yarl-1.8.1
	 WARNING: You are using pip version 21.3.1; however, version 22.3 is available.
	 You should consider upgrading via the '/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-outreach/.venv/bin/python -m pip install --upgrade pip' command.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
38 actionable tasks: 25 executed, 13 up-to-date

Publishing build scan...
https://gradle.com/s/todhoy7fpm7ji

marcosmarxm · 2022-11-22T20:23:33Z

Hello 👋, first thank you for this amazing contribution.

We really appreciate the effort you've made to improve the project.
We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.

If you have any questions feel free to send me a message in Slack!
Thanks!

marcosmarxm · 2022-12-20T19:50:37Z

Hello 👋:skin-tone-2: and thank you for your contribution!

Airbyte has instituted a code freeze between 19 and 30 December, to make sure there are no disruptions during the holidays.
Because of this, reviewing and merging your contribution may take longer than usual.
We apologize for the delay, but we want everyone to have a quiet and happy holiday.

If you have any questions or need further clarification, please don't hesitate to ping via Slack.

marcosmarxm · 2023-03-21T20:07:12Z

/test connector=connectors/source-outreach

🕑 connectors/source-outreach https://github.com/airbytehq/airbyte/actions/runs/4483360192
✅ connectors/source-outreach https://github.com/airbytehq/airbyte/actions/runs/4483360192
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_outreach/__init__.py       2      0   100%
source_outreach/source.py       113     30    73%
-------------------------------------------------
TOTAL                           115     30    74%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
============ 32 passed, 2 skipped, 34 warnings in 66.59s (0:01:06) =============

marcosmarxm

Thanks @FransDel I'd just removed the page size parameter and use the max instead.

marcosmarxm · 2023-03-21T20:15:56Z

/publish connector=connectors/source-outreach

🕑 Publishing the following connectors:
connectors/source-outreach
https://github.com/airbytehq/airbyte/actions/runs/4483431536

Connector	Did it publish?	Were definitions generated?
connectors/source-outreach	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* [Feature] New Tables in Outreach connector Tables added: * mailboxes * accounts * maillings * opportunities * personas * stages * [Feature] Handle relationships in source code * [Feature] Handle relationships in schemas * [Feature] Use new outreach logo * [Fix] Bad class names didn't follow convention * [Fix] Errors in schemas and relationship data not being available * [Feature] Add Call Data * [Docs] Updated documentation with info on new streams * [Fix][Tests] Pass to 1000 records per page (max) + tests update * [Feature] Add page size variable * [Docs] Add change log * fix schemas * auto-bump connector version --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

FransDel added 10 commits August 23, 2022 10:20

[Feature] New Tables in Outreach connector

737d76d

Tables added: * mailboxes * accounts * maillings * opportunities * personas * stages

[Feature] Handle relationships in source code

1387293

[Feature] Handle relationships in schemas

7319075

[Feature] Use new outreach logo

c0bbd53

[Fix] Bad class names didn't follow convention

c06e5c5

[Fix] Errors in schemas and relationship data not being available

5572dc4

[Feature] Add Call Data

f4bb9bd

[Docs] Updated documentation with info on new streams

13216e9

[Fix][Tests] Pass to 1000 records per page (max) + tests update

a77bb29

Merge branch 'master' into feature-more-data-from-outreach-source

9d467cf

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 29, 2022

FransDel mentioned this pull request Sep 29, 2022

Source Outreach: Request for more streams #15873

Closed

octavia-squidington-iv added bounty community connectors/source/outreach labels Sep 29, 2022

sajarin added the bounty-XL Maintainer program: claimable extra large bounty PR label Sep 29, 2022

sajarin changed the title ~~Feature more data from outreach source~~ Source Outreach - Feature more data from more streams Sep 29, 2022

tuanchris self-assigned this Oct 22, 2022

FransDel added 2 commits October 27, 2022 13:54

[Feature] Add page size variable

defbfc0

[Docs] Add change log

e95fe02

tuanchris approved these changes Oct 30, 2022

View reviewed changes

bleonard added the reward-200 label Oct 31, 2022

marcosmarxm and others added 2 commits March 21, 2023 14:16

Merge branch 'master' into feature-more-data-from-outreach-source

f33e984

fix schemas

0f9e04f

marcosmarxm approved these changes Mar 21, 2023

View reviewed changes

auto-bump connector version

9e4b2cd

marcosmarxm merged commit 1644791 into airbytehq:master Mar 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Outreach - Feature more data from more streams #17385

Source Outreach - Feature more data from more streams #17385

FransDel commented Sep 29, 2022 •

edited

Loading

CLAassistant commented Sep 29, 2022 •

edited

Loading

FransDel commented Sep 29, 2022

sajarin commented Sep 29, 2022

tuanchris commented Oct 25, 2022 •

edited

Loading

FransDel commented Oct 27, 2022 •

edited

Loading

tuanchris commented Oct 27, 2022

FransDel commented Oct 28, 2022

tuanchris commented Oct 30, 2022

sajarin commented Oct 31, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 22, 2022

marcosmarxm commented Dec 20, 2022

marcosmarxm commented Mar 21, 2023 •

edited by github-actions bot

Loading

marcosmarxm left a comment

marcosmarxm commented Mar 21, 2023 •

edited by github-actions bot

Loading

Source Outreach - Feature more data from more streams #17385

Source Outreach - Feature more data from more streams #17385

Conversation

FransDel commented Sep 29, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Sep 29, 2022 • edited Loading

FransDel commented Sep 29, 2022

sajarin commented Sep 29, 2022

tuanchris commented Oct 25, 2022 • edited Loading

FransDel commented Oct 27, 2022 • edited Loading

tuanchris commented Oct 27, 2022

FransDel commented Oct 28, 2022

tuanchris commented Oct 30, 2022

sajarin commented Oct 31, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm commented Nov 22, 2022

marcosmarxm commented Dec 20, 2022

marcosmarxm commented Mar 21, 2023 • edited by github-actions bot Loading

Build Passed

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm commented Mar 21, 2023 • edited by github-actions bot Loading

FransDel commented Sep 29, 2022 •

edited

Loading

CLAassistant commented Sep 29, 2022 •

edited

Loading

tuanchris commented Oct 25, 2022 •

edited

Loading

FransDel commented Oct 27, 2022 •

edited

Loading

sajarin commented Oct 31, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Mar 21, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented Mar 21, 2023 •

edited by github-actions bot

Loading