🎉 New Source: Freshsales #6963

tuanchris · 2021-10-12T01:09:37Z

What

Create a new source for Freshales. API document here
Participating in hacktoberfest-2021, issue no 33

How

Describe the solution

Connection ran successfully in local env.

Stream settings

Local JSON files

Example output

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

CLAassistant · 2021-10-12T01:09:49Z

All committers have signed the CLA.

marcosmarxm · 2021-10-13T16:10:24Z

@tuanchris any possibility to run a sync with a postgres destinaiton and basic normalization on? =)

tuanchris · 2021-10-13T16:53:31Z

@marcosmarxm I actually did that with BigQuery, hope that it is fine too. All seems to be working well but this.

2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 6 of 10 OK created table model freshsales.upcoming_appointments.............................................. [�[32mCREATE TABLE (2.0 rows, 1.3 KB processed)�[0m in 3.54s]
2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 3 of 10 OK created table model freshsales.lost_deals......................................................... [�[32mCREATE TABLE (1.0 rows, 2.2 KB processed)�[0m in 3.67s]
2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 5 of 10 OK created table model freshsales.open_deals......................................................... [�[32mCREATE TABLE (1.0 rows, 2.2 KB processed)�[0m in 3.68s]
2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 8 of 10 OK created table model freshsales.won_deals.......................................................... [�[32mCREATE TABLE (1.0 rows, 2.2 KB processed)�[0m in 3.79s]
2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 4 of 10 OK created table model freshsales.past_appointments.................................................. [�[32mCREATE TABLE (2.0 rows, 1.3 KB processed)�[0m in 3.90s]
2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 10 of 10 OK created table model freshsales.accounts.......................................................... [�[32mCREATE TABLE (1.0 rows, 1.5 KB processed)�[0m in 3.70s]
2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 2 of 10 OK created table model freshsales.contacts........................................................... [�[32mCREATE TABLE (100.0 rows, 208.1 KB processed)�[0m in 4.00s]
2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 1 of 10 ERROR creating table model freshsales.completed_tasks................................................ [�[31mERROR�[0m in 4.04s]
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 16:53:03 | 7 of 10 OK created table model freshsales.sales_activities................................................... [�[32mCREATE TABLE (0.0 rows, 0.0 Bytes processed)�[0m in 4.11s]
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 16:53:03 | 9 of 10 ERROR creating table model freshsales.open_tasks..................................................... [�[31mERROR�[0m in 3.99s]
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 16:53:03 | 
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 16:53:03 | Finished running 10 table models in 6.65s.
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - �[31mCompleted with 2 errors and 0 warnings:�[0m
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - �[33mDatabase Error in model completed_tasks (models/generated/airbyte_tables/freshsales/completed_tasks.sql)�[0m
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 -   Bad bool value: 1
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 -   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/freshsales/completed_tasks.sql
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - �[33mDatabase Error in model open_tasks (models/generated/airbyte_tables/freshsales/open_tasks.sql)�[0m
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 -   Bad bool value: 0
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 -   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/freshsales/open_tasks.sql
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 
2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - Done. PASS=8 WARN=0 ERROR=2 SKIP=0 TOTAL=10

Should I change the boolean schema to integer or what is the preferred way of dealing with this?

marcosmarxm

Some comments, but overall looks good! I'll set up this locally and test. Can you check the comments?

airbyte-integrations/connectors/source-freshsales/source_freshsales/source.py

tuanchris · 2021-10-14T04:06:26Z

@marcosmarxm Running all good locally with basic normalization to BigQuery now

marcosmarxm · 2021-10-19T17:49:55Z

thanks @tuanchris going to review and test the connector later today.

marcosmarxm · 2021-10-20T01:48:23Z

Hi @tuanchris the unit tests are not passing, can you check them?
The integration tests work great! So after correct then I think we can merge this contribution.

marcosmarxm

Unit tests are not passing.

================================================================================ short test summary info ================================================================================
FAILED unit_tests/test_source.py::test_check_connection - assert (False,\n ConnectionError(MaxRetryError('HTTPSConnectionPool(host="%3cmagicmock%20name=\'mock.__getitem__()\'%20id=\'...
FAILED unit_tests/test_streams.py::test_request_params - TypeError: __init__() missing 1 required positional argument: 'domain_name'
FAILED unit_tests/test_streams.py::test_next_page_token - TypeError: __init__() missing 1 required positional argument: 'domain_name'
FAILED unit_tests/test_streams.py::test_parse_response - TypeError: __init__() missing 1 required positional argument: 'domain_name'
FAILED unit_tests/test_streams.py::test_request_headers - TypeError: __init__() missing 1 required positional argument: 'domain_name'
FAILED unit_tests/test_streams.py::test_http_method - TypeError: __init__() missing 1 required positional argument: 'domain_name'
FAILED unit_tests/test_streams.py::test_should_retry[HTTPStatus.OK-False] - TypeError: __init__() missing 1 required positional argument: 'domain_name'
FAILED unit_tests/test_streams.py::test_should_retry[HTTPStatus.BAD_REQUEST-False] - TypeError: __init__() missing 1 required positional argument: 'domain_name'
FAILED unit_tests/test_streams.py::test_should_retry[HTTPStatus.TOO_MANY_REQUESTS-True] - TypeError: __init__() missing 1 required positional argument: 'domain_name'
FAILED unit_tests/test_streams.py::test_should_retry[HTTPStatus.INTERNAL_SERVER_ERROR-True] - TypeError: __init__() missing 1 required positional argument: 'domain_name'
FAILED unit_tests/test_streams.py::test_backoff_time - TypeError: __init__() missing 1 required positional argument: 'domain_name'

airbyte-integrations/connectors/source-freshsales/unit_tests/test_source.py

tuanchris · 2021-10-20T15:22:19Z

all fixed @marcosmarxm

marcosmarxm · 2021-10-22T03:50:49Z

Unit tests are passing but Integration tests aren't.

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError:  Rec...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
=================== 2 failed, 10 passed, 1 skipped in 53.96s ===================

Make sure ./gradlew airbyte-integrations:connectors:source-freshsales:integrationTest is working and share a screenshoot showing the result.

tuanchris · 2021-10-22T07:51:56Z

@marcosmarxm I'm stuck here, can you help take a look? Two problems that I'm facing is with the following message running integration test:

INFO     detailed_logger acceptance_tests_logs/test_full_refresh.py__TestFullRefresh__test_sequential_reads[inputs0].txt:test_full_refresh.py:25 The two sequential reads should produce either equal set of records or one of them is a strict subset of the other

So it seems that the results from Freshsales gave more columns than in the doc. I have consolidated all possible columns, change the type to ["null", "data_type"] but the error still persists.

Another warning that I'm seeing is like follow but I cannot debug it also.

WARNING  root:connector_runner.py:112 Unable to parse connector's output 1 validation error for AirbyteMessage
type
  field required (type=value_error.missing)

tuanchris · 2021-10-25T11:11:00Z

@marcosmarxm Problem found. response was returning tokens, which changed with different requests. How do I remove this column? Removing it from the schema doesn't seem to help.

marcosmarxm · 2021-10-25T18:01:57Z

@tuanchris the problem is not the token data but the connector you built doesnt run a second full refresh properly.
All streams are Full Refresh | Overwrite. Can you check that?

The first sync: works and ingest data
Second: no data ingested.

marcosmarxm · 2021-10-25T18:26:37Z

Updating, looks maybe the Freshsales is throwing a Max call when trying to run the test. Could you try to add a sleep between record1 and record2 in accept tests?

tuanchris · 2021-10-26T10:04:41Z

@marcosmarxm I couldn't reproduce the failed attempt you show locally. Running two consecutive sync works for me

Furthermore, running the following test script for five minutes doesn't seem to throw any error:

auth = TokenAuthenticator(token=f'token={config["api_key"]}', auth_method="Token").get_auth_header()
while True: 
    url = f'https://{config["domain_name"]}/crm/sales/api/deals/view/17001600269'
    r = requests.get(filters_url, headers=auth)
    r.raise_for_status()
    print(r.json())

With the following config setting:

{
  "streams": [
    {
      "stream": {
        "name": "won_deals",
        "json_schema": {},
        "supported_sync_modes": ["full_refresh"],
        "source_defined_primary_key": [["id"]]
      },
      "sync_mode": "full_refresh",
      "destination_sync_mode": "overwrite"
    }
  ]
}

The integration test failed and I got this diff (only the token is different).

I still think that this is the culprit.

tuanchris · 2021-11-02T14:56:32Z

@marcosmarxm I removed the fc_widget_collaboration column and integration tests passed

marcosmarxm

thanks @tuanchris now integration test are working.

* initial commit * finish implementing full_refresh * add other tables * cleaning up * add docs and use requests_native_auth * fix function return different number of values * change package author * fix schema * fix schema bug * linting * fix unit test * clean up * add null for schemas * remove fc_widget_collaboration col

tuanchris added 4 commits October 11, 2021 18:38

initial commit

21c4db5

finish implementing full_refresh

bfc4306

add other tables

fd5680d

cleaning up

cd61452

github-actions bot added the area/connectors Connector related issues label Oct 12, 2021

tuanchris changed the title ~~33 freshsales source~~ 🎉 New Source: Freshsales Oct 12, 2021

tuanchris mentioned this pull request Oct 12, 2021

Source: Freshservice airbytehq/connector-contest#55

Closed

marcosmarxm added the community label Oct 12, 2021

avaidyanatha added the community_pr label Oct 12, 2021

marcosmarxm added the hacktober label Oct 13, 2021

marcosmarxm mentioned this pull request Oct 13, 2021

CI Sandbox for Freshsales #7008

Closed

marcosmarxm requested changes Oct 13, 2021

View reviewed changes

marcosmarxm self-assigned this Oct 13, 2021

marcosmarxm removed the community_pr label Oct 14, 2021

tuanchris added 4 commits October 14, 2021 09:01

add docs and use requests_native_auth

b36175c

fix function return different number of values

dac516a

change package author

9fa45f6

fix schema

d18a787

fix schema bug

e630c11

tuanchris requested a review from marcosmarxm October 18, 2021 04:04

marcosmarxm requested changes Oct 20, 2021

View reviewed changes

airbyte-integrations/connectors/source-freshsales/unit_tests/test_source.py Show resolved Hide resolved

marcosmarxm reviewed Oct 20, 2021

View reviewed changes

airbyte-integrations/connectors/source-freshsales/unit_tests/test_source.py Outdated Show resolved Hide resolved

tuanchris added 2 commits October 20, 2021 09:15

linting

420cafe

fix unit test

d07834f

tuanchris requested a review from marcosmarxm October 20, 2021 15:22

tuanchris added 2 commits October 22, 2021 12:21

clean up

b60884c

add null for schemas

057ed2c

remove fc_widget_collaboration col

a4164c8

marcosmarxm approved these changes Nov 2, 2021

View reviewed changes

marcosmarxm merged commit 73b8589 into airbytehq:master Nov 3, 2021

jrhizor mentioned this pull request Nov 3, 2021

Bump Airbyte version from 0.30.25-alpha to 0.30.26-alpha #7599

Merged

igrankova added connectors/sources-api connectors/source/freshsales labels Jan 6, 2022

marcosmarxm removed the hacktober label Oct 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Freshsales #6963

🎉 New Source: Freshsales #6963

tuanchris commented Oct 12, 2021 •

edited

Loading

CLAassistant commented Oct 12, 2021 •

edited

Loading

marcosmarxm commented Oct 13, 2021

tuanchris commented Oct 13, 2021 •

edited

Loading

marcosmarxm left a comment

tuanchris commented Oct 14, 2021

marcosmarxm commented Oct 19, 2021

marcosmarxm commented Oct 20, 2021 •

edited

Loading

marcosmarxm left a comment

tuanchris commented Oct 20, 2021

marcosmarxm commented Oct 22, 2021

tuanchris commented Oct 22, 2021

tuanchris commented Oct 25, 2021

marcosmarxm commented Oct 25, 2021 •

edited

Loading

marcosmarxm commented Oct 25, 2021

tuanchris commented Oct 26, 2021

tuanchris commented Nov 2, 2021

marcosmarxm left a comment

🎉 New Source: Freshsales #6963

🎉 New Source: Freshsales #6963

Conversation

tuanchris commented Oct 12, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

CLAassistant commented Oct 12, 2021 • edited Loading

marcosmarxm commented Oct 13, 2021

tuanchris commented Oct 13, 2021 • edited Loading

marcosmarxm left a comment

Choose a reason for hiding this comment

tuanchris commented Oct 14, 2021

marcosmarxm commented Oct 19, 2021

marcosmarxm commented Oct 20, 2021 • edited Loading

marcosmarxm left a comment

Choose a reason for hiding this comment

tuanchris commented Oct 20, 2021

marcosmarxm commented Oct 22, 2021

tuanchris commented Oct 22, 2021

tuanchris commented Oct 25, 2021

marcosmarxm commented Oct 25, 2021 • edited Loading

marcosmarxm commented Oct 25, 2021

tuanchris commented Oct 26, 2021

tuanchris commented Nov 2, 2021

marcosmarxm left a comment

Choose a reason for hiding this comment

tuanchris commented Oct 12, 2021 •

edited

Loading

CLAassistant commented Oct 12, 2021 •

edited

Loading

tuanchris commented Oct 13, 2021 •

edited

Loading

marcosmarxm commented Oct 20, 2021 •

edited

Loading

marcosmarxm commented Oct 25, 2021 •

edited

Loading