🎉 New Source: 3PL Central #7322

monai · 2021-10-25T09:18:40Z

What

3PL Central API source implementing the same streams as tap-3plcentral.
Participating in Hacktoberfest 2021, issue Source: 3PL Central connector-contest#72

Upstream API documentation:

How

The source queries 3PL Central HTTP API using Python CDK and applies the following transformations:

Converts all JSON object keys from TitleCase to snake_case.
Recursively removes HAL _link keys from all objects.
Adds primary key and cursor fields.

Unlike the Singer tap mentioned above, the source implementation does not apply any opinionated fetched data cleanup or flattening.

Pre-merge Checklist

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

harshithmullapudi · 2021-11-15T08:34:06Z

airbyte-integrations/connectors/source-tplcentral/integration_tests/catalog.json

+            "$schema": "http://json-schema.org/draft-07/schema#",
+            "type": "object",
+            "additionalProperties": false,
+            "properties": {


hey we can keep properties as empty. Since these are just integration tests it's ok to leave these as empty objects

monai · 2021-12-22T15:15:24Z

Recent changes:

Added schema source files to the repo since Basic normalization doesn't extract nested objects using $ref/$defs #8241 is fixed.
Added two new properties to the schema that were added in 3PL Central API.
Split configured catalog to one stream per file. Acceptance tests timed out due to long-running streams.
Updated documentationUrl in spec.json.
Run gradlew format.

This reverts commit d4e8fea.

harshithmullapudi

LGTM

Can you run ./gradlew format
Also share the screenshots for integration_tests and unit_tests

monai · 2021-12-27T14:44:42Z

It is already done. Just run again, and no changes were generated.

Unit tests

~/projects/lt/airbyte/airbyte-integrations/connectors/source-tplcentral (source-tplcentral-alt) [um] python -m pytest -v unit_tests/                 [.venv]
Test session starts (platform: darwin, Python 3.9.7, pytest 6.1.2, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/juozas/projects/lt/airbyte, configfile: pytest.ini
plugins: cov-2.12.1, sugar-0.9.4, requests-mock-1.9.3, mock-3.6.1, timeout-1.4.2
collecting ...
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_incremental_streams.py::test_cursor_field ✓                                  5% ▌
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_incremental_streams.py::test_get_updated_state ✓                            10% ▉
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_incremental_streams.py::test_stream_slices ✓                                14% █▌
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_incremental_streams.py::test_supports_incremental ✓                         19% █▉
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_incremental_streams.py::test_source_defined_cursor ✓                        24% ██▍
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_incremental_streams.py::test_stream_checkpoint_interval ✓                   29% ██▉
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_source.py::test_check_connection ✓                                          33% ███▍
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_source.py::test_streams ✓                                                   38% ███▊
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_request_params ✓                                           43% ████▍
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_next_page_token ✓                                          48% ████▊
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_next_page_token_with_page_size ✓                           52% █████▎
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_parse_response ✓                                           57% █████▊
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_parse_response_with_primary_key ✓                          62% ██████▎
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_parse_response_with_cursor_field ✓                         67% ██████▋
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_request_headers ✓                                          71% ███████▎
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_http_method ✓                                              76% ███████▋
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_should_retry[HTTPStatus.OK-False] ✓                        81% ████████▏
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_should_retry[HTTPStatus.BAD_REQUEST-False] ✓               86% ████████▋
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_should_retry[HTTPStatus.TOO_MANY_REQUESTS-True] ✓          90% █████████▏
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_should_retry[HTTPStatus.INTERNAL_SERVER_ERROR-True] ✓      95% █████████▌
 airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py::test_backoff_time ✓                                            100% ██████████
===================================================================== warnings summary ======================================================================
airbyte-integrations/connectors/source-tplcentral/unit_tests/test_incremental_streams.py: 6 warnings
airbyte-integrations/connectors/source-tplcentral/unit_tests/test_source.py: 6 warnings
airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py: 13 warnings
  /Users/juozas/projects/lt/airbyte/airbyte-integrations/connectors/source-tplcentral/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:38: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator = NoAuth()

airbyte-integrations/connectors/source-tplcentral/unit_tests/test_incremental_streams.py: 6 warnings
airbyte-integrations/connectors/source-tplcentral/unit_tests/test_source.py: 6 warnings
airbyte-integrations/connectors/source-tplcentral/unit_tests/test_streams.py: 13 warnings
  /Users/juozas/projects/lt/airbyte/airbyte-integrations/connectors/source-tplcentral/.venv/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.62s):
      21 passed

Acceptance tests

~/projects/lt/airbyte/airbyte-integrations/connectors/source-tplcentral python -m pytest -p integration_tests.acceptance                                                                         [.venv]
Test session starts (platform: darwin, Python 3.9.7, pytest 6.1.2, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/juozas/projects/lt/airbyte, configfile: pytest.ini
plugins: cov-2.12.1, sugar-0.9.4, requests-mock-1.9.3, mock-3.6.1, timeout-1.4.2
collecting ...
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓                                                       4% ▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓                                                             7% ▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓                                                            11% █▏
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓                                                          14% █▌
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓                                          18% █▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓                                21% ██▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓                                               25% ██▌
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓                                                         29% ██▉
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓                                                         32% ███▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓                                                       36% ███▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓                                39% ███▉
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓                                   43% ████▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓                                                           46% ████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs1] ✓                                                           50% █████
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs2] ✓                                                           54% █████▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs3] ✓                                                           57% █████▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs4] ✓                                                           61% ██████▏
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs5] ✓                                                           64% ██████▌
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓                                     68% ██████▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs1] ✓                                     71% ███████▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs0] ✓                                  75% ███████▌
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs1] ✓                                  79% ███████▉
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs2] ✓                                  82% ████████▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs3] ✓                                  86% ████████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs0] ✓                    89% ████████▉
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs1] ✓                    93% █████████▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs2] ✓                    96% █████████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs3] ✓                   100% ██████████

Results (79.58s):
      28 passed

monai added 30 commits October 4, 2021 14:14

Add source-tplcentral

973d860

Add source definition

f0577f7

Add missing file

1f53872

Update catalog

f282bdf

Update sample config

ad01871

Update formatting

f9e216a

Update invalid config

ad81677

Fix primary key identifier

fde6519

Update naming

39120c2

Rename variable

6511f59

Parametrize page size

0fff769

Merge branch 'upstream-master' into source-tplcentral

4e1b7d0

Implement Items stream

f034ea1

Merge branch 'master' into source-tplcentral

be16d7c

Update schemas

d977e8e

Update response normalization

3be6c78

Add arrow

1344445

Implement items stream cursor

e1daed1

Fix types

0afa54b

Cleanup

e24f3f6

Add customers stream

342ad54

Fix items sort

4790d8d

Add stock_details stream

6687556

Add inventory stream

1bd8514

Add orders stream

833fbc2

Refactor streams to a separate file

84f3dfe

Update cursor handling

edeb6b0

Refactor request_params

0e4327c

Simplify

37e2c51

Update params generation

5de3ccb

alafanechere assigned harshithmullapudi Dec 21, 2021

alafanechere removed the waiting-for-cred Waiting for CI credentials label Dec 21, 2021

harshithmullapudi suggested changes Dec 22, 2021

View reviewed changes

monai added 9 commits December 22, 2021 15:01

Merge branch 'master' into source-tplcentral-alt

cb88ddd

Update catalog.json

6dc2250

Fix tests

472374d

Add schema source files

1555bc8

Split configured catalogs

9e8bc00

Cleanup

d8f574a

Update documentationUrl

d54ac53

Add new properties

187a28f

Run gradlew format

4925c0d

monai added 2 commits December 22, 2021 17:45

Remove additionalProperties: false from the schemas

d4e8fea

Revert "Remove additionalProperties: false from the schemas"

7e16b5b

This reverts commit d4e8fea.

harshithmullapudi approved these changes Dec 27, 2021

View reviewed changes

harshithmullapudi mentioned this pull request Dec 28, 2021

Publish new source: 3PL Central #9135

Merged

harshithmullapudi temporarily deployed to more-secrets December 28, 2021 07:45 Inactive

jrhizor temporarily deployed to more-secrets December 28, 2021 07:47 Inactive

jrhizor temporarily deployed to more-secrets December 28, 2021 07:56 Inactive

harshithmullapudi merged commit 22dcb0f into airbytehq:master Dec 28, 2021

jrhizor mentioned this pull request Jan 3, 2022

Bump Airbyte version from 0.35.2-alpha to 0.35.3-alpha #9262

Merged

monai deleted the source-tplcentral-alt branch January 31, 2022 08:12

marcosmarxm removed the hacktober label Oct 20, 2022

octavia-squidington-iv added bounty connectors/source/tplcentral labels Oct 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: 3PL Central #7322

🎉 New Source: 3PL Central #7322

monai commented Oct 25, 2021 •

edited

Loading

harshithmullapudi Nov 15, 2021

monai commented Dec 22, 2021 •

edited

Loading

harshithmullapudi left a comment •

edited by alafanechere

Loading

monai commented Dec 27, 2021 •

edited

Loading

🎉 New Source: 3PL Central #7322

🎉 New Source: 3PL Central #7322

Conversation

monai commented Oct 25, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

harshithmullapudi Nov 15, 2021

Choose a reason for hiding this comment

monai commented Dec 22, 2021 • edited Loading

harshithmullapudi left a comment • edited by alafanechere Loading

Choose a reason for hiding this comment

monai commented Dec 27, 2021 • edited Loading

monai commented Oct 25, 2021 •

edited

Loading

monai commented Dec 22, 2021 •

edited

Loading

harshithmullapudi left a comment •

edited by alafanechere

Loading

monai commented Dec 27, 2021 •

edited

Loading