🎉 Source bigcommerce: add brands and categories streams #20518

PhilipCorr · 2022-12-15T14:22:07Z

What

Add two new streams. One for brands and one for categories.

How

The endpoints for these are described here:
https://developer.bigcommerce.com/api-reference/c2610608c20c8-get-all-brands
https://developer.bigcommerce.com/api-reference/9cc3a53863922-get-all-categories

They are both very similar in that the url matches:

https://api.bigcommerce.com/stores/{store_hash}/v3/catalog/{data_field}

where the data field is either brands or categories.
Neither of these endpoints have a date field that can be used for incremental syncing.
Instead, similar to the existing Pages source, they have an id field.
This is why we need to override the read_records method and explicitly call the filter_records_newer_than_state method.

🚨 User Impact 🚨

No breaking changes. This just adds two new streams which the user can opt into pulling if they want.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

There are no unit tests for the bigcommerce connector.

Integration

There are no integration tests for the bigcommerce connector.

Acceptance

===========================================================================================
../../bases/source-acceptance-test/source_acceptance_test/config.py:257: 30 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 1 warning
/Users/phil/Repos/phil/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/config.py:257: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
logging.warn("The acceptance-test-config.yml file is in a legacy format. Please migrate to the latest format.")

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (26.17s):
31 passed

…product-info-to-orders-stream

…-categories-to-bigcommerce

marcosmarxm · 2022-12-20T19:53:18Z

Hello 👋:skin-tone-2: and thank you for your contribution!

Airbyte has instituted a code freeze between 19 and 30 December, to make sure there are no disruptions during the holidays.
Because of this, reviewing and merging your contribution may take longer than usual.
We apologize for the delay, but we want everyone to have a quiet and happy holiday.

If you have any questions or need further clarification, please don't hesitate to ping via Slack.

haithem-souala · 2022-12-20T21:47:17Z

Hey @PhilipCorr, thank you for your contribution.
Ill review your PR very soon.

haithem-souala · 2022-12-21T16:06:24Z

/test connector=connectors/source-bigcommerce

🕑 connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/3750893697
❌ connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/3750893697
🐛 https://gradle.com/s/nsmifzmq456ma

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
======= 1 failed, 29 passed, 1 skipped, 31 warnings in 63.98s (0:01:03) ========

haithem-souala · 2022-12-21T17:04:26Z

Your PR looks good, i will publish the connector soon.

haithem-souala · 2022-12-22T11:32:09Z

/publish connector=connectors/source-bigcommerce

🕑 Publishing the following connectors:
connectors/source-bigcommerce
https://github.com/airbytehq/airbyte/actions/runs/3757394824

Connector	Did it publish?	Were definitions generated?
connectors/source-bigcommerce	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

haithem-souala · 2022-12-22T12:41:46Z

/publish connector=connectors/source-bigcommerce run-tests=false

🕑 Publishing the following connectors:
connectors/source-bigcommerce
https://github.com/airbytehq/airbyte/actions/runs/3757843131

Connector	Did it publish?	Were definitions generated?
connectors/source-bigcommerce	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

haithem-souala · 2022-12-22T13:03:53Z

@sajarin this PR is ready to merge.
Thank you @PhilipCorr again, for your contribution!

PhilipCorr · 2023-01-05T08:57:11Z

Hi @haithem-souala and @sajarin, are we happy to merge this PR? Thanks!

PhilipCorr · 2023-01-10T10:18:30Z

Hi @haithem-souala and @sajarin it would be great to get this merged soon if possible. It's starting to block some work that I have ongoing. Thanks!

haithem-souala · 2023-01-10T14:57:43Z

Hi @haithem-souala and @sajarin it would be great to get this merged soon if possible. It's starting to block some work that I have ongoing. Thanks!

Hi @PhilipCorr, I am reaching out to some members of the Airbyte team to request the merging of this PR. As you may be aware, the merge process was on hold from December 19th to December 30th. I appreciate your patience.

PhilipCorr · 2023-01-10T16:41:04Z

That's great, thanks for the help @haithem-souala!

…-categories-to-bigcommerce

sajarin · 2023-01-12T17:02:10Z

Thanks @PhilipCorr for the PR and thanks @haithem-souala for the review :)

* Add product info to the orders stream * Update docs * Refactor order_products into its own stream * formatting * Update docs and docker version * remove duplicate field * Add brands and categories streams to bigcommerce * remove duplicate state * remove duplicate source * update docs * update docs * bump docker version * update markdown * lint * auto-bump connector version Co-authored-by: Sajarin <sajarindider@gmail.com> Co-authored-by: Haithem SOUALA <haithem.souala@woopit.fr> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

PhilipCorr and others added 14 commits December 5, 2022 18:21

Add product info to the orders stream

4eb3464

Update docs

244d66c

Merge branch 'master' of github.com:PhilipCorr/airbyte into phil/add-…

0289721

…product-info-to-orders-stream

Refactor order_products into its own stream

d38bdcf

formatting

e207226

Update docs and docker version

1a2588f

Merge branch 'master' into phil/add-product-info-to-orders-stream

69e3eb6

Merge branch 'master' into phil/add-product-info-to-orders-stream

a80720f

remove duplicate field

17f4002

Merge branch 'master' into phil/add-product-info-to-orders-stream

8d34356

Add brands and categories streams to bigcommerce

96b6306

Merge remote-tracking branch 'origin/master' into phil/add-brands-and…

ab5861f

…-categories-to-bigcommerce

remove duplicate state

d39516b

remove duplicate source

31a1136

octavia-squidington-iv added community connectors/source/bigcommerce area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Dec 15, 2022

update docs

546410a

PhilipCorr changed the title ~~Phil/add brands and categories to bigcommerce~~ 🎉 Source bigcommerce: add brands and categories streams Dec 15, 2022

update docs

2759187

PhilipCorr marked this pull request as ready for review December 15, 2022 14:48

octavia-squidington-iv added the bounty label Dec 15, 2022

PhilipCorr added 3 commits December 16, 2022 10:11

Merge remote-tracking branch 'origin/master' into phil/add-brands-and…

9363479

…-categories-to-bigcommerce

bump docker version

c6b34a9

update markdown

f90e923

sajarin added the bounty-L Maintainer program: claimable large bounty PR label Dec 19, 2022

haithem-souala self-assigned this Dec 20, 2022

lint

c53939a

Merge branch 'master' into phil/add-brands-and-categories-to-bigcommerce

30ae0bf

haithem-souala self-requested a review December 21, 2022 17:02

haithem-souala approved these changes Dec 21, 2022

View reviewed changes

Merge branch 'master' into phil/add-brands-and-categories-to-bigcommerce

02154a4

auto-bump connector version

7061ecd

haithem-souala enabled auto-merge (squash) January 5, 2023 13:08

haithem-souala and others added 5 commits January 5, 2023 14:08

Merge branch 'master' into phil/add-brands-and-categories-to-bigcommerce

69341c0

Merge branch 'master' into phil/add-brands-and-categories-to-bigcommerce

9db532c

Merge branch 'master' into phil/add-brands-and-categories-to-bigcommerce

d96bcce

Merge branch 'master' into phil/add-brands-and-categories-to-bigcommerce

a96a08a

Merge branch 'master' into phil/add-brands-and-categories-to-bigcommerce

9406beb

Merge branch 'master' into phil/add-brands-and-categories-to-bigcommerce

6f77656

haithem-souala disabled auto-merge January 10, 2023 14:45

Merge remote-tracking branch 'origin/master' into phil/add-brands-and…

b120b44

…-categories-to-bigcommerce

sajarin merged commit ceadf31 into airbytehq:master Jan 12, 2023

octavia-squidington-iii mentioned this pull request Jan 12, 2023

Bump Airbyte version from 0.40.27 to 0.40.28 #21359

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source bigcommerce: add brands and categories streams #20518

🎉 Source bigcommerce: add brands and categories streams #20518

PhilipCorr commented Dec 15, 2022 •

edited

Loading

marcosmarxm commented Dec 20, 2022

haithem-souala commented Dec 20, 2022

haithem-souala commented Dec 21, 2022 •

edited by github-actions bot

Loading

haithem-souala commented Dec 21, 2022

haithem-souala commented Dec 22, 2022 •

edited by github-actions bot

Loading

haithem-souala commented Dec 22, 2022 •

edited by github-actions bot

Loading

haithem-souala commented Dec 22, 2022

PhilipCorr commented Jan 5, 2023

PhilipCorr commented Jan 10, 2023

haithem-souala commented Jan 10, 2023

PhilipCorr commented Jan 10, 2023

sajarin commented Jan 12, 2023

🎉 Source bigcommerce: add brands and categories streams #20518

🎉 Source bigcommerce: add brands and categories streams #20518

Conversation

PhilipCorr commented Dec 15, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

marcosmarxm commented Dec 20, 2022

haithem-souala commented Dec 20, 2022

haithem-souala commented Dec 21, 2022 • edited by github-actions bot Loading

Build Failed

haithem-souala commented Dec 21, 2022

haithem-souala commented Dec 22, 2022 • edited by github-actions bot Loading

haithem-souala commented Dec 22, 2022 • edited by github-actions bot Loading

haithem-souala commented Dec 22, 2022

PhilipCorr commented Jan 5, 2023

PhilipCorr commented Jan 10, 2023

haithem-souala commented Jan 10, 2023

PhilipCorr commented Jan 10, 2023

sajarin commented Jan 12, 2023

PhilipCorr commented Dec 15, 2022 •

edited

Loading

haithem-souala commented Dec 21, 2022 •

edited by github-actions bot

Loading

haithem-souala commented Dec 22, 2022 •

edited by github-actions bot

Loading

haithem-souala commented Dec 22, 2022 •

edited by github-actions bot

Loading