🐛 Source Hubspot: Removed stage history from deals stream

[lgomezm]

What

• Fixes: HubSpot Sync Job Hangs on Deals Sync #10544
• The deal stream currently gets all deal stage history: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-hubspot/source_hubspot/streams.py#L1046-L1048
• This in turn gets all deals: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-hubspot/source_hubspot/streams.py#L999-L1006
• That is why deal jobs hang. For full syncs, it will get all deals twice.
• Even on getting incremental updates, it will get all deals (to get the stage history), to then start yielding only the new/updated deals.

How

• I have removed, from the Deals stream, the code that gets all deal stage history records.
• I believe getting all records from another stream to set one property is not a good practice.
• Some other relationships work if one of the records includes the ID of the associated object (i.e. for the relationship between Hubspot contacts and companies, they have the associated object ID).
• Something like that can be done, at the destination, with deals and deal stage history.

Recommended reading order

1. streams.py
2. test_source.py

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/SUMMARY.md
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

[marcosmarxm]

@lgomezm it's possible to show the integration tests results? After doing this you're able to run the deals stream?

[lgomezm]

@lgomezm it's possible to show the integration tests results? After doing this you're able to run the deals stream?

@marcosmarxm Here they are. I had to update the CRMSearchStream class. They were iterating on the super class' slices, which is not necessary for that kind of stream -- It was taking too long if you used a very old start_date in your config. See 1a0596c for reference.

[lgomezm]

@marcosmarxm I'm also able to run the deals stream successfully. The acceptance tests already do that both as full refresh and as incremental updates.

[lazebnyi]

LGTM.
Small comment below.
Pls, add a changelog to hubspot.md file. And bump version in docker file.

[lazebnyi]

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/1934236400
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/1934236400
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                         Stmts   Miss  Cover
------------------------------------------------
source_hubspot/errors.py         6      0   100%
source_hubspot/__init__.py       2      0   100%
source_hubspot/streams.py      655    126    81%
source_hubspot/source.py        69     16    77%
------------------------------------------------
TOTAL                          732    142    81%

[lgomezm]

LGTM. Small comment below. Pls, add a changelog to hubspot.md file. And bump version in docker file.

@lazebnyi I've bumped the version in 5da84c8.

[lazebnyi]

/publish connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/1935067838
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/1935067838