Emit multiple error trace messages and continue syncs by default #34636

brianjlai · 2024-01-29T22:13:59Z

What

Adds new behavior to the abstract source inherited by most connectors so that if one stream raises an exception, subsequent streams will continue to get synced. This is the new default behavior, but connectors can opt out via the stop_sync_on_stream_failure property on `AbstractSource.

This change is also necessary to fix Sentry error reporting logic that wouldn't work when the final error reported was a summary of previously seen stream exceptions.

How

The implementation is pretty straightforward in that every exception raised by a stream will yield one error trace message. We also collect these exceptions and include them in a final exception that is thrown at the end of the sync if any exceptions were found. This is important because the connector must end with a non-zero status code or the platform will not recognize the sync as failed.

I also got rid of all the original raise exception flow from the legacy behavior so that there is consistency in the pattern we emit error traces messages (whether it be 1 or many) regardless of the value of the stop_sync_on_stream_failure.

This has been tested using various pre-release versions of source-klaviyo on Cloud and verifying that syncs continue onto other streams and that the first stream error that is received gets propagated to Sentry so we retain the grouping behavior.

Sentry Dashboard w/ errors caught: https://airbytehq.sentry.io/issues/?query=is%3Aunresolved+klaviyo+connector_name%3AKlaviyo&referrer=issue-list&statsPeriod=30d

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user?

This does change the potential amount of messages emitted by a connector, especially if an existing connector and sync has many failing streams. However, beyond that, this change is additive. I will also include a follow up PR to fix source-klaviyo to remove the temporary continue_sync_on_stream_failure() method used for the original short term fix.

vercel · 2024-01-29T22:14:06Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 7, 2024 1:24am

…e non-zero error code on exception

brianjlai · 2024-02-02T00:39:33Z

airbyte-cdk/python/airbyte_cdk/sources/abstract_source.py

@@ -282,17 +303,17 @@ def message_repository(self) -> Union[None, MessageRepository]:
        return _default_message_repository

    @property
-    def continue_sync_on_stream_failure(self) -> bool:
+    def stop_sync_on_stream_failure(self) -> bool:


If we're 👍 w/ this interface name I can remove the warning

brianjlai · 2024-02-02T00:48:16Z

airbyte-cdk/python/unit_tests/sources/test_source_read.py

@@ -343,7 +343,7 @@ def test_concurrent_source_yields_the_same_messages_as_abstract_source_when_an_e
    source, concurrent_source = _init_sources([stream_slice_to_partition], state, logger)
    config = {}
    catalog = _create_configured_catalog(source._streams)
-    messages_from_abstract_source = _read_from_source(source, logger, config, catalog, state, RuntimeError)
+    messages_from_abstract_source = _read_from_source(source, logger, config, catalog, state, AirbyteTracedException)


This is one area of deviation between the concurrent and regular abstract source. The behavior in abstract_source.py catches exceptions and emits trace errors. And at the end if at least one exception was found, it raises a final AirbyteTracedException that will make sure the process ends w/ a non-zero error code.

As far as I can tell, the concurrent just emits the same exception the source implementation emits. I am curious, but does that mean that a failure on one-stream in concurrent will derail all the concurrently running steams/slices?

I'm not entirely sure what parity should look like because the last error message for the regular abstract source is a summary of prior errors instead of a specific one. How do we feel about leaving this functionality outside of concurrent and filing a ticket to revisit this?

your understanding is correct. Can we write a follow up issue for concurrent? I think it would be good to implement this as well

brianjlai · 2024-02-02T00:48:32Z

airbyte-cdk/python/airbyte_cdk/sources/abstract_source.py

+            error_message = self._generate_failed_streams_error_message(stream_name_to_exception)
+            logger.info(error_message)
+            # We still raise at least one exception when a stream raises an exception because the platform
+            # currently relies on a non-zero exit code to determine if a sync attempt has failed


girarda

✅ with a request for a follow up for the concurrent cdk

girarda · 2024-02-05T20:55:51Z

airbyte-cdk/python/unit_tests/sources/test_source_read.py

@@ -343,7 +343,7 @@ def test_concurrent_source_yields_the_same_messages_as_abstract_source_when_an_e
    source, concurrent_source = _init_sources([stream_slice_to_partition], state, logger)
    config = {}
    catalog = _create_configured_catalog(source._streams)
-    messages_from_abstract_source = _read_from_source(source, logger, config, catalog, state, RuntimeError)
+    messages_from_abstract_source = _read_from_source(source, logger, config, catalog, state, AirbyteTracedException)


your understanding is correct. Can we write a follow up issue for concurrent? I think it would be good to implement this as well

)

…bytehq#34636)

)

Emit multiple error trace messages and continue syncs by default

469f82b

octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/source/klaviyo labels Jan 29, 2024

brianjlai added 12 commits January 29, 2024 14:26

fix flake checks

5c89869

fix more integration tests

e126c44

fix imports

d69114e

formatting

28c5f0b

fix discover for test connector

09c08de

Merge branch 'master' into brian/per_stream_error_reporting

fc5c5d1

rewrite abstract_source to still emit at least one exception to ensur…

57e8ad5

…e non-zero error code on exception

remove stop_sync_on_stream_failure override from klaviyo

f0cd314

only one error stream

b942080

fix failing test but slight divergence from concurrent behavior

91b6fa9

remove klaviyo pre-release testing code

76be107

formatting and add back the original method

3ff86e0

octavia-squidington-iii removed the area/connectors Connector related issues label Feb 2, 2024

brianjlai commented Feb 2, 2024

View reviewed changes

brianjlai marked this pull request as ready for review February 2, 2024 01:23

brianjlai requested a review from a team as a code owner February 2, 2024 01:23

airbytehq deleted a comment from github-actions bot Feb 2, 2024

remove extra comments

1a2c77c

brianjlai requested a review from girarda February 5, 2024 19:46

girarda approved these changes Feb 5, 2024

View reviewed changes

Merge branch 'master' into brian/per_stream_error_reporting

0ab8ae6

vercel bot deployed to Preview February 7, 2024 01:24 View deployment

brianjlai merged commit cc2a6e2 into master Feb 7, 2024
27 checks passed

brianjlai deleted the brian/per_stream_error_reporting branch February 7, 2024 18:34

This was referenced Feb 8, 2024

Revert "Emit multiple error trace messages and continue syncs by default" #34990

Merged

Emit multiple error trace messages and continue syncs by default #35129

Merged

xiaohansong pushed a commit that referenced this pull request Feb 13, 2024

Emit multiple error trace messages and continue syncs by default (#34636

504684c

)

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 21, 2024

Emit multiple error trace messages and continue syncs by default (air…

f857762

…bytehq#34636)

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

Emit multiple error trace messages and continue syncs by default (air…

719264b

…bytehq#34636)

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

Emit multiple error trace messages and continue syncs by default (air…

7cec82a

…bytehq#34636)

xiaohansong pushed a commit that referenced this pull request Feb 27, 2024

Emit multiple error trace messages and continue syncs by default (#34636

991a552

)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Emit multiple error trace messages and continue syncs by default #34636

Emit multiple error trace messages and continue syncs by default #34636

brianjlai commented Jan 29, 2024 •

edited

Loading

vercel bot commented Jan 29, 2024 •

edited

Loading

brianjlai Feb 2, 2024 •

edited

Loading

brianjlai Feb 2, 2024

girarda Feb 5, 2024

brianjlai Feb 2, 2024 •

edited

Loading

girarda left a comment

girarda Feb 5, 2024

Emit multiple error trace messages and continue syncs by default #34636

Emit multiple error trace messages and continue syncs by default #34636

Conversation

brianjlai commented Jan 29, 2024 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

vercel bot commented Jan 29, 2024 • edited Loading

brianjlai Feb 2, 2024 • edited Loading

Choose a reason for hiding this comment

brianjlai Feb 2, 2024

Choose a reason for hiding this comment

girarda Feb 5, 2024

Choose a reason for hiding this comment

brianjlai Feb 2, 2024 • edited Loading

Choose a reason for hiding this comment

girarda left a comment

Choose a reason for hiding this comment

girarda Feb 5, 2024

Choose a reason for hiding this comment

brianjlai commented Jan 29, 2024 •

edited

Loading

vercel bot commented Jan 29, 2024 •

edited

Loading

brianjlai Feb 2, 2024 •

edited

Loading

brianjlai Feb 2, 2024 •

edited

Loading