Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source source-pinterest: implement AvailabilityStrategy #22265

Closed
erohmensing opened this issue Feb 1, 2023 · 0 comments · Fixed by #32299
Closed

Source source-pinterest: implement AvailabilityStrategy #22265

erohmensing opened this issue Feb 1, 2023 · 0 comments · Fixed by #32299
Assignees
Labels
area/connectors Connector related issues team/connectors-python type/enhancement New feature or request

Comments

@erohmensing
Copy link
Contributor

What

We want GA connectors to make use of an AvailabilityStrategy in their streams to be able to perform error handling on the
stream level. The base use case supported by the HttpAvailabilityStrategy is for stream-based permissions,
in order to be able to skip a stream that the connector no longer has access to (because of e.g. scope
changes on an Oauth user config). An AvailabilityStrategy can be used to address any error that may occur
on a stream level, to avoid hitting errors, or having to except the same error for every record in a stream,
instead of skipping that stream altogether.

How

There are a few different ways to go about this, usually depending on how much custom error handling we already
do in a connector, but also due to how the connector works.

  • If we have no custom error handling: Inherit the default HttpAvailabilityStrategy. This is as simple as removing the override setting that currently sets
    the AvailabilityStrategy for the streams to None. This is useful if we don't currently do much custom error handling
    in the connector.
  • If we have custom error handling for 403: Forbidden errors: Do the above to inherit the HttpAvailabilityStrategy,
    removing the custom error handling, but continuing any testing we have that 403s are handled gracefully by the connector
  • If we have custom error handling for lots of per-stream errors: Extend the HttpAvailabilityStrategy. You can subclass
    the HttpAvailabilityStrategy to change the reasons_for_unavailable_status_codes method to except more error codes and
    provide messages for those error codes.
  • If there is a smarter way to determine availability than trying to read the first record and excepting errors: Inherit
    the AvailabilityStrategy base class. For example, if there are clearly defined scopes required to access each stream in
    a given source, you might instead determine whether a stream is available by checking the scopes on the authenticator to
    see if it has the scopes required by the stream.

Note that the AvailabilityStrategy is defined on each stream. For most cases, it is easiest to set it in your stream's
base stream class, where the override is, but for some use cases, different streams might have different availability strategies -
and that's totally fine.

@erohmensing erohmensing added area/connectors Connector related issues team/connectors-python type/enhancement New feature or request labels Feb 1, 2023
@midavadim midavadim self-assigned this Nov 11, 2023
@midavadim midavadim linked a pull request Nov 13, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues team/connectors-python type/enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants