Source source-pinterest: implement AvailabilityStrategy #22265
Labels
area/connectors
Connector related issues
team/connectors-python
type/enhancement
New feature or request
What
We want GA connectors to make use of an AvailabilityStrategy in their streams to be able to perform error handling on the
stream level. The base use case supported by the
HttpAvailabilityStrategy
is for stream-based permissions,in order to be able to skip a stream that the connector no longer has access to (because of e.g. scope
changes on an Oauth user config). An
AvailabilityStrategy
can be used to address any error that may occuron a stream level, to avoid hitting errors, or having to except the same error for every record in a stream,
instead of skipping that stream altogether.
How
There are a few different ways to go about this, usually depending on how much custom error handling we already
do in a connector, but also due to how the connector works.
the AvailabilityStrategy for the streams to
None
. This is useful if we don't currently do much custom error handlingin the connector.
removing the custom error handling, but continuing any testing we have that 403s are handled gracefully by the connector
the HttpAvailabilityStrategy to change the
reasons_for_unavailable_status_codes
method to except more error codes andprovide messages for those error codes.
the AvailabilityStrategy base class. For example, if there are clearly defined scopes required to access each stream in
a given source, you might instead determine whether a stream is available by checking the scopes on the authenticator to
see if it has the scopes required by the stream.
Note that the AvailabilityStrategy is defined on each stream. For most cases, it is easiest to set it in your stream's
base stream class, where the override is, but for some use cases, different streams might have different availability strategies -
and that's totally fine.
The text was updated successfully, but these errors were encountered: