You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
During a sync, connectors receive stream state from the platform by a reference to input_state.json. This JSON blob contains the state of all streams in the following LEGACY format:
When we read the JSON file in entrypoint.py / source.py, we should accept either format and parse it into a list of AirbyteStateMessage(s) which will then be interpreted by abstract_source.py later while processing an incremental sync.
After the changes:
LEGACY state should be parsed into a list of one AirbyteStateMessage:
brianjlai
changed the title
Identify if incoming STATE message is legacy or per-stream
Transform incoming LEGACY and per-stream state JSON to AirbyteStateMessage
Aug 28, 2022
brianjlai
changed the title
Transform incoming LEGACY and per-stream state JSON to AirbyteStateMessage
[per-stream cdk] Transform incoming LEGACY and per-stream state JSON to AirbyteStateMessage
Aug 29, 2022
During a sync, connectors receive stream state from the platform by a reference to input_state.json. This JSON blob contains the state of all streams in the following
LEGACY
format:For connections supporting
STREAM
state, the platform will begin persisting state and passing it to a connector as:When we read the JSON file in
entrypoint.py
/source.py
, we should accept either format and parse it into a list of AirbyteStateMessage(s) which will then be interpreted by abstract_source.py later while processing an incremental sync.After the changes:
LEGACY state should be parsed into a list of one AirbyteStateMessage:
STREAM/GLOBAL state should be parsed into a list in the same format that in came in as.
This ticket will also involve defining a class
ConnectorState
which will help make future tickets more parallelizable.The text was updated successfully, but these errors were encountered: