Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have StateBuilder return our actual state object and not simply a dict #34625

Merged
merged 2 commits into from
Jan 30, 2024

Conversation

maxi297
Copy link
Contributor

@maxi297 maxi297 commented Jan 29, 2024

What

Have the state builder return our action TState domain object and not the internal state[X].stream. stream_state. Without this, all the integration tests/scenario based test needs to know how are state messages created which is flaky as if the implementation change, all those tests would have to be updated while if it changes and we use StateBuilder, we might have a change to make the change non-breaking. This is how it looks for both:

Scenario-based

https://github.com/airbytehq/airbyte/compare/maxi297/statebuilder-create-state-not-dict?expand=1#diff-7a5029caa174d3c73bca535ec76e6ea66b577aef2149b2cac27baaf53d72aac7L42-R45

Integration tests

This is what triggered the change as we would only have to pass the state to the entrypoint wrapper that would save this list/dicts as json. However, adding the state as a parameter of SourceStripe.init would make the integration tests handle two types of state: the one for the entrypoint wrapper and the one that is returned by .read_state which is not a List[AirbyteStateMessage]. So usage changes to:

SourceStripe(catalog, config, StateBuilder(). with_stream_state(_STREAM_NAME, state_dict).build())

... instead of

state = [
    {
        "type": "STREAM",
        "stream": {
            "stream_state": state_dict,
            "stream_descriptor": {"name": _STREAM_NAME},
        },
    }
]
SourceStripe(catalog, config, state)

🚨 User Impact 🚨

Scenario based tests were updated as part of this PR. As for integration tests, they will be updated as part of #32057. This might impact #33996 (comment) though @askarpets

@maxi297 maxi297 requested a review from a team as a code owner January 29, 2024 19:17
Copy link

vercel bot commented Jan 29, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Jan 29, 2024 7:29pm

@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label Jan 29, 2024
@maxi297 maxi297 requested review from clnoll and askarpets January 29, 2024 19:17
Copy link
Contributor

@girarda girarda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢

@maxi297 maxi297 merged commit ca8590e into master Jan 30, 2024
21 checks passed
@maxi297 maxi297 deleted the maxi297/statebuilder-create-state-not-dict branch January 30, 2024 13:46
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 21, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CDK Connector Development Kit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants