Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix stream read given stream doesn't have any slice #28746

Merged
merged 3 commits into from
Jul 27, 2023

Conversation

maxi297
Copy link
Contributor

@maxi297 maxi297 commented Jul 26, 2023

What

Resolving message grouper error where there is not request performed for the stream (outside of auxiliary requests)

How

Remove the validation as it shows that the assumption that we would always have a request is false.

🚨 User Impact 🚨

Once merged in the Connector Builder project, it will stop failing if the substream partition router does not a return a slice

@maxi297 maxi297 requested a review from a team as a code owner July 26, 2023 19:20
@maxi297 maxi297 requested a review from girarda July 26, 2023 19:20
@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label Jul 26, 2023
Copy link
Contributor

@girarda girarda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

what's the error message we want to show here? something along the lines of "No requests were submitted. Check the configuration of the partition routers"?

@maxi297
Copy link
Contributor Author

maxi297 commented Jul 26, 2023

Currently, this is not considered to be an error by the CDK as it will only ignore this loop. Should this be an error in the Connector Builder?

@maxi297
Copy link
Contributor Author

maxi297 commented Jul 27, 2023

I've asked the frontend team if they could show a warning when we have slices == []. If this is good, I'll merge with the fix where we return

{
    "type": "RECORD",
    "record": {
        "stream": "List Events",
        "data": {
            "logs": [],
            "slices": [],
            "test_read_limit_reached": false,
            "auxiliary_requests": [...],
            "inferred_schema": null,
            "inferred_datetime_formats": {},
            "latest_config_update": null
        },
        "emitted_at": 1690399116000
    }
}

instead of

{
    "type": "RECORD",
    "record": {
        "stream": "List Events",
        "data": {
            "logs": [],
            "slices": [
                {
                    "pages": [
                        {
                            "records": [],
                            "request": null,
                            "response": null
                        }
                    ],
                    "slice_descriptor": null,
                    "state": null
                }
            ],
            "test_read_limit_reached": false,
            "auxiliary_requests": [...],
            "inferred_schema": null,
            "inferred_datetime_formats": {},
            "latest_config_update": null
        },
        "emitted_at": 1690399116000
    }
}

@maxi297 maxi297 merged commit 48bf520 into master Jul 27, 2023
@maxi297 maxi297 deleted the maxi297/fix-sync-without-http-request branch July 27, 2023 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CDK Connector Development Kit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants