You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm implementing a connector from a REST API to Snowflake and I found what seems to be a bug
I'm calling a first endpoint /workflows which returns a list of workflows. This endpoint returns a single page.
Then I call a second endpoint /workflows/{workflow_uid}/jobs which returns for each workflow a list of jobs. This endpoint is paginated with a page param.
Let's say I have two workflows (workflow_uid_1 & workflow_uid_2).
dlt starts by doing all these requests:
/workflows/workflow_uid_1/jobs?page=0
/workflows/workflow_uid_1/jobs?page=1
/workflows/workflow_uid_1/jobs?page=2
/workflows/workflow_uid_1/jobs?page=3
until the page is empty, which is the intended behavior.
Then dlt switches to workflow_uid_2 but starts at page 3:
/workflows/workflow_uid_2/jobs?page=3
/workflows/workflow_uid_2/jobs?page=4
Expected behavior
The requests made to /workflows/workflow_uid_2/jobs should start with page=0
Steps to reproduce
See description
Operating system
macOS
Runtime environment
Local
Python version
3.11
dlt data source
rest_api
dlt destination
Snowflake
Other deployment details
No response
Additional information
The issue can be fixed by adding two lines of code in the init_request of RangePaginator:
dlt version
1.1.0
Describe the problem
I'm implementing a connector from a REST API to Snowflake and I found what seems to be a bug
I'm calling a first endpoint /workflows which returns a list of workflows. This endpoint returns a single page.
Then I call a second endpoint /workflows/{workflow_uid}/jobs which returns for each workflow a list of jobs. This endpoint is paginated with a page param.
Let's say I have two workflows (workflow_uid_1 & workflow_uid_2).
dlt starts by doing all these requests:
/workflows/workflow_uid_1/jobs?page=0
/workflows/workflow_uid_1/jobs?page=1
/workflows/workflow_uid_1/jobs?page=2
/workflows/workflow_uid_1/jobs?page=3
until the page is empty, which is the intended behavior.
Then dlt switches to workflow_uid_2 but starts at page 3:
/workflows/workflow_uid_2/jobs?page=3
/workflows/workflow_uid_2/jobs?page=4
Expected behavior
The requests made to
/workflows/workflow_uid_2/jobs
should start withpage=0
Steps to reproduce
See description
Operating system
macOS
Runtime environment
Local
Python version
3.11
dlt data source
rest_api
dlt destination
Snowflake
Other deployment details
No response
Additional information
The issue can be fixed by adding two lines of code in the
init_request
ofRangePaginator
:This way, the paginator is reset every time it runs through a new parent resource
The text was updated successfully, but these errors were encountered: