-
Notifications
You must be signed in to change notification settings - Fork 164
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[rest_api source] can't detect pagination for github api and poke api #1915
Comments
Thank your for the issue @AstrakhantsevaAA, I believe the |
@burnash yeah, I think you are right, it's not clear from the warning message. Anyway this part of tutorial should be adjusted, by default we can't run this pipeline, because of rate limits, I think we can reduce the amount of data for issues endpoints:
And these warning scares our new users :D can we log this warning in the beginning not for each request? |
dlt version
1.1.0
Describe the problem
The rest_api source cannot autodetect pagination for github api and poke api, this impacts our tutorial.
If you run this pipeline, you will get a list of
Fallback paginator
warnings for both: github api and poke api.This fallback also causes a rate limiting error, the rest_api source continually requests github api until the error occurs.
Expected behavior
according to our tutorial, rest_api source should automatically detect such simple types of pagination.
Steps to reproduce
Operating system
macOS
Runtime environment
Local
Python version
3.11
dlt data source
No response
dlt destination
No response
Other deployment details
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: