Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use cursor based pagination #64

Merged
merged 18 commits into from
Sep 14, 2021
Merged

Conversation

cosimon
Copy link
Contributor

@cosimon cosimon commented Sep 9, 2021

Description of change

Changes in this PR:

  • Add to setup.py a "test" extra_requires so that CI doesn't have to install ipdb
  • Add to setup.py dependencies on backoff and requests
  • Add retry logic to the requests made for the Organizations, Satisfaction Ratings, Ticket Fields, Tags, and GroupMemberships streams
  • Add unit tests around our new cursor based pagination logic
  • Update singer-python from 5.2.1 to 5.12.2
  • Update zenpy from 2.0.0 to 2.0.24
  • Update the Organizations, Satisfaction Ratings, Ticket Fields, Tags, GroupMemberships, and Tickets to use cursor-based pagination
    • Tickets leans on zenpy to do pagination
    • Users, Tags, and GroupMemberships do not
  • Fix Circle Config to install "test" dependencies instead of "dev"

Manual QA steps

  • Ran the tap

Risks

Rollback steps

  • revert this branch

@cosimon
Copy link
Contributor Author

cosimon commented Sep 14, 2021

@luandy64 Should probably update the PR text to more accurately describe whats happening now.

@cosimon
Copy link
Contributor Author

cosimon commented Sep 14, 2021

LGTM, but I cannot approve as I opened the PR.

@luandy64 luandy64 merged commit c5cee72 into master Sep 14, 2021
@luandy64 luandy64 deleted the feature/use-cursor-based-pagination branch September 14, 2021 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants