Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Simplify request queue implementation #653

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

janbuchar
Copy link
Collaborator

Follow up on the points raised in apify/crawlee#2700

@janbuchar janbuchar added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 5, 2024
@github-actions github-actions bot added this to the 102nd sprint - Tooling team milestone Nov 5, 2024
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

@janbuchar janbuchar added the adhoc Ad-hoc unplanned task added during the sprint. label Nov 6, 2024
@B4nan
Copy link
Member

B4nan commented Nov 6, 2024

I was hoping we would first address this in the JS version where its actually costing us money, do we have some reported problems on the python side too?

@janbuchar
Copy link
Collaborator Author

I was hoping we would first address this in the JS version where its actually costing us money, do we have some reported problems on the python side too?

I get your angle, but I wanted to try it on code where the implementation isn't scattered between RequestProvider and RequestQueue. Plus, yeah, actors also get stuck with the Python SDK and the request queue is a prime suspect.

@B4nan
Copy link
Member

B4nan commented Nov 6, 2024

If that abstraction is problematic I am fine with removing it and leaving the base class acting like an interface mostly (we can't just make it one as that would be technically breaking), duplication is not a huge deal if we plan to drop the RQ v1 implementation anyway - but first we need to make sure v2 is working fine, we still get peeps from delivery moving back to v1 to deal with those weird issues...

@janbuchar
Copy link
Collaborator Author

If that abstraction is problematic I am fine with removing it and leaving the base class acting like an interface mostly (we can't just make it one as that would be technically breaking), duplication is not a huge deal if we plan to drop the RQ v1 implementation anyway - but first we need to make sure v2 is working fine, we still get peeps from delivery moving back to v1 to deal with those weird issues...

...and all of the above is why I chose to validate the points by Kuba in the python version 😄

@B4nan
Copy link
Member

B4nan commented Nov 6, 2024

My issue is that not many people use the python version, and those issues reported by delivery are quite random, so hard to confirm something helps here. Maybe it will resolve the issue we have a repro for here, that's great, but it might be a different one than those where we actually bleed money (we usually refund people with stuck runs completely).

I am fine doing it first here, but let's not wait for some confirmation before trying to fix it in the JS version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants