Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve DISTINCT query for finding queue names in PostgreSQL #397

Closed
wants to merge 2 commits into from

Conversation

bubiche
Copy link

@bubiche bubiche commented Oct 29, 2024

workerB

Related issue in Mission Control: rails/mission_control-jobs#160

We use SolidQueue with PostgreSQL and sometimes viewing Mission Control slows the database down due to the SELECT DISTINCT query. The discussion in the linked issue suggests using recursive CTE, which seems to have better performance in practice for us.

This PR uses recursive CTE for PostgreSQL if it is supported for better performance.

@bubiche
Copy link
Author

bubiche commented Nov 18, 2024

Hi @rosa, sorry for pinging but it's been a while, is this something in the plans for Solid Queue? 1 use case other than Mission Control I can think of is when we need to pause queue(s) when the database/workers are overwhelmed, the query to get non-paused queues won't worsen the problem.

I'll close this PR if it's not something Solid Queue needs.

@rosa
Copy link
Member

rosa commented Nov 18, 2024

Hey @bubiche, sorry for the delay!

1 use case other than Mission Control I can think of is when we need to pause queue(s) when the database/workers are overwhelmed, the query to get non-paused queues won't worsen the problem.

This query is used only in Mission Control. For paused queues, the solid_queue_ready_executions table is checked, which should be much smaller than the solid_queue_jobs table. I wrote a bit about that here.

I'm interested in this, but the code here is very complicated and big. I'd like to have an alternative to this, something simpler or at least encapsulated differently, not in your face like this for only a specific database. I haven't spent enough time thinking how it'd be, sorry.

@bubiche bubiche closed this Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants