Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Datastores can return large numbers runs that are processing synchronously #1858

Closed
johnaohara opened this issue Jul 3, 2024 · 0 comments · Fixed by #1859
Closed

Datastores can return large numbers runs that are processing synchronously #1858

johnaohara opened this issue Jul 3, 2024 · 0 comments · Fixed by #1859
Assignees
@johnaohara
Labels
area/backend branch/master The master branch branch/stable The stable branch type/enhancement An enhancement to an existing feature

Comments

@johnaohara
Copy link
Member

Describe the bug

Some backend datastore integrations can return a large number of runs.

This can cause issues with importing a lot of run data in one query, such as long transaction times, failure to process a single run data will fail the entire import etc.

The processing of run data should be performed asynchronously in these cases and runs should be offloaded to an external queue.

See #1703 for an example datastore that can return a large number of run results
johnaohara added a commit to johnaohara/Horreum that referenced this issue Jul 3, 2024
@johnaohara
Offload multi run upload processing to message queue: Fixes Hyperfoil…
725e1af 
…#1858
@johnaohara johnaohara mentioned this issue Jul 3, 2024
Merged
@johnaohara johnaohara added type/enhancement An enhancement to an existing feature branch/master The master branch branch/stable The stable branch area/backend labels Jul 3, 2024
johnaohara added a commit to johnaohara/Horreum that referenced this issue Jul 4, 2024
@johnaohara
Offload multi run upload processing to message queue: Fixes Hyperfoil…
ae27c23 
…#1858
@johnaohara johnaohara self-assigned this Jul 9, 2024
johnaohara added a commit to johnaohara/Horreum that referenced this issue Jul 24, 2024
@johnaohara
Offload multi run upload processing to message queue: Fixes Hyperfoil…
43de3c1 
…#1858
johnaohara added a commit to johnaohara/Horreum that referenced this issue Jul 25, 2024
@johnaohara
Offload multi run upload processing to message queue: Fixes Hyperfoil…
f1678ee 
…#1858
johnaohara added a commit to johnaohara/Horreum that referenced this issue Sep 5, 2024
@johnaohara
Offload multi run upload processing to message queue: Fixes Hyperfoil…
dfad5d8 
…#1858
johnaohara added a commit that referenced this issue Sep 5, 2024
@johnaohara
Offload multi run upload processing to message queue: Fixes #1858
9818f54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@johnaohara johnaohara

Labels
area/backend branch/master The master branch branch/stable The stable branch type/enhancement An enhancement to an existing feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Offload multi run upload processing to message queue: Fixes #1858 johnaohara/Horreum
1 participant
@johnaohara