You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
We have tenants that have tens of terrabytes of data per day. Building blooms for these tenants require a lot of resource, but each of bloom_build components is only able to utilise one cpu core as there is no parallelization in place. So the only way to overcome this issue is to deploy a huge number of small pods with one core cpu to satisfy the needs. But since we use Simple Scalable Deployment mode, scaling out bloom_build means scaling out backend component which makes the issue more complex and annoying.
Describe the solution you'd like
Having multiple workers per bloom_build to process planned tasks.
Describe alternatives you've considered
Scaling out with small pods which means 100 pods for 100 cores which is unpleasant.
The text was updated successfully, but these errors were encountered:
emadolsky
changed the title
Implement parallelization on bloom_build component
Implement parallelization in bloom_build component
Dec 19, 2024
Is your feature request related to a problem? Please describe.
We have tenants that have tens of terrabytes of data per day. Building blooms for these tenants require a lot of resource, but each of bloom_build components is only able to utilise one cpu core as there is no parallelization in place. So the only way to overcome this issue is to deploy a huge number of small pods with one core cpu to satisfy the needs. But since we use Simple Scalable Deployment mode, scaling out bloom_build means scaling out backend component which makes the issue more complex and annoying.
Describe the solution you'd like
Having multiple workers per bloom_build to process planned tasks.
Describe alternatives you've considered
Scaling out with small pods which means 100 pods for 100 cores which is unpleasant.
The text was updated successfully, but these errors were encountered: