-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement quota tracking options per ObjectStore. #14047
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jmchilton
force-pushed
the
quota_4
branch
2 times, most recently
from
June 11, 2022 16:32
dc187ac
to
d9eb2cf
Compare
3 tasks
jmchilton
force-pushed
the
quota_4
branch
8 times, most recently
from
June 20, 2022 16:24
ed3b4c3
to
b1665a7
Compare
jmchilton
changed the title
[WIP] Implement quota tracking options per ObjectStore.
Implement quota tracking options per ObjectStore.
Jun 20, 2022
davelopez
reviewed
Jul 5, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks awesome!
Just some minor comments below from my limited point of view 😅
client/src/components/ObjectStore/ObjectStoreRestrictionSpan.vue
Outdated
Show resolved
Hide resolved
client/src/components/User/DiskUsage/Quota/QuotaUsageProvider.js
Outdated
Show resolved
Hide resolved
This one needs a rebase unfortunately. |
jmchilton
force-pushed
the
quota_4
branch
3 times, most recently
from
February 14, 2023 17:10
3e5bfc2
to
b273ebf
Compare
Setup abstractions to prevent sharing transient or private per-user objects in an objectstore.
Merged as part of #14073 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Redoing #10221 / #10977 but building on #14044.
Overview
#6552 implemented the ability for admins to assign job outputs to different object stores at runtime (this could take into account tool/workflow injected parameters or just be based on user, tool, destination, cluster state, etc..). But all the stored data would consume the same quota - regardless of the source selected.
This pull request allows different object stores or different groups of object stores to have different quotas or no quota at all. This enables uses cases such as sending job to cheaper data when a user's quota is getting near full or allowing admin to setup tool and/of workflow parameters to send job outputs higher quality, more redundant storage based on user selected options or user preferences.
This is a substantial step forward toward allowing scratch-space histories, while I suspect we want to implement some higher level convince functions and interface around that (per history preferences, object store preferences types) - I think that would all be based on these abstractions - abstractions that allow even more flexibility for admins who require it.
Implementation
This adds the quota tag to XML/YAML object store declarations - that allow specifying a "quota source label" for each objectstore in a nested objectstore or disabling quota all together on objectstores.
The following quota block would assign all this storage to a quota source labelled with
s3
.Whereas this would disable quota usage for this object store altogether.
In order to implement this a new table/model has been added to track a user's usage per quota source label - namely
UserQuotaSourceUsage
. Object stores that did not have a source label are still tracked using the User model'sdisk_usage
attribute. I've updated all the scripts that recalculate user usage.UI + API
The quota dialog adds the option to pick a quota source label from those defined on the object stores, though this option only appears if quota source labels are configured.
Likewise, by default the quota meter is unaffected but when multiple quota source labels are configured the meter becomes a link that shows the usage of each quota source.
A new API
/api/users/<user_id|current>/usage
enables this.Abstractions for #4840
While this PR adds significant complexity related to recalculating a User's quota - it does reduce the duplication, adds tests (made more useful by having fewer paths through the quota recalculation code), and bring object store information into the calculation. I think this is all stuff that would be needed for #4840 and currently missing.
Part of this establishes a pattern for how to exclude certain datasets from usage calculation both when it is being added (included in #4840) and when re-calculdated (not included in #4840).
The API endpoints for disk usage across object stores and the UI entry point for displaying that information will hopefully both enable a more robust implementation of #4840.
How to test the changes?
(Select all options that apply)
License