Implement quota tracking options per ObjectStore.

[jmchilton]

Builds on #10212.

Overview

#6552 implemented the ability for admins to assign job outputs to different object stores at runtime (this could take into account tool/workflow injected parameters or just be based on user, tool, destination, cluster state, etc..). But all the stored data would consume the same quota - regardless of the source selected.

This pull request allows different object stores or different groups of object stores to have different quotas or no quota at all. This enables uses cases such as sending job to cheaper data when a user's quota is getting near full or allowing admin to setup tool and/of workflow parameters to send job outputs higher quality, more redundant storage based on user selected options or user preferences.

This is a substantial step forward toward allowing scratch-space histories, while I suspect we want to implement some higher level convince functions and interface around that (per history preferences, object store preferences types) - I think that would all be based on these abstractions - abstractions that allow even more flexibility for admins who require it.

Implementation

This adds the quota tag to XML/YAML object store declarations - that allow specifying a "quota source label" for each objectstore in a nested objectstore or disabling quota all together on objectstores.

The following quota block would assign all this storage to a quota source labelled with s3.

        <backend id="dynamic_s3" type="disk" weight="0">
            <quota source="s3" />
            <files_dir path="${temp_directory}/files_dynamic_s3"/>

Whereas this would disable quota usage for this object store altogether.

        <backend id="temp_disk" type="disk" weight="0">
            <quota enabled="false" />
            <files_dir path="${temp_directory}/files_cloud_scratch"/>

In order to implement this a new table/model has been added to track a user's usage per quota source label - namely UserQuotaSourceUsage. Object stores that did not have a source label are still tracked using the User model's disk_usage attribute. I've updated all the scripts that recalculate user usage.

UI + API

The quota dialog adds the option to pick a quota source label from those defined on the object stores, though this option only appears if quota source labels are configured.

Likewise, by default the quota meter is unaffected but when multiple quota source labels are configured the meter becomes a link that shows the usage of each quota source.

A new API /api/users/<user_id|current>/usage enables this.

Abstractions for #4840

While this PR adds significant complexity related to recalculating a User's quota - it does reduce the duplication, adds tests (made more useful by having fewer paths through the quota recalculation code), and bring object store information into the calculation. I think this is all stuff that would be needed for #4840 and currently missing.

Part of this establishes a pattern for how to exclude certain datasets from usage calculation both when it is being added (included in #4840) and when re-calculdated (not included in #4840).

The API endpoints for disk usage across object stores and the UI entry point for displaying that information will hopefully both enable a more robust implementation of #4840.

[jmchilton]

In long conversation with @natefoo and @mvdbeek we decided this needs to go a bit further at least before being rolled out on to main.

• Longer term we need to have the ability to copy from one objectstore to another asynchronously, but until that is ready there are certain copies that are effectively just changing the object_store_id on a dataset and those should be implemented - with quota recalculation, a UI, etc...
• Histories need to be filterable by datasets in a given objectstore - so users can see data scheduled for deletion. This can just piggyback on existing UI filtering plumbing.

I'd also love a little summary of objectstore, usage, etc.. within a history - perhaps using disk usage per dataset widget Dannon demo'd years ago (@dannon do you have a link to that sitting in a branch somewhere?) - but that might be something that should be an iteration 2 type of thing.

[dannon]

@jmchilton I'll see if I can dig it up -- I know I have it somewhere and it'd be great for that to see use somewhere.