File and dataset limits: Add a programmatic way to limit file size and dataset size

[CCMumma]

TDR has set file (2gb) and dataset (10gb) size limits for the time being, but we can only enforce them manually. It would be good if setting limits were programmatic, set by admins. I am interested in knowing whether others are interested in this functionality and hope it can make its way to the roadmap at some point. We can probably commit some resources to helping out with use cases, docs, code and/or review.

[pdurbin]

@CCMumma thanks for the suggestion. I think I'm being tripped up by the word "programmatic", which to me means that a sysadmin can change the settings using a script written in Python or whatever. This is already possible.

Currently, all limits affect the entire installation of Dataverse. Are you saying that you'd like limits to apply to sub-dataverses? That is, that https://dataverse.tdl.org/dataverse/utexas and https://dataverse.tdl.org/dataverse/tamu might have different limits?

[scolapasta]

I discussed with @CCMumma at the Community Meeting. I didn't realize she wasn't aware of the file limiting that is available, but more generally we discussed dataset limits. This is currently unavailable - i.e you could upload as many 2gb files as you'd like to one dataset.

I'm fairly certain that is what is being asked for here, but @CCMumma can confirm. :)

[CCMumma]

It is, thanks... and specifically having the ability to set the limits in the GUI. Sorry, I wasn't aware of the python file limiting capability.

[pdurbin]

To help highlight the fact that you can set limits, perhaps we should plug this in the new "Going Live: Launching Your Production Deployment" section I recently added to the Installation Guide. For now, you can preview it here: https://github.com/IQSS/dataverse/blob/9708f57ad970abb2b068e08d926dc9c318bdd8aa/doc/sphinx-guides/source/installation/config.rst#going-live-launching-your-production-deployment

#938 is about setting storage quotas per user or group. Dataverse doesn't currently support any concept of storage quotas at any level (user, group, dataverse, dataset, etc.).

[pdurbin]

Related: #4339

[pdurbin]

Related:

• Limit the amount of data depositors may upload every day #7829
• Add mechanism for collection-wise storage size quotas #8549

Also, thank you @CCMumma for bringing quotas up at the community call this week! https://docs.google.com/document/d/15yUGslKMUr4QCxGFuEsRyd2FdRoPWv3OzxZcVe5yEbs/edit?usp=sharing

[pdurbin]

I would keep an eye on this:

• Add mechanism for collection-wise storage size quotas #8549

[pdurbin]

Just merged:

• Collection Quotas #10144

[cmbz]

2024/09/09: Close, currently addressed with collection size limiting functionality.