API call for total size of specified dataverse

[CCMumma]

We need an API call to exist that currently does not: total size of a specified dataverse

The use case for us is that the Texas Data Repository hosts multiple Institutional Dataverses and I need a simple way to determine the size of all of the content, published and unpublished, in their entire dataverse.

[pdurbin]

@CCMumma thanks for creating this issue. The following issues are related:

• Storage metering: Allow tracking and limiting total and individual file upload space per user/group. Storage metering: Allow tracking and limiting total and individual file upload space per user/group. #938
• File and dataset limits: Add a programmatic way to limit file size and dataset size File and dataset limits: Add a programmatic way to limit file size and dataset size #3939
• Storage allocation quota Storage allocation quota per user #4339

[djbrooke]

@CCMumma - since this is concerning storage space, are you only worried about files? Or metadata as well?

[djbrooke]

In sprint planning I offered to link back to the original message:

https://groups.google.com/forum/#!searchin/dataverse-community/courtney%7Csort:date/dataverse-community/OYzY4P2Bvfw/Eus-FULwAgAJ

[CCMumma]

We are more interested in content storage, but including metadata, or having a sep. call for md, would also be valuable.

[djbrooke]

Thanks @CCMumma !

[landreev]

@CCMumma Hi, I've put together a new API call for reporting the total file storage size.
Just wanted to run it by you before I move it along.
The API will output the sum of all the file sizes uploaded by users, published and unpublished; for tabular data files, it counts both the size of the file uploaded by the user, and the size of the archival version (the tab-delimited file that we generate on ingest). These files are what we consider the real "payload", the archival content of the dataverse. (it will count the files in the dataverse specified, and in all its sub-dataverses recursively).

However, if you go to the filesystem where the datasets are stored and add the sizes of all the files found there, you'll end up with a larger bytes number. This is because we also cache some extra files generated as the datasets are being served: resized thumbnail-size copies of image files; metadata exports for published datasets, etc. The logic behind not counting these files is that they are generated on top of the archival content; they can be erased, and the system will regenerate them automatically.

Is this ok for your purposes?

[CCMumma]

Thank you so much - that's excellent work. The total size of the 'archival payload' would be a good start and it's wise to include published and unpublished in the number.

The 'total storage used' (including generated files and metadata) by a dataverse would also be valuable for instances, like ours in Texas, where we're trying to create a service model that sets fees based on storage used above a set maximum per institutional dataverse.

[landreev]

@CCMumma Thank you.
As implemented, the API will report the total size without the cached sizes by default; but will include everything, exports and all, if an optional includeCached argument is supplied.
I just wasn't sure whether to include that optional mode; if it was needed/useful at all. But it sounds like it could be potentially useful to you - so I am including and documenting it.

[CCMumma]

That is fantastic news. Thank you for your work.