ownCloud remaining Quota bug

[Y4PHI]

Describe the bug

We have the problem that if we set the quota above 884.2GB, only 884.2GB is displayed with the remaining quota.

Expected behavior

If we set the total quota to 2.5TB as desired, only 884.2GB will be displayed with the available quota. If I set the total quota below this 884.2GB, it is displayed correctly. We have over 40TB of storage available

[rhafer]

@Y4PHI which ocis version are you using, also which architecture are you running on? Where are you setting that quota? I just tried a few things myself, but I was not able to reproduce the problem.

[micbar]

If we set the total quota to 2.5TB as desired, only 884.2GB will be displayed with the available quota. If I set the total quota below this 884.2GB, it is displayed correctly. We have over 40TB of storage available

Please note that the available quota is adjusted when the disk space is not enough. The ocis process in the storage-users service does a syscall to the disk and asks for the remaining space.

[micbar]

@Y4PHI Is this still an issue? We got no reaction so far.

[Y4PHI]

That whole thing is on a Minio (S3) storage backend and the bucket has no limit.
The ocis version is 5.0.4

[Y4PHI]

If I create the space and then change the quota to 2.5TB and then click on Change Quota again, it doesn't say 2.5TB but unlimited. Otherwise it doesn't matter what I call the space, the error is the same

[micbar]

Can you post the response of the /me/drives call?

[Y4PHI]

The system is provisioned with a dedicated storage capacity of 32 terabytes. It utilizes Minio, an S3-compatible object storage service, configured to support an unlimited bucket size, meaning there is no predefined limit on the amount of data that can be stored within a single bucket. There is a discrepancy in the quota calculations displayed in the web interface, although the client application appears to function correctly and displays the expected quota values. When new files are added to the storage, the total quota displayed is altered, which is incorrect because the quota should remain consistent regardless of file insertions. The underlying issue might be related to errors in the system's handling of quota calculations when dealing with unlimited bucket sizes, causing the incorrect quota display and the changes observed when inserting files.

[Y4PHI]

Here is the response of the /me/drives. The WebDAV URL has been removed and the IDs have been modified

'''{
"value": [
{
"driveAlias": "project/einkauf",
"driveType": "project",
"id": "766e4e1c-7a7f-4391-b56e-46cafbchfhf56456444dh4rhhca7eb$4fa58964-c444-4d3d-bb08-ddf3ec31f594",
"lastModifiedDateTime": "2024-07-05T06:41:24.455113065Z",
"name": "Einkauf",
"owner": {
"user": {
"displayName": "",
"id": "4fa58964-c444-4d3d-bb08-ddf3ecnfnfn7722004"
}
},
"quota": {
"remaining": 881811755008,
"state": "normal",
"total": 0,
"used": 65
},
"root": {
"eTag": ""8be69f17703f275bd9445445b2688c86f"",
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$4fa58964-c444-4d3d-bb08-ddf3ec31f594",
"permissions": [
{
"grantedToIdentities": [
{
"group": {
"displayName": "Owncloud-Admin",
"id": "1d617185-05c7-48f0-b17c-6a15161875a"
}
}
],
"roles": [
"manager"
]
}
],
"webDavUrl": "
},
"special": [
{
"eTag": ""77b0d84305059eba53d154614615c3a09"",
"file": {
"mimeType": "text/markdown"
},
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$4fa58964-c444-4d3d-bb08-ddf3ec31f594!b4464646466-2714-4201-9836-4f84daf24d27",
"lastModifiedDateTime": "2024-07-05T06:30:27.390361747Z",
"name": "readme.md",
"parentReference": {
"driveId": "766e4e1c-7a7f-4391-b56e-46cafbcc6466464456145$4fa58964-c444-4d3d-bb08-ddf3ec31f594",
"driveType": "project",
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$4fa58964-c444-4d3d-bb08-ddf3ec3156drgsdxg26166156161116f594!832198f8-db33-47b0-bb92-2946e6f8fc1c",
"name": ".space",
"path": "/.space"
},
"size": 65,
"specialFolder": {
"name": "readme"
},
"webDavUrl": "
],
"webUrl":
},
{
"driveAlias": "project/marketing",
"driveType": "project",
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca715661jfgjvgjkfgkjeb$863b09c2-2d16-4fff-a64c-7000673a764a",
"lastModifiedDateTime": "2024-07-03T15:51:03.627901232Z",
"name": "Marketing",
"owner": {
"user": {
"displayName": "",
"id": "863b09c2-2d16-4fff-a64c-700hyvasv46546464a"
}
},
"quota": {
"remaining": 881811755008,
"state": "normal",
"total": 0,
"used": 1508688649
},
"root": {
"eTag": ""e4938cd9524893a3dbcxvydfvbfcbv99a9496323646"",
"id": "766e4e1c-7a7f-4391-b56e-46cafbybcbxcvydfcca7eb$863b09c2-2d16-4fff-a64c-7000673a764a",
"permissions": [
{
"grantedToIdentities": [
{
"group": {
"displayName": "Owncloud-Admin",
"id": "1d617185-05c7-48f0-b17c-6axdfvbx14545e84a1b875a"
}
}
],
"roles": [
"manager"
]
},
{
"grantedToIdentities": [
{
"group": {
"displayName": "Marketing",
"id": "e15347c1-5872-4fbf-a286-ea5d4645665a"
}
}
],
"roles": [
"editor"
]
}
],
"webDavUrl": " },
"special": [
{
"eTag": ""b3e0a2c4a768ad7e07d39c1bf2095336"",
"file": {
"mimeType": "text/markdown"
},
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$863b09c2-2d16-4fff-a64c-7000673adgdthtcdghsg764a!eb24131f-9604-4657-9adb-b7fafd89c4b4",
"lastModifiedDateTime": "2024-05-23T11:19:28.948599058Z",
"name": "readme.md",
"parentReference": {
"driveId": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$863b09c2-2d16-4fff-a64c-70dgvyfvy00673a764a",
"driveType": "project",
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$863b09c2-2d16-4fff-a64c-7000673a764a!81d8292d-98ae-4741-81b0-3b32vdyvydvydfvbd56a8e7",
"name": ".space",
"path": "/.space"
},
"size": 65,
"specialFolder": {
"name": "readme"
},
"webDavUrl": ""
}
],
"webUrl": ""
},
{
"driveAlias": "project/abc-archiv",
"driveType": "project",
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$567f58ac-eb88-4393-bfb7-00acaehsdgzdkddhfczh229c0a",
"lastModifiedDateTime": "2024-07-08T19:02:36.515236428Z",
"name": "Abc-Archiv",
"owner": {
"user": {
"displayName": "",
"id": "567f58ac-eb88-4393-bfb7-00acdhehxh45654ae229c0a"
}
},
"quota": {
"remaining": 881811755008,
"state": "normal",
"total": 3500000000000,
"used": 90080719206
},
"root": {
"eTag": ""b0296538e5eeccf7cff94e20855l46546jhncfc76e8aaa"",
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$567f58ac-eb88-4393-bfb7-00acadhdfd656468647e229c0a",
"permissions": [
{
"grantedToIdentities": [
{
"group": {
"displayName": "Owncloud-Admin",
"id": "1d617185-05c7-48f0-b17c-6ae84kbhb454646a1b875a"
}
}
],
"roles": [
"manager"
]
},
{
"grantedToIdentities": [
{
"group": {
"displayName": "abf-Archiv",
"id": "72921652-b2a5-4970-8fa2-2634216546futfu33dc889af"
}
}
],
"roles": [
"editor"
]
}
],
"webDavUrl": ""
},
"special": [
{
"eTag": ""529c0f0bb424f2af973cecf2a2buj8f62183"",
"file": {
"mimeType": "text/markdown"
},
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$567f58ac-eb88-4393-bfb7-00acae229c0a!e67c3bfa-a440-479e-b330-61fc02b5468d",
"lastModifiedDateTime": "2024-07-05T12:22:59.9855632Z",
"name": "readme.md",
"parentReference": {
"driveId": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$567f58ac-eb88-4393-bfb7-00acae229c0a",
"driveType": "project",
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$567f58ac-eb88-4393-bfb7-00acae229c0a!b07c422c-7b23-4ff0-8ccf-201de2416fc8",
"name": ".space",
"path": "/.space"
},
"size": 36,
"specialFolder": {
"name": "readme"
},
"webDavUrl": ""
}
],
"webUrl": ""
},
{
"driveAlias": "project/vertrieb",
"driveType": "project",
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$df896d30-0118-4430-b0db-479048fb35ab",
"lastModifiedDateTime": "2024-07-05T12:16:21.001413606Z",
"name": "Vertrieb",
"owner": {
"user": {
"displayName": "",
"id": "df896d30-0118-4430-b0db-479048fb35ab"
}
},
"quota": {
"remaining": 881811755008,
"state": "normal",
"total": 0,
"used": 65
},
"root": {
"eTag": ""835e8baf890e6dd691dedf3a3a9ad345"",
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$df896d30-0118-4430-b0db-479048fb35ab",
"permissions": [
{
"grantedToIdentities": [
{
"group": {
"displayName": "OwncloudAdmin",
"id": "1d617185-05c7-48f0-b17c-61161660078xfgbbxbxbdgae84a1b875a"
}
}
],
"roles": [
"manager"
]
}
],
"webDavUrl": ""
},d
"special": [
{
"eTag": ""e977e11c3a174021724e716ad0e6ysva51ae"",
"file": {
"mimeType": "text/markdown"
},
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$df896d30-0118-4430-b0db-479048fb3ngzcjngvn69645ab!ec303f93-25d5-4176-a5b5-8fd51d4b645f",
"lastModifiedDateTime": "2024-07-05T06:30:33.135539659Z",
"name": "readme.md",
"parentReference": {
"driveId": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$df896d30-0118-4430-b0db-476545cngn9048fb35ab",
"driveType": "project",
"id": "766e4e1c-7a7f-4391-b56e-46cafbcca7eb$df896d30-0118-4430-b0db-479048fb35ab!d71fxbdbhjkasbvk54648eeda-0318-4d87-bb09-67cbfedda494",
"name": ".space",
"path": "/.space"
},
"size": 65,
"specialFolder": {
"name": "readme"
},
"webDavUrl": ""
}
],
"webUrl": ""
}
]
}

[JonnyBDev]

Just go give some more information here. One of our teammembers is reporting issues while uploading to a space. We want to upload a 300gb file to one of our spaces.

Space configuration
Total quota: Unlimited
Used quota: 15,4 TB
Unused quota: 226,5 GB

This upload is now failing because of the incorrect calculated unused quota. Just a reminder, that this bucket (s3 storage backend) is a Minio instance with 80TB storage. It's far away from being full as of now.

[micbar]

Weird. Needs debugging.

[JonnyBDev]

Thanks! If you need help debugging this or need input from our side feel free to contact me.

[JonnyBDev]

Hi. I have some additional information that could be important. As I said we've a 1TB SSD where OCIS (Dockerized) is running on and a storage server (s3) with around 80TB.

We've now observed a possible correlation between the free disk space of the 1TB SSD and the available quota. We've started a upload at around 615GB available disk space (of the 1TB SSD). The quota was around 610-620 GB too. We see the available quota going down at the same time. We can only compare the quota in OCIS and the available disk space (of the 1TB SSD) in our monitoring which is getting refreshed every 60s. And they are around the same number + / - a current upload deviation.

So our guess is, that the "available quota" is in fact the available space on the disk, or the directory OCIS is storing its files and not the actual S3 storage behind it. That's legit, but a bit confusing.

[rhafer]

Hm, additionally to the fact that we're calculating the remain space base on the free disk space even when using an s3 based blobstore (which is obviously a bug), there is also a bit of discrepancy in the UI regarding the remaining space.

In the space list (when using the table view instead of the grid view) we see the remaining diskspace listed as the remaining quota which is at least misleading, when quota > remaining diskspace.

The the details view on the sidebar we the the quota usage calcuated based on the quota and actual space usage.

This is at least confusing. Also I am wondering if we really should expose the remaining diskspace at all, independent of which blobstore is being used. At least we should make a clear differentiation between how much quota is used and the amount of remaining diskspace.

[rhafer]

https://github.com/cs3org/reva/blob/edge/pkg/storage/utils/decomposedfs/spaces.go#L1031 🤷‍♂️

[rhafer]

@tbsbdr @JammingBen Reopening since there is still the issue with the consistency between the two views in web. See: #9245 (comment)

Should I open a web issue for that?

[rhafer]

I opened owncloud/web#11679 for web