Posixfs trashbin #4779

aduffeck · 2024-07-26T06:51:51Z

Implement trashbin functionality for posixfs

update-docs · 2024-07-26T06:51:55Z

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

butonic · 2024-08-26T10:36:40Z

This is so awcesome!

When I delete a file the trishbin shows the deletion time as 1h in the future... hm what does the freedesktop org spec say about timezones ...

The key “DeletionDate” contains the date and time when the file/directory was trashed. The date and time are to be in the YYYY-MM-DDThh:mm:ss format (see RFC 3339). The time zone should be the user's (or filesystem's) local time. The value type for this key is “string”.

So ... we assume it to be the local time if no timezone is given.

There is a bug in the trash cli: sindresorhus/trash#56 and the pr to fix it specifically uses localtime: sindresorhus/trash#114 also pointing to the gnome implementation: https://gitlab.gnome.org/GNOME/glib/blob/glib-2-56/gio/glocalfile.c#L2195-2202

And there is a go tool implementing the spec: https://github.com/umlx5h/gtrash/blob/460e0b73615c653eee8941a3dd43fc1465260a3a/internal/xdg/trashinfo.go#L27

butonic · 2024-08-26T11:19:24Z

I tried restoring a folder with children

butonic · 2024-08-26T11:23:00Z

even after refreshing I still see foo twice ...

The propfind returns it twice:

<d:multistatus xmlns:s="http://sabredav.org/ns" xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
    <d:response>
        <d:href>/remote.php/dav/spaces/storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1/</d:href>
        <d:propstat>
            <d:prop>
                <oc:permissions>RDNVCKZP</oc:permissions>
                <oc:favorite>0</oc:favorite>
                <oc:fileid>
                    storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1!e2b2edae-0b58-420c-a340-1f979a358ba1</oc:fileid>
                <oc:file-parent>storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1</oc:file-parent>
                <oc:name>Admin</oc:name>
                <oc:owner-id>admin</oc:owner-id>
                <oc:owner-display-name>Admin</oc:owner-display-name>
                <oc:privatelink>
                    https://cloud.ocis.test/f/storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1%21e2b2edae-0b58-420c-a340-1f979a358ba1</oc:privatelink>
                <oc:size>0</oc:size>
                <d:getlastmodified>Mon, 26 Aug 2024 11:17:41 GMT</d:getlastmodified>
                <d:getetag>"caf680035d2b8ab19fe27a9c74883e56"</d:getetag>
                <d:resourcetype>
                    <d:collection />
                </d:resourcetype>
                <oc:tags></oc:tags>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
    <d:response>
        <d:href>/remote.php/dav/spaces/storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1/foo/</d:href>
        <d:propstat>
            <d:prop>
                <oc:permissions>RDNVCKZP</oc:permissions>
                <oc:favorite>0</oc:favorite>
                <oc:fileid>
                    storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1!aa244b8d-256d-4220-9acf-a05c6b25b8a3</oc:fileid>
                <oc:file-parent>
                    storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1!e2b2edae-0b58-420c-a340-1f979a358ba1</oc:file-parent>
                <oc:name>foo</oc:name>
                <oc:owner-id>admin</oc:owner-id>
                <oc:owner-display-name>Admin</oc:owner-display-name>
                <oc:privatelink>
                    https://cloud.ocis.test/f/storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1%21aa244b8d-256d-4220-9acf-a05c6b25b8a3</oc:privatelink>
                <oc:size>0</oc:size>
                <d:getlastmodified>Mon, 26 Aug 2024 10:35:11 GMT</d:getlastmodified>
                <d:getetag>"2a72cce05f0a2b0e54ae22a18e23cc03"</d:getetag>
                <d:resourcetype>
                    <d:collection />
                </d:resourcetype>
                <oc:tags></oc:tags>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
    <d:response>
        <d:href>/remote.php/dav/spaces/storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1/foo/</d:href>
        <d:propstat>
            <d:prop>
                <oc:permissions>RDNVCKZP</oc:permissions>
                <oc:favorite>0</oc:favorite>
                <oc:fileid>
                    storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1!aa244b8d-256d-4220-9acf-a05c6b25b8a3</oc:fileid>
                <oc:file-parent>
                    storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1!e2b2edae-0b58-420c-a340-1f979a358ba1</oc:file-parent>
                <oc:name>foo</oc:name>
                <oc:owner-id>admin</oc:owner-id>
                <oc:owner-display-name>Admin</oc:owner-display-name>
                <oc:privatelink>
                    https://cloud.ocis.test/f/storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1%21aa244b8d-256d-4220-9acf-a05c6b25b8a3</oc:privatelink>
                <oc:size>0</oc:size>
                <d:getlastmodified>Mon, 26 Aug 2024 10:35:11 GMT</d:getlastmodified>
                <d:getetag>"2a72cce05f0a2b0e54ae22a18e23cc03"</d:getetag>
                <d:resourcetype>
                    <d:collection />
                </d:resourcetype>
                <oc:tags></oc:tags>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
    <d:response>
        <d:href>
            /remote.php/dav/spaces/storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1/New%20folder/</d:href>
        <d:propstat>
            <d:prop>
                <oc:permissions>RDNVCKZP</oc:permissions>
                <oc:favorite>0</oc:favorite>
                <oc:fileid>
                    storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1!c986dd3f-da27-4a0c-8e7d-70d2f0a4bb8e</oc:fileid>
                <oc:file-parent>
                    storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1!e2b2edae-0b58-420c-a340-1f979a358ba1</oc:file-parent>
                <oc:name>New folder</oc:name>
                <oc:owner-id>admin</oc:owner-id>
                <oc:owner-display-name>Admin</oc:owner-display-name>
                <oc:privatelink>
                    https://cloud.ocis.test/f/storage-users-1$e2b2edae-0b58-420c-a340-1f979a358ba1%21c986dd3f-da27-4a0c-8e7d-70d2f0a4bb8e</oc:privatelink>
                <oc:size>0</oc:size>
                <d:getlastmodified>Mon, 26 Aug 2024 10:29:00 GMT</d:getlastmodified>
                <d:getetag>"c7d05d3e15d15bf59da4cb9e0ea4606b"</d:getetag>
                <d:resourcetype>
                    <d:collection />
                </d:resourcetype>
                <oc:tags></oc:tags>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
</d:multistatus>

hm ... but on disk the folder seems fine:

➜  reva git:(posixfs-trashbin) ls -lah /home/jfd/tmp/posix-storage/users/admin/       
insgesamt 36K
drwxr-x--- 5 jfd jfd 4,0K 26. Aug 13:17  .
drwxr-xr-x 3 jfd jfd 4,0K 26. Aug 13:17  ..
drwxr-xr-x 3 jfd jfd 4,0K 26. Aug 12:35  foo
drwx------ 2 jfd jfd 4,0K 26. Aug 12:29 'New folder'
drwxr-xr-x 4 jfd jfd 4,0K 26. Aug 13:16  .Trash

butonic · 2024-08-26T11:44:44Z

reproducer:

create /foo/bar/bif/123.txt (i did that on the cli)
trash /foo
restore /foo from the trash
list / -> see foo 4 times

aduffeck · 2024-09-10T07:15:25Z

@butonic hm, do you mind sharing your config? I can not reproduce the issue both with personal and projects spaces.

butonic · 2024-09-10T07:49:43Z

my config:

			{
				"name": "cloud.ocis.test posix",
				"type": "go",
				"request": "launch",
				//"buildFlags": "-race",
				"mode": "debug",
				"program": "${workspaceFolder:ocis}/ocis/cmd/ocis",
				"args": [ "server" ],
				"env": {
					// set insecure options because we don't have valid certificates in dev environments
					"OCIS_INSECURE": "true",
					// enable basic auth for dev setup so that we can use curl for testing
					"PROXY_ENABLE_BASIC_AUTH": "true",
					
					"OCIS_URL": "https://cloud.ocis.test",
					"OCIS_LOG_LEVEL": "error",
					"OCIS_LOG_PRETTY": "true",
					"OCIS_LOG_COLOR": "true",
					"NATS_LOG_LEVEL": "error",
			  	  
					
					// setup as per https://owncloud.dev/architecture/posixfs-storage-driver/#setup-configuration
					"STORAGE_USERS_DRIVER": "posix",
					"STORAGE_USERS_POSIX_ROOT":"/home/jfd/tmp/posix-storage",
					"STORAGE_USERS_POSIX_WATCH_TYPE":"inotifywait",
					"STORAGE_USERS_ID_CACHE_STORE":"nats-js-kv", // why does it have to be nats?
					"STORAGE_USERS_ID_CACHE_STORE_NODES":"localhost:9233",
					
					// Optionally enable gid based space access
					//export STORAGE_USERS_POSIX_USE_SPACE_GROUPS="true"   

					
					/*
					 * Keep secrets and passwords in one block to allow easy uncommenting
					 */
					// user id of "admin", for user creation and admin role assignement
					"OCIS_ADMIN_USER_ID": "some-admin-user-id-0000-000000000000", // FIXME currently must have the length of a UUID, see reva/pkg/storage/utils/decomposedfs/spaces.go:228
					// admin user default password
					"IDM_ADMIN_PASSWORD": "admin",
					// system user
					"OCIS_SYSTEM_USER_ID": "some-system-user-id-000-000000000000", // FIXME currently must have the length of a UUID, see reva/pkg/storage/utils/decomposedfs/spaces.go:228
					"OCIS_SYSTEM_USER_API_KEY": "some-system-user-machine-auth-api-key",
					// set some hardcoded secrets
					"OCIS_JWT_SECRET": "some-ocis-jwt-secret",
					"OCIS_MACHINE_AUTH_API_KEY": "some-ocis-machine-auth-api-key",
					"OCIS_TRANSFER_SECRET": "some-ocis-transfer-secret",
					// idm ldap
					"IDM_SVC_PASSWORD": "some-ldap-idm-password",
					"GRAPH_LDAP_BIND_PASSWORD": "some-ldap-idm-password",
					// reva ldap
					"IDM_REVASVC_PASSWORD": "some-ldap-reva-password",
					"GROUPS_LDAP_BIND_PASSWORD": "some-ldap-reva-password",
					"USERS_LDAP_BIND_PASSWORD": "some-ldap-reva-password",
					"AUTH_BASIC_LDAP_BIND_PASSWORD": "some-ldap-reva-password",
					// idp ldap
					"IDM_IDPSVC_PASSWORD": "some-ldap-idp-password",
					"IDP_LDAP_BIND_PASSWORD": "some-ldap-idp-password",
					// storage users mount ID
					"GATEWAY_STORAGE_USERS_MOUNT_ID": "storage-users-1",
					"STORAGE_USERS_MOUNT_ID": "storage-users-1",
					// graph application ID
					"GRAPH_APPLICATION_ID": "application-1",

					// service accounts
					"OCIS_SERVICE_ACCOUNT_ID": "service-account-id",
					"OCIS_SERVICE_ACCOUNT_SECRET": "service-account-secret",

				}
			},

butonic · 2024-09-10T07:55:08Z

hm

2024-09-10T09:54:00+02:00 ERR error reading trash link error="readlink /home/jfd/tmp/posix-storage/spaces/e2/b2edae-0b58-420c-a340-1f979a358ba1/trash/fa/21/50/2d/-9bbf-45f3-bb17-916109f370c7: no such file or directory" key=fa21502d-9bbf-45f3-bb17-916109f370c7 pkg=rgrpc relative_path= service=storage-users spaceid=e2b2edae-0b58-420c-a340-1f979a358ba1 traceid=0b569fe44df68d41abf6500481c59bbb trashRoot=/home/jfd/tmp/posix-storage/spaces/e2/b2edae-0b58-420c-a340-1f979a358ba1/trash/fa/21/50/2d/-9bbf-45f3-bb17-916109f370c7
2024-09-10T09:54:00+02:00 ERR failed to list recycle error="readlink /home/jfd/tmp/posix-storage/spaces/e2/b2edae-0b58-420c-a340-1f979a358ba1/trash/fa/21/50/2d/-9bbf-45f3-bb17-916109f370c7: no such file or directory" key=fa21502d-9bbf-45f3-bb17-916109f370c7 pkg=rgrpc reference={"path":"./a1","resource_id":{"opaque_id":"e2b2edae-0b58-420c-a340-1f979a358ba1","space_id":"e2b2edae-0b58-420c-a340-1f979a358ba1","storage_id":"storage-users-1"}} service=storage-users status={"code":15,"message":"error listing recycle","trace":"0b569fe44df68d41abf6500481c59bbb"} traceid=0b569fe44df68d41abf6500481c59bbb
2024-09-10T09:54:00+02:00 ERR error gathering members for event error="error listing recycle: error listing recycle" event={"Event":{"Executant":{"idp":"https://cloud.ocis.test","opaque_id":"some-admin-user-id-0000-000000000000","type":1},"ID":{"opaque_id":"fa21502d-9bbf-45f3-bb17-916109f370c7","space_id":"e2b2edae-0b58-420c-a340-1f979a358ba1","storage_id":"storage-users-1"},"Owner":null,"Ref":{"path":"./a1","resource_id":{"opaque_id":"e2b2edae-0b58-420c-a340-1f979a358ba1","space_id":"e2b2edae-0b58-420c-a340-1f979a358ba1","storage_id":"storage-users-1"}},"SpaceOwner":{"idp":"https://cloud.ocis.test","opaque_id":"some-admin-user-id-0000-000000000000","type":1},"Timestamp":{"nanos":43455299,"seconds":1725954840}},"ID":"0cffd4b3-87cf-4998-8a0f-4e98dd82a47a","InitiatorID":"9361559d-7913-4337-b657-ec752d07b101","TraceParent":"00-176aaf3a64f9936fedff60caaf02edde-b86103ea760f187a-00","Type":"events.ItemTrashed"} service=clientlog

tree .Trash 
.Trash
├── files
│   ├── 6a257aa4-330a-48b5-9b93-61d383986b74.trashitem
│   └── 6a59b4e3-2318-452a-b101-559d3321a20d.trashitem
│       └── 123.txt
└── info
    ├── 6a257aa4-330a-48b5-9b93-61d383986b74.trashinfo
    └── 6a59b4e3-2318-452a-b101-559d3321a20d.trashinfo

5 directories, 3 files

…rchs

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

aduffeck · 2024-09-10T10:56:17Z

@butonic Could you take another look? It works as expected also with memory filemetadata cache for me now.

Fixes owncloud/ocis#9909

butonic

cool, yeah this works!

aduffeck force-pushed the posixfs-trashbin branch 5 times, most recently from 71bb7e7 to a81d6f9 Compare August 1, 2024 10:12

aduffeck force-pushed the posixfs-trashbin branch 4 times, most recently from 26400f3 to 35d40e5 Compare August 9, 2024 08:11

aduffeck force-pushed the posixfs-trashbin branch from 5cdee17 to eef3559 Compare August 14, 2024 10:59

aduffeck marked this pull request as ready for review August 14, 2024 11:27

aduffeck requested review from a team, labkode and glpatcern as code owners August 14, 2024 11:27

aduffeck mentioned this pull request Aug 15, 2024

[full-ci] Adapt to posixfs changes owncloud/ocis#9817

Merged

aduffeck force-pushed the posixfs-trashbin branch 4 times, most recently from cbf1393 to b4dc71c Compare August 22, 2024 08:04

aduffeck added 3 commits September 10, 2024 10:13

Refactor decomposedfs to make trashbin functionality modular

c5a314a

Implement a freedesktop.org compliant trashbin for posixfs

5bd4651

posixfs does do trashbin now

e7326e8

aduffeck and others added 19 commits September 10, 2024 10:13

Fix linter issues

b011ac6

Adapt expected failures

fa1f423

Fix error handling when warming up the id caches

61c56ad

Propagate the tree time change when a directory changes

f35e595

Adapt expected failures

3497c5f

Fix compilation issue on certain architectures

8c4a69e

Fix false positive linter warning. The conversion is needed on some a…

895b144

…rchs

Fix size propagation

f5c69ff

Switch the ScanDebounceDelay option to a time.Time

8e1c5d1

Rely on the assimilation to update the tree sizes for us

087aebf

Refactor handling of blob id and size attrs to allow specialization

a1e8fa4

Remove unused revision code

212e697

Fix linter issues

5914e66

Add changelog

b34f424

Also purge the metadata from the actual file when purging metadata

17e9fc6

Handle file id clashes when a file/dir was copied/restored with xattrs

9545750

Fix linter issues

b249f93

Do not destroy metadata when moving items into the trashbin

86147b3

parse DeletionDate in local time

14103ee

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

aduffeck force-pushed the posixfs-trashbin branch from ac7736c to 14103ee Compare September 10, 2024 08:13

aduffeck added 2 commits September 10, 2024 10:18

Fix merge error

4aa6b5f

Fix broken cached metadata after restore

9a503b1

aduffeck mentioned this pull request Sep 10, 2024

posix storage driver: cannot open files owncloud/ocis#9909

Closed

aduffeck force-pushed the posixfs-trashbin branch from 1d0ca1c to bb6c1da Compare September 10, 2024 13:54

Refuse to run on a non-shared or noop id cache

a1d58bf

Fixes owncloud/ocis#9909

aduffeck force-pushed the posixfs-trashbin branch from bb6c1da to a1d58bf Compare September 10, 2024 14:19

butonic approved these changes Sep 16, 2024

View reviewed changes

butonic merged commit cedd4ef into cs3org:edge Sep 16, 2024
9 of 10 checks passed

butonic deleted the posixfs-trashbin branch September 16, 2024 08:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Posixfs trashbin #4779

Posixfs trashbin #4779

aduffeck commented Jul 26, 2024

update-docs bot commented Jul 26, 2024

butonic commented Aug 26, 2024 •

edited

Loading

butonic commented Aug 26, 2024

butonic commented Aug 26, 2024 •

edited

Loading

butonic commented Aug 26, 2024

aduffeck commented Sep 10, 2024

butonic commented Sep 10, 2024

butonic commented Sep 10, 2024 •

edited

Loading

aduffeck commented Sep 10, 2024

butonic left a comment

Posixfs trashbin #4779

Posixfs trashbin #4779

Conversation

aduffeck commented Jul 26, 2024

update-docs bot commented Jul 26, 2024

butonic commented Aug 26, 2024 • edited Loading

butonic commented Aug 26, 2024

butonic commented Aug 26, 2024 • edited Loading

butonic commented Aug 26, 2024

aduffeck commented Sep 10, 2024

butonic commented Sep 10, 2024

butonic commented Sep 10, 2024 • edited Loading

aduffeck commented Sep 10, 2024

butonic left a comment

Choose a reason for hiding this comment

butonic commented Aug 26, 2024 •

edited

Loading

butonic commented Aug 26, 2024 •

edited

Loading

butonic commented Sep 10, 2024 •

edited

Loading