Failed node: cannot push .Trash-1000 to S3

[elisabettai]

User: Hello guys, this is an osparc project (it has a failed interactive node)

dev answer: Unfortunately we have no good solution for dealing with files and folders generated by the user with wrong permissions. In the interest of data integrity we fail the closing of the service. If an important file, accidentally has wrong permissions, it is is an issue if it gets lost.

Issue created from a Mattermost message by @cassara.

In the sidecar logs, we have this:

2024-03-19 00:07:21 ERROR : .Trash-1000: failed to open directory ".Trash-1000": open /dy-volumes/home/jovyan/work/workspace/.Trash-1000: permission denied
2024-03-19 00:07:21 ERROR : .Trash-1000: failed to open directory ".Trash-1000": open /dy-volumes/home/jovyan/work/workspace/.Trash-1000: permission denied
Transferred:   	          0 B / 5.757 MiB, 0%, 0 B/s, ETA -

We should think about ways to improve this probably, as it's not the first time that happens

[elisabettai]

Just happened to me again: a dynamic service "Failed" when opening a study on osparc.io.
In my service (JupyterLab SPARC) I have a .Trash-1000 folder which can't be uploaded to S3, thus making the service fail.

Steps to resolve:

1. In Portainer, open a console in the offending dynamic sidecar (uuid 0410f5d6-5293-5a08-93c8-097faeeb84d9)
2. Check the output of ls -la /dy-volumes/home/jovyan/work/workspace/ (note the permission of .Trash-1000:

root@17dfe643bc3d:/home/scu# ls -la /dy-volumes/home/jovyan/work/workspace/
total 484
drwxrwxr-- 3 1000 scdocker   4096 Apr 25 16:13 .
drwxrwxr-- 5 scu  scdocker   4096 Apr 25 14:38 ..
drwx------ 4 1000 users      4096 Apr 25 15:33 .Trash-1000
-rw-r--r-- 1 1000 scu         228 Apr 25 14:37 .hidden_do_not_remove
-rw-r--r-- 1 1000 scu      473886 Apr 25 16:13 Tutorial.ipynb
-rw-r--r-- 1 1000 users       234 Apr 25 15:40 config.ini

3. rm the weird folder: rm -r /dy-volumes/home/jovyan/work/workspace/.Trash-1000/
4. In Portainer, open a console in director-v2
5. Close the service: simcore-service-director-v2 close-and-save-service 0410f5d6-5293-5a08-93c8-097faeeb84d9

[elisabettai]

Found a way to reproduce the issue:

1. Open a jupyter-axondeepseg or jupyter-sparc
2. Delete a file from the file tree using the JupyterLab GUI
3. A .Trash-1000 folder with problematic permissions (drwx------ see previous comment) is created in workspace

Interestingly, this does not happen with jupyter-math.

Services affected
jupyterlab-axondeepseg
jupyterlab-sparc
jupyterlab-julia
jupyterlab-R
cookiecutter for jupyters