Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Could not delete created temporary path #63

Closed
zdyxry opened this issue Jun 25, 2023 · 10 comments
Closed

Could not delete created temporary path #63

zdyxry opened this issue Jun 25, 2023 · 10 comments
Labels
duplicate This issue or pull request already exists

Comments

@zdyxry
Copy link
Contributor

zdyxry commented Jun 25, 2023

OS info:

➜  try git:(main) ✗ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Minimal Reproducible Script：

#!/usr/bin/bash

cur=$(pwd)

sudo rm -rf /tmp/tmp.*
$cur/try -y touch $HOME/testfile
ls -l /tmp/testfile
ls -l /tmp/tmp.*
rm -rf /tmp/tmp.*
$cur/try -y rm $HOME/testfile
ls -l /tmp/testfile
ls -l /tmp/tmp.*
rm -rf /tmp/tmp.*

Output logs:

➜  try git:(main) ✗ sh a.sh
Warning: Failed mounting /boot as an overlay, see /tmp/tmp.iWAZWwEcHw
Warning: Failed mounting /snap as an overlay, see /tmp/tmp.iWAZWwEcHw
Warning: Failed mounting /var as an overlay, see /tmp/tmp.iWAZWwEcHw
Warning: Failed mounting /snap/bare/5 as an overlay, see /tmp/tmp.iWAZWwEcHw
...
Warning: Failed mounting /snap/firefox/2800 as an overlay, see /tmp/tmp.iWAZWwEcHw
ls: cannot access '/tmp/testfile': No such file or directory
-rw------- 1 yiran yiran 2668  6月 25 17:28 /tmp/tmp.iWAZWwEcHw
-rwx--x--x 1 yiran yiran  120  6月 25 17:28 /tmp/tmp.pU1iJHkqNZ
-rwx--x--x 1 yiran yiran 1824  6月 25 17:28 /tmp/tmp.ZJln7EqgWV

/tmp/tmp.a5YeVGkibC:
total 12
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 temproot
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 upperdir
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 workdir
Warning: Failed mounting /boot as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /var as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/bare/5 as an overlay, see /tmp/tmp.1amwhWmW0M
...
Warning: Failed mounting /var/snap/firefox/common/host-hunspell as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/snapd/19361 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /boot/efi as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/core18/2785 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/slack/79 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/gnome-42-2204/111 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/snapd/19457 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/core20/1950 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/core22/766 as an overlay, see /tmp/tmp.1amwhWmW0M
Warning: Failed mounting /snap/firefox/2800 as an overlay, see /tmp/tmp.1amwhWmW0M
ls: cannot access '/tmp/testfile': No such file or directory
-rw------- 1 yiran yiran 2668  6月 25 17:28 /tmp/tmp.1amwhWmW0M
-rwx--x--x 1 yiran yiran  117  6月 25 17:28 /tmp/tmp.iNGI2499F3
-rwx--x--x 1 yiran yiran 1824  6月 25 17:28 /tmp/tmp.LjL0GhNZOQ

/tmp/tmp.j8aZKEw1zb:
total 12
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 temproot
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 upperdir
drwxrwxr-x 29 yiran yiran 4096  6月 25 17:28 workdir
rm: cannot remove '/tmp/tmp.j8aZKEw1zb/workdir/home/work': Permission denied

The file path operated by the try command will remain, and then cannot be deleted by the current user.
@mgree
Copy link
Contributor

mgree commented Jun 25, 2023

A few things will help us debug this:

  • Can you tell us how your /tmp is mounted?
  • Can you please show the permissions of the files in workdir at each stage?

Thanks!

@zdyxry
Copy link
Contributor Author

zdyxry commented Jun 26, 2023

/tmp dir info:

➜  try git:(main) ✗ mount |grep tmp
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16393976k,nr_inodes=4098494,mode=755,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3287196k,mode=755,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
tmpfs on /run/user/125 type tmpfs (rw,nosuid,nodev,relatime,size=3287192k,nr_inodes=821798,mode=700,uid=125,gid=130,inode64)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3287192k,nr_inodes=821798,mode=700,uid=1000,gid=1000,inode64)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3287196k,mode=755,inode64)
tmpfs on /run/qemu type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
➜  try git:(main) ✗ ll / |grep tmp
drwxrwxrwt  25 root root  60K  6月 26 08:10 tmp

Exec ./try touch $HOME/testfile prompt Commit these changes? [y/N] tmp dir permissions:

➜  /tmp ll -trh
drwxrwxr-x 8 yiran yiran 4.0K  6月 25 17:42 try
drwx------ 5 yiran yiran 4.0K  6月 26 08:12 tmp.N2h9j5rPDT
-rwx--x--x 1 yiran yiran 1.8K  6月 26 08:12 tmp.N11kDiHPxu
-rwx--x--x 1 yiran yiran  121  6月 26 08:12 tmp.bFLaUHf8mR
-rw------- 1 yiran yiran 2.7K  6月 26 08:12 tmp.812oSzBLj0
➜  /tmp ll -trh /tmp/tmp.N2h9j5rPDT
total 12K
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:12 workdir
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:12 upperdir
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:12 temproot
➜  /tmp ll -trh /tmp/tmp.N2h9j5rPDT/workdir
total 108K
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:12 dev
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:12 proc
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:12 run
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:12 sys
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 bin
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 cdrom
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 boot
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 etc
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 home
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 lib32
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 lib
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 lib64
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 lost+found
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 libx32
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 media
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 mydir
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 mnt
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 nix
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 opt
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 root
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 snap
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 sbin
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 srv
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 tmp
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 usr
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 yiran
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:12 var
➜  /tmp ll -trh /tmp/tmp.N2h9j5rPDT/workdir/home
total 4.0K
d--------- 2 yiran yiran 4.0K  6月 26 08:12 work

Exec ./try rm $HOME/testfileprompt Commit these changes? [y/N] tmp dir permissions:

➜  /tmp ll -trh tmp.pmesVRCcFi
total 12K
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:15 workdir
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:15 upperdir
drwxrwxr-x 29 yiran yiran 4.0K  6月 26 08:15 temproot
➜  /tmp ll -trh tmp.pmesVRCcFi/workdir
total 108K
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:15 dev
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:15 run
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:15 proc
drwxrwxr-x 2 yiran yiran 4.0K  6月 26 08:15 sys
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 boot
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 bin
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 cdrom
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 etc
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 lib
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 home
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 lib32
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 lib64
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 lost+found
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 libx32
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 media
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 mydir
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 mnt
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 nix
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 opt
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 root
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 snap
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 sbin
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 srv
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 tmp
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 var
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 usr
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 yiran
➜  /tmp ll -trh tmp.pmesVRCcFi/workdir/home
total 4.0K
d--------- 2 yiran yiran 4.0K  6月 26 08:15 work

When I commit these changes and try to delete the temporary directory:

➜  /tmp rm -rf tmp.pmesVRCcFi
rm: cannot remove 'tmp.pmesVRCcFi/workdir/home/work': Permission denied
➜  /tmp tree tmp.pmesVRCcFi
tmp.pmesVRCcFi
└── workdir
    └── home
        └── work  [error opening dir]

3 directories, 0 files

@ezrizhu
Copy link
Collaborator

ezrizhu commented Jun 26, 2023 •
edited
Loading 

in fs
drwxrwxrwt  25 root root  60K  6月 26 08:10 tmp

in workdir
drwxrwxr-x 3 yiran yiran 4.0K  6月 26 08:15 tmp

Could this be also because of tmp is losing 777t in overlay #6 ?

@zdyxry
Copy link
Contributor Author

zdyxry commented Jun 26, 2023

I also tried it with AlmaLinux 9 and it didn't reproduce the problem.

Do you have any debugging experience to share?

@mgree
Copy link
Contributor

mgree commented Jun 26, 2023

So it seems that /tmp is an ordinary directory.

➜  /tmp ll -trh /tmp/tmp.N2h9j5rPDT/workdir/home
total 4.0K
d--------- 2 yiran yiran 4.0K  6月 26 08:12 work

indicates that overlayfs's in-flight work is getting no perms at all, which seems odd. (Though I've never checked to see what overlayfs does with it, and I believe it's always empty by the time I've committed.)

These logs don't quite cover all of our bases---what's $HOME set to, and is it in the same mount as /tmp? What's the FS type of /?

@zdyxry
Copy link
Contributor Author

zdyxry commented Jun 27, 2023

Hi , $HOME is an ordinary directory, / filesystem is ext4 .

➜  try git:(main) ✗ echo $HOME
/home/yiran
➜  try git:(main) ✗ mount |grep -i home
➜  try git:(main) ✗ ll / |grep home
drwxr-xr-x   3 root root 4.0K  8月 12  2021 home
➜  try git:(main) ✗ ll /home
total 4.0K
drwxr-xr-x 61 yiran yiran 4.0K  6月 27 08:43 yiran
➜  try git:(main) ✗ findmnt| head -n 2
TARGET                                                                                             SOURCE                         FSTYPE              OPTIONS
/                                                                                                  /dev/sda2                      ext4                rw,relatime,errors=remount-ro
➜  try git:(main) ✗ lsblk |grep -iv snap # exclude snap mount points
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   200G  0 disk
├─sda1   8:1    0   512M  0 part /boot/efi
└─sda2   8:2    0 199.5G  0 part /var/snap/firefox/common/host-hunspell
                                 /
sr0     11:0    1  1024M  0 rom

This doesn't seem to be related to $HOME, changing $HOME to /tmp/ has the same result.

@mgree
Copy link
Contributor

mgree commented Jun 27, 2023

Just wanted to be sure that $HOME didn't matter. The ext4 rules out underlying fs weirdness.

What kernel version are you running?

@zdyxry
Copy link
Contributor Author

zdyxry commented Jun 28, 2023

kernel is the default version provided by Ubuntu:

kernel version and /proc/cmdline :

➜  try git:(main) ✗ uname -a
Linux yiran-virtual-machine 5.19.0-43-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon May 22 13:39:36 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
➜  try git:(main) ✗ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.19.0-43-generic root=UUID=77e940a3-e385-4f7f-ba35-b2506cf25be9 ro quiet splash vt.handoff=7

@mgree
Copy link
Contributor

mgree commented Jun 28, 2023

Huh. I'm seeing something similar in the (failing) test case of #66, which tells me that cleaning these up is not trivial and/or something is weird in overlayfs's workdir.

@angelhof angelhof mentioned this issue Jun 29, 2023
Open
@mgree
Copy link
Contributor

mgree commented Jun 29, 2023

Okay: work for this will proceed in #93, so I'm marking this as a duplicate.

@mgree mgree added the duplicate This issue or pull request already exists label Jun 29, 2023
@mgree mgree closed this as completed Jul 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate This issue or pull request already exists
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mgree @zdyxry @ezrizhu