Permission denied error when starting Gitea after updating to 1.17.0

[mpeter50]

Description

I have updated to Gitea 1.17.0, but when starting it I receive an error in the console, and then it exits. The message is available in the first file of my log gist.

In the docker compose file I have 2 volumes defined for use directly by Gitea:

• /mnt/gitea/data/ - /var/lib/gitea
• /mnt/gitea/custom - /etc/gitea

Gitea should run under user 1000, as set up in the docker compose file:

environment:
    - USER_UID=1000
    - USER_GID=1000

According to the console log, Gitea tries to create the /var/lib/gitea/custom/home directory, which is mapped to /mnt/gitea/data/custom/home on the host system.
After inspecting this latter path, I see that no one is granted write permissions on the custom directory in it:

/mnt/gitea/data $ ls -hal
total 20K
drwxr-xr-x  5 1000 1000 4.0K okt   29  2021 .
drwxr-xr-x  5 root root 4.0K jún   14  2021 ..
dr-x------ 13 1000 1000 4.0K nov   10  2021 custom
drwx------  5 1000 1000 4.0K márc  26 00:21 git
drwx------  2 root root 4.0K jún   13  2021 ssh

However, if I grant write permission to the owner with sudo chmod u+w custom, Gitea will still produce the same error on startup, then exit, and when exited, the write permission on the directory have disappeared.

I suspect that Gitea is unable to create the new home directory because it removes the write permission from it's parent directory, but I also suspect that I might be doing something wrong, as no one else has reported this bug yet.

If I try to fix this problem by hand, by manually creating the directory and setting ownership, as seen here:

sudo mkdir custom/home
sudo chown 1000:1000 custom/home

then starting up Gitea can continue a little, but will exit again because it wants yet another directory besides the new home one. The output from this run is in the second file in my log gist.

Gitea Version

v1.17.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/mpeter50/c7ba7eb7fc5e74fd708736736800a4e6

Screenshots

No response

Git Version

2.36.2

Operating System

Raspbian

How are you running Gitea?

I'm running Gitea in Docker, for which I have built the container image myself, using the Docker.rootless DOckerfile in the repo.

Database

MySQL

[mpeter50]

For the time being, I have reverted to 1.16.5, and it seems to run as expected.

[zeripath]

Set [git].HOME_PATH to an appropriate place for your configuration in app.ini.

See the breaking changes notices in our blog post: https://blog.gitea.io/2022/07/gitea-1.17.0-is-released/#-internal-gitconfig-19732httpsgithubcomgo-giteagiteapull19732

Copied below:

Internal Gitconfig (#19732)

Previously, Gitea used the users gitconfig ($HOME/.gitconfig) in addition to the system gitconfig (/etc/gitconfig).

Now, Gitea uses the system gitconfig (/etc/gitconfig) combined with an internal gitconfig located in {[git].HOME_PATH}/.gitconfig. If you customized your user gitconfig for Gitea, you should add these customizations to one of the available gitconfigs. Additional git-relevant files that are normally in your user home directory, like $HOME/.gnupg, should be moved/ copied to {[git].HOME_PATH}/ as well.

[mpeter50]

Thank you for the information.

I've read the mentioned breaking change notices, and I haven't modified the gitconfig of the user that runs Gitea.

On the other hand, maybe I misunderstand something, but why should I override the git home path?

The current path is fine to me, and even more, I prefer it to be in the original location. The problem is that Gitea is unable to create this new directory, because it does not have permission to do so. I think this might be caused by Gitea trying to fix the permissions of its own directory, but does it wrongly, and it gets in its own way.

[zeripath]

The error you're receiving is stating it can't create what is the new default [git].HOME_PATH so I suggest you override this.

[mpeter50]

The error you're receiving is stating it can't create what is the new default

I understand, but is that expected behavior? Is this specific to the Docker environment? I'm asking because you have defined that path as the default for a reason.
As far as I'm aware I did not do anything to harden the filesystem permissions of Gitea directories, but Gitea itself does that, and in doing so it breaks itself. If this is the case, I think this is a bug.

[Draic]

I changed nothing to any ssh configs but the 1.17 update still breaks my SSH.

[techknowlogick]

What is the output of docker info?

[mpeter50]

$ sudo docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)

Server:
 Containers: 9
  Running: 9
  Paused: 0
  Stopped: 0
 Images: 27
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.103-v7l+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 1.87GiB
 Name: rpi
 ID: 3KSL:X4E5:RLEW:IP66:X6LR:CN5Z:Z7PV:DNNB:5TQF:KIPY:UMWV:AXXV
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support

[zeripath]

The [git] HOME_PATH is a new setting from 1.17.

It defaults to APP_DATA_PATH/home. Where [server] APP_DATA_PATH is /data/gitea in the docker and thus /data/gitea/home

Simply create the directory /data/gitea/home on the docker and give Gitea permission to write to that directory.

[mpeter50]

It defaults to APP_DATA_PATH/home. Where [server] APP_DATA_PATH is /data/gitea in the docker and thus /data/gitea/home

In my configuration file [server] APP_DATA_PATH has the value /var/lib/gitea/custom.
I don't remember customizing this value, and according to the compose snippet in the Docker Rootless installation document, most of Gitea data is stored in the parent of this directory by default, so probably this was the default for Docker, at least when I started using Gitea.
I also don't have a /data directory in the container:

$ sudo docker exec -it gitea bash
bash-5.1$ ls -a /
.           ..          .dockerenv  app         bin         dev         etc         home        lib         media       mnt         opt         proc        root        run         sbin        srv         sys         tmp         usr         var
bash-5.1$ ls -a /var/lib/gitea
.       ..      custom  git     ssh

I haven't customized the value of [git] HOME_PATH (I don't have a git section), so this means it should point to APP_DATA_PATH/home, which is /var/lib/gitea/custom/home in my case.
According to the second paragraph in this document. Gitea should try to create the directories it needs, including this one. That is fine because it actually tries to create the directory at /var/lib/gitea/custom/home, but it fails, because it does not have write permissions on the directory /var/lib/gitea/custom:

$ sudo docker exec -it gitea bash
bash-5.1$ ls -hal /var/lib/gitea/custom
total 52K
dr-x------   13 git      git         4.0K Jul 31 18:34 .
drwxr-xr-x    5 git      git         4.0K Oct 29  2021 ..
drwxr-xr-x   19 git      git         4.0K Nov 12  2021 attachments
drwxr-xr-x    3 git      git         4.0K Sep 20  2021 avatars
drwxr-xr-x    2 git      git         4.0K Jul 31 11:31 conf
drwxr-xr-x    4 git      git         4.0K Jun 14  2021 indexers
drwxr-xr-x    2 git      git         4.0K Oct 22  2021 jwt
drwxr-xr-x    2 git      git         4.0K Jun 13  2021 log
drwxr-xr-x    8 git      git         4.0K Oct 22  2021 queues
drwxr-xr-x    2 git      git         4.0K Oct 22  2021 repo-archive
drwxr-xr-x    2 git      git         4.0K Jun 14  2021 repo-avatars
drwx------   18 git      git         4.0K May 24 20:11 sessions
drwxr-xr-x    3 git      git         4.0K Aug  1 23:50 tmp

As you may see, the permissions for the directory is only reading and execution for the git user.
I haven't modified the permissions, except trying to give the owner of the /var/lib/gitea/custom directory write permissions to it, but that was reverted as soon as Gitea started, and it still wasn't able to create the directories that it wanted.

---

Simply create the directory /data/gitea/home on the docker and give Gitea permission to write to that directory.

I don't have a /data directory, but if I try to create /var/lib/gitea/custom/home:

• I need to use sudo, because otherwise I don't have permissions
• I will just shift the problem a little bit: next I'll also need to create the packages directory, and who knows how much more, because Gitea won't be able to create that either

[swang-harbin]

I installed gitea 1.70 using sudo snap install gitea on Ubuntu 22.04 LTS, it doesn't automatically create /etc/gitea/app.ini.
I create a user called gitea and set its home directory to /var/lib/gitea , and execute sudo mkdir /var/lib/gitea && sudo chown gitee: /var/lib/gitea to modify permission. The gitea user has all access rights to /var/lib/gitea, as I understand it should not have this error.
The /etc/systemd/system/gitea.service is:

[Unit]
Description=Gitea (Git with a cup of tea)
After=syslog.target
After=network.target
Wants=mysql.service
After=mysql.service

[Service]
RestartSec=2s
Type=simple
User=gitea
Group=gitea
WorkingDirectory=/var/lib/gitea/
RuntimeDirectory=gitea
ExecStart=/usr/local/bin/gitea web --config /etc/gitea/app.ini
Restart=always
Environment=USER=gitea HOME=/var/lib/gitea/ GITEA_WORK_DIR=/var/lib/gitea

[Install]
WantedBy=multi-user.target

When I set the Repository Root Path to /var/lib/gitea/common/data/gitea-repositories and Run As Username to gitea on the initial configuration page, it cannot be installed successfully, and it prompts The repository root path is invalid: mkdir / var/lib/gitea: read-only file system.
I don't know what [git].HOME_PATH is or where it is, because it doesn't have a default configuration file nor is it detailed in the installation process.
I think if [git].HOME_PATH is mandatory, then it should have a default configuration, or specify it before startup as an environment variable. This configuration confuses people.

[chris-fj]

I am receiving the same error. I have the following mount:

- /home/git/.gitconfig:/data/gitea/home/.gitconfig

and I have set the following env definitions

GITEA__git__HOME_PATH=/data/gitea/home
USER_UID=116
USER_GID=123

and the file .gitconfig has the following permissions

$ sudo -u git ls -ln /home/git/.gitconfig 
-rw-rw-r-- 1 116 123 1 Aug 16 21:57 /home/git/.gitconfig

however, when starting up the container, I receive this error:

- error: could not lock config file /data/gitea/home/.gitconfig: Permission denied

I understand that the user whose UID/GID is defined in the environment is the one performing the operations and, as stated in the ls command, said user can read/write the .gitconfig file. Am I missing something here?

[Draic]

I made a fresh install on 1.17, which worked for a while. But now it is having the exact same problem as before, with permission errors when it tries to read the SSH key despite its permissions being correct as we established while troubleshooting the issue on Discord. I think 1.17 is flawed

[mpeter50]

@swang-harbin I don't know how to install Gitea that way, but some things I wanted to let you know:

[...] and it prompts The repository root path is invalid: mkdir / var/lib/gitea: read-only file system.

This looks like a path specification problem. I'm not sure where did you set "Repository Root Path", but looking at the error message it looks the first / has been separated somehow from the rest of the path. Also, I think paths (and other strings) contain characters that need to be escaped if used in systemd unit files; in turn maybe your "Repository Root Path" is ok, and you need to fix the paths in the Environment line and other lines of your unit file that you inserted in your comment. Please check the systemd unit file specification for that.

I don't know what [git].HOME_PATH is or where it is, because it doesn't have a default configuration file nor is it detailed in the installation process.

There is the gitea.ini file (somewhere) that Gitea uses. It is mostly a traditional ini file.
The mentioned "path" refers to a HOME_PATH setting under the [git] section. If you want to customize it, it should look like this in the said file:

[git]
HOME_PATH=/your/path/goes/here

Other settings and other sections can be before and after this excerpt, and this section can also contain other settings, but I think you should only define the [git] section once, so it shouldn't appear more than once.
BUT, you don't have to do anything with this setting, or the [git] section, if you are fine with the default configuration, as by default this setting has the value of <TODO: find it's value, insert here>.

---

@c4tich

I am receiving the same error.

however, when starting up the container, I receive this error:

- error: could not lock config file /data/gitea/home/.gitconfig: Permission denied

I think that is a different error, but it might have some similarities.

Does the git user have the UID of 116, or is it in the group of GID 123?

[mpeter50]

@techknowlogick @zeripath I feel my issue is ignored.

To recap what has happened so far:
I have introduced my problem: Gitea is changing filesystem permissions so that it gets in its own way, and in turn it cannot create the new directories used for the new features of 1.17.
You recommended to change a setting permanently (store a few directories elsewhere), to work around the issue.
I didn't accept that, because it does not solve the actual problem, but creates more on the topic of maintenance.

The reason I didn't accept the workaround to set a different directory for the git home directory, is that my Gitea instance uses the default storage path configurations, and yet it mismanages the directory where it stores all its data.
Gitea has total authority of the data directory it uses (by default), and yet it cannot create its subdirectory there.
If I try to fix the permissions so that Gitea can create its own new subdirectories, it reverts the changes before it would do anything else, including creating the new subdirectories.

[swang-harbin]

@mpeter50 First of all,thanks for your reply.
"Repository Root Path" and "Run As Username" are both on the Initial Configuration page for port 3000：

)

And the access permissions of the /var/lib/gitea directory are：

$ ll -d /var/lib/gitea
drwxr-xr-x 5 gitea gitea 4096 Aug 16 04:41 /var/lib/gitea/

And systemd unit file copied from here

I found instructions for [git].HOME here, but it still gives the same error after I configure APP_DATA_PATH=/var/lib/gitea in the Environment in the systemd unit file

At the same time, after I set Environment to Environment="USER=gitea HOME=/var/lib/gitea/ GITEA_WORK_DIR=/var/lib/gitea/" or Environment="USER=gitea HOME=/var/lib/gitea/ GITEA_WORK_DIR=/var/lib/gitea/ APP_DATA_PATH=/var/lib/gitea", it still gives the same error

[lunny]

Could you run ./gitea doctor to check if there is any error hints?

[wxiaoguang]

There are many different voices here and some are caused by different problems.

IMO most of the problems are not related to Gitea itself, but related to the environment.

I will provide a summary for various "permission" problems:

1. Make sure your Gitea is running with the correct Linux user (uid). You can check a user's uid by command id the-user-name. And you can google (ps show uid/username) to show a process's uid or username (eg: ps aux, it may differ if you are using busybox). Docker users should check them inside the container
2. Make sure your filesystem has the correct uid/username as the Gitea process
3. Make sure there is no other program changing the uid on your filesystem (for example, some incorrect backup/restore program will break the permissions)
4. If you are using systemd, setting Environment="USER=gitea could be incorrect (although document said so ...). You can just set a service's user correctly by User=xxx
5. If you are using Docker, make sure the UID/GID mapping is correct inside the container, ie: you should make sure the ps aux show the same user as the the ls -l inside the container. In some cases there could be a docker (rare) bug that makes the permissions messy.
6. OpenSSH also requires that the .ssh and related directories/files have the correct permission, otherwise OpenSSH won't start or work (also not related to Gitea directly)

In short, in most cases permission problems are related to the environment, understand your environment and tune the permissions carefully (maybe in the future Gitea can provide a tool to help to check the environment)

If you believe there is a bug, please report: What's the username and uid are for Gitea process, what are the filesystem permissions(owner/mod) for the Gitea's custom/data and related directories/files. If you are running Gitea with docker, please collect these information inside the docker container.

[wxiaoguang]

@swang-harbin

If you see The repository root path is invalid: mkdir /var/lib/gitea: read-only file system., please use sudo to switch your command line user to gitea to test if you can do something like mkdir /var/lib/gitea/test-dir. It's highly likely your filesystem can not be written correctly (especially when you are using a Router or NAS device), or your Gitea process is running as a different user.

Or, you could try to use another directory for Gitea, eg: mkdir /data/gitea; chown ... /data/gitea then use it.

[swang-harbin]

@lunny
I run gitea doctor,and it prints the error below

$ gitea doctor
panic: Failed to create sublogger (doctor): open doctor.log: permission denied

goroutine 1 [running]:
code.gitea.io/gitea/modules/log.NewLogger(0x2a69fb5?, {0x2848a7e?, 0xc00221e728?}, {0x282bb70?, 0x1?}, {0xc00016d400?, 0x45ff69?})
        /build/gitea/parts/gitea/build/modules/log/log.go:49 +0x105
code.gitea.io/gitea/cmd.runDoctor(0xc000181080)
        /build/gitea/parts/gitea/build/cmd/doctor.go:152 +0x27e
github.com/urfave/cli.HandleAction({0x24010c0?, 0x36320a0?}, 0xc000eda1c0?)
        /root/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:524 +0xa8
github.com/urfave/cli.(*App).RunAsSubcommand(0xc000eda1c0, 0xc000180f20)
        /root/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:410 +0x831
github.com/urfave/cli.Command.startApp({{0x2848a7e, 0x6}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x294371f, 0x11}, {0x0, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:372 +0x6e7
github.com/urfave/cli.Command.Run({{0x2848a7e, 0x6}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x294371f, 0x11}, {0x0, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:102 +0x808
github.com/urfave/cli.(*App).Run(0xc000eda000, {0xc000134000, 0x2, 0x2})
        /root/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:277 +0x8a7
main.main()
        /build/gitea/parts/gitea/build/main.go:115 +0xcf9

[swang-harbin]

@wxiaoguang
I see from here that some environment variables need to be specified at runtime, so I use USER=gitea USERNAME=gitea HOME=/home/gitea/ GITEA_WORK_DIR=/home/gitea/ gitea web to start, but its print information is still start with /var/snap/gitea

gitea@xx:~$ USER=gitea USERNAME=gitea HOME=/home/gitea/ GITEA_WORK_DIR=/home/gitea/ gitea web
2022/08/18 05:06:09 cmd/web.go:102:runWeb() [I] Starting Gitea on PID: 119245
2022/08/18 05:06:09 ...s/install/setting.go:21:PreloadSettings() [I] AppPath: /snap/gitea/6058/gitea
2022/08/18 05:06:09 ...s/install/setting.go:22:PreloadSettings() [I] AppWorkPath: /var/snap/gitea/common
2022/08/18 05:06:09 ...s/install/setting.go:23:PreloadSettings() [I] Custom path: /var/snap/gitea/common
2022/08/18 05:06:09 ...s/install/setting.go:24:PreloadSettings() [I] Log path: /var/snap/gitea/common/log
2022/08/18 05:06:09 ...s/install/setting.go:25:PreloadSettings() [I] Configuration file: /var/snap/gitea/common/conf/app.ini
2022/08/18 05:06:09 ...s/install/setting.go:26:PreloadSettings() [I] Prepare to run install page
2022/08/18 05:06:10 ...s/install/setting.go:29:PreloadSettings() [I] SQLite3 is supported
2022/08/18 05:06:11 cmd/web.go:208:listen() [I] Listen: http://0.0.0.0:3000
2022/08/18 05:06:11 cmd/web.go:212:listen() [I] AppURL(ROOT_URL): http://localhost:3000/
2022/08/18 05:06:11 ...s/graceful/server.go:61:NewServer() [I] Starting new Web server: tcp:0.0.0.0:3000 on PID: 119245

It can launch the configuration page.I use its default repository root path: /var/snap/gitea/common/data/gitea-repositories,its error message becomes: Failed to save configuration: mkdir /var/snap/gitea/common/conf: permission denied.
If I change /var/snap/gitea/common/data/gitea-repositories to /home/gitea/common/data/gitea-repositories, it gives: The repository root path is invalid: mkdir /home/gitea/common: permission denied.
But I can manually execute the following command successfully:

gitea@xxx:~$ pwd
/home/gitea
gitea@xxx:~$ ls -ld /home/gitea/
drwxr-xr-x 3 gitea gitea 4096 Aug 18 04:59 /home/gitea/
gitea@xxx:~$ mkdir -p /home/gitea/common/data/gitea-repositories
gitea@xxx:~$ ls -ld /home/gitea/common/data/gitea-repositories
drwxrwxr-x 2 gitea gitea 4096 Aug 18 05:18 /home/gitea/common/data/gitea-repositories

I think this error may be because I installed using snap, maybe I should use docker.

[lunny]

@lunny I run gitea doctor,and it prints the error below

$ gitea doctor
panic: Failed to create sublogger (doctor): open doctor.log: permission denied

goroutine 1 [running]:
code.gitea.io/gitea/modules/log.NewLogger(0x2a69fb5?, {0x2848a7e?, 0xc00221e728?}, {0x282bb70?, 0x1?}, {0xc00016d400?, 0x45ff69?})
        /build/gitea/parts/gitea/build/modules/log/log.go:49 +0x105
code.gitea.io/gitea/cmd.runDoctor(0xc000181080)
        /build/gitea/parts/gitea/build/cmd/doctor.go:152 +0x27e
github.com/urfave/cli.HandleAction({0x24010c0?, 0x36320a0?}, 0xc000eda1c0?)
        /root/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:524 +0xa8
github.com/urfave/cli.(*App).RunAsSubcommand(0xc000eda1c0, 0xc000180f20)
        /root/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:410 +0x831
github.com/urfave/cli.Command.startApp({{0x2848a7e, 0x6}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x294371f, 0x11}, {0x0, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:372 +0x6e7
github.com/urfave/cli.Command.Run({{0x2848a7e, 0x6}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x294371f, 0x11}, {0x0, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:102 +0x808
github.com/urfave/cli.(*App).Run(0xc000eda000, {0xc000134000, 0x2, 0x2})
        /root/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:277 +0x8a7
main.main()
        /build/gitea/parts/gitea/build/main.go:115 +0xcf9

Looks like your current user have no write permissions to the log directory.

[mpeter50]

@lunny

Could you run ./gitea doctor to check if there is any error hints?

Do you say this to me or to swang-harbin?

If to me, is it ok to try that in my Gitea 1.16.5 container, or should I use the 1.17.0 container to execute this command?

---

@wxiaoguang

1. Make sure your Gitea is running with the correct Linux user (uid).
2. Make sure your filesystem has the correct uid/username as the Gitea process

I have ran the following commands inside the Gitea container, while it is running with version 1.16.5:

# docker exec -it gitea bash
bash-5.1$ id git
uid=1000(git) gid=1000(git) groups=1000(git),1000(git)
bash-5.1$
bash-5.1$ ps aux | grep gitea
    1 git       8h14 /usr/local/bin/gitea -c /etc/gitea/app.ini web
27406 git       0:00 grep gitea
bash-5.1$
bash-5.1$ ls -hal /var/lib/gitea
total 20K
drwxr-xr-x    5 git      git         4.0K Oct 29  2021 .
drwxr-xr-x    1 root     root        4.0K Mar 24 11:49 ..
dr-x------   13 git      git         4.0K Jul 31 18:34 custom
drwx------    5 git      git         4.0K Mar 26 00:21 git
drwx------    2 root     root        4.0K Jun 13  2021 ssh
bash-5.1$
bash-5.1$ stat -c "%u %g" /proc/1
1000 1000
bash-5.1$ stat -c "%u %g" /var/lib/gitea
1000 1000
bash-5.1$ stat -c "%u %g" /var/lib/gitea/custom/
1000 1000

ps cannot print the UID of processes as it seems the busybox of the container has a minimal version of it, so I used stat instead.

3. Make sure there is no other program changing the uid on your filesystem (for example, some incorrect backup/restore program will break the permissions)

I currently don't have any program that manages backups or similar tasks.
If you want, I may try to install the audit tools on this system and set up detection for file permission changes to see what actually makes the changes. Or rather in the container, probably that would make more sense.

4. If you are using systemd [...]

I do, but it is only involved in starting the Docker daemon. The containers are managed by docker-compose.

5. If you are using Docker, make sure the UID/GID mapping is correct inside the container, ie: you should make sure the ps aux show the same user as the the ls -l inside the container. In some cases there could be a docker (rare) bug that makes the permissions messy.

Do you mean that both of the mentioned commands need to be run in the container?
If so, the user names are matching.

6. OpenSSH also requires that the .ssh and related directories/files have the correct permission, otherwise OpenSSH won't start or work (also not related to Gitea directly)

Thanks for letting me know, the ownership of that directory was weird to me too, but didn't know what it should be. I'll let it alone for now, as I don't use it, and I think that feature has been removed from 1.17 anyway.

In short, in most cases permission problems are related to the environment, understand your environment and tune the permissions carefully (maybe in the future Gitea can provide a tool to help to check the environment)

In my opinion I have already tried to tune the permissions carefully, but my tunings were reverted as soon as I started the Gitea container.

If you believe there is a bug, please report: What's the username and uid are for Gitea process, what are the filesystem permissions(owner/mod) for the Gitea's custom/data and related directories/files. If you are running Gitea with docker, please collect these information inside the docker container.

I have included these near the top of this comment, in my response to your first and second points.

[wxiaoguang]

@mpeter50 This issue has became very long, I am not sure whether your problem has been resolved.

My first thought about your original problem is that the custom directory didn't have correct mode, it was dr-x------ , it's not writable, then it couldn't be written by Gitea correctly. Gitea expects that the custom directory should be writable. And during the startup, Gitea may create directories under the custom directory automatically.

Maybe you could do sudo chmod 0700 custom to fix the problem.

[wxiaoguang]

but my tunings were reverted as soon as I started the Gitea container.

What are reverted? If you mean that you can not make the custom directory writable, then it's highly likely caused by Docker's UID/GID mapping behavior (another long story, especially for users with CIFS/SMB/Windows storage).

[zeripath]

@swang-harbin:

I run gitea doctor,and it prints the error below

$ gitea doctor
panic: Failed to create sublogger (doctor): open doctor.log: permission denied
...

Whilst the error message is very unfriendly and should be fixed, you should read it and then think a bit more instead of throwing your hands up.

What is the problem?

The doctor command is panicking with Failed to create sublogger (doctor): open doctor.log: permission denied

What does that mean?

The doctor command cannot write to doctor.log in its current directory

How can this be solved?

1. Switch to a user who can write to doctor.log in the current directory
2. Consider if there is a way to change where the doctor command will write its logs.

Can the doctor command change where it logs?

Look at the help sheet for the command online or by running:

$ gitea doctor --help

What does that say?

NAME:
   Gitea doctor - A command to diagnose problems with the current Gitea instance according to the given configuration. Some problems can optionally be fixed by modifying the database or data storage.

...

   --log-file value               Name of the log file (default: "doctor.log"). Set to "-" to output to stdout, set to "" to disable

...

So how can I change the log file location?

Add the --log-file "-" option to the command e.g.:

gitea doctor --log-file "-"  --all

[mpeter50]

@wxiaoguang

@mpeter50 This issue has became very long, I am not sure whether your problem has been resolved.

I agree, multiple people have been replying with different problems. My issue has not been resolved, I'm still running 1.16.5, and I would like to update to 1.17.

My first thought about your original problem is that the custom directory didn't have correct mode, it was dr-x------ , it's not writable, then it couldn't be written by Gitea correctly. Gitea expects that the custom directory should be writable. And during the startup, Gitea may create directories under the custom directory automatically.

Yes, I understand that.
I have tried to manually add write permissions for the owner of the folder with chmod, but as soon as I start the Gitea container, the write permission gets revoked for some reason.

What are reverted? If you mean that you can not make the custom directory writable, then it's highly likely caused by Docker's UID/GID mapping behavior (another long story, especially for users with CIFS/SMB/Windows storage).

I meant this, yes. The permissions of the custom directory get changed from drwx------ to dr-x------.
I don't use network storage for any directories of Gitea, everything is stored on the root partition of the host system, which is a USB connected portable hard drive.
I'll try to look up this UID/GID remapping behavior, though could you please give me soime pointers? An article would do. I request this to make sure we are on the same page.

[mpeter50]

If you mean this feature of Docker, I have been using it for something else earlier, but currently it is disabled.

To make sure that this is how it is actually, inside the container I see the same UIDs as the owner of the custom, git and ssh directories, as I see outside of the container, looking at my docker volumes:

Inside the container:

bash-5.1$ ls -haln /var/lib/gitea/
total 20K
drwxr-xr-x    5 1000     1000        4.0K Oct 29  2021 .
drwxr-xr-x    1 0        0           4.0K Mar 24 11:49 ..
dr-x------   13 1000     1000        4.0K Jul 31 18:34 custom
drwx------    5 1000     1000        4.0K Mar 26 00:21 git
drwx------    2 0        0           4.0K Jun 13  2021 ssh

On the host system:

ls -haln /mnt/gitea/data/
total 20K
drwxr-xr-x  5 1000 1000 4.0K okt   29  2021 .
drwxr-xr-x  5    0    0 4.0K jún   14  2021 ..
dr-x------ 13 1000 1000 4.0K júl   31 18:34 custom
drwx------  5 1000 1000 4.0K márc  26 00:21 git
drwx------  2    0    0 4.0K jún   13  2021 ssh

The relevant volume mapping in the docker-compose file:

[...]
volumes:
    gitea_data:
        driver: local
        driver_opts:
            type: none
            o: bind
            device: /mnt/gitea/data/
[...]
services:
    server:
        image: local/gitea/gitea:${GITEA_VERSION}
        container_name: gitea
        [...]
        volumes:
            - gitea_data:/var/lib/gitea
[...]

[wxiaoguang]

It really depends a lot of facts. Docker has root and rootless version. There doesn't seem to be an official document for it, I just searched and tried by myself before. So, the following information may not be accurate, just for your information.

For docker-with-root, the UID/GID/mode are stored with the file on the file system. I guess you are using the docker-with-root image. For docker-rootless, the UID/GID/mode are stored with xattr or NTFS extended attribute. Docker for Windows has more complex behaviors, depending on which backend is used (WSL2 or not), it's off-topic for this problem.

In my environment, the directories looks like this:

drwxr-xr-x  5 1000 1000 4.0K Aug 21  2021 git
drwxr-xr-x 17 1000 1000 4.0K Jul  8 16:47 gitea
drwx------  2 0    0    4.0K Aug  9  2021 ssh

Then the question is, why your custom directory always has the dr-x------ mode.

I think there could be some tests to try (if you have time ...)

• Stop the Gitea container
• Manually chmod 0700 custom on the host to see whether the mode can be changed. If no, then that's the root problem (filesystem).
• If the mode can be changed, then mount it to a simple container and start, exec into the shell to see whether the mode is correct. If no, then that's the root problem (docker).
• If the custom inside the simple container is correct, then stop the container to see whether the mode is still correct on the host.
• If everything is still correct, then mount the custom to the production Gitea container and start. If the mode becomes incorrect again, then it's the Gitea docker image or program problem.

[mpeter50]

@wxiaoguang thanks, I'll try these tests soon.

[mpeter50]

Oh, one question. I should do the with-Gitea parts of the test with the 1.17 image, right?
That is, the last test.

[wxiaoguang]

You could try it with 1.16.x first to see whether the behavior differs, in case it's a regression bug for 1.17? 😂

If you only has 1.17 database, only 1.17 is also fine, at least we can know the problem

[chris-fj]

@mpeter50 regarding to the question you asked me, yes, git user's UID is 116 and GID is 123

cut -d ':' -f1-4 /etc/passwd | grep git 
git:x:116:123

[mpeter50]

The tests are done, here are the results.

Docker bug test

First I checked if just mounting to a docker container changes the permissions.
I gave write permissions to the owner on the custom directory:

$ sudo chmod u+w custom/
$ ls -hal
total 20K
drwxr-xr-x  5 1000 1000 4.0K okt   29  2021 .
drwxr-xr-x  5 root root 4.0K jún   14  2021 ..
drwx------ 13 1000 1000 4.0K júl   31 18:34 custom
drwx------  5 1000 1000 4.0K márc  26 00:21 git
drwx------  2 root root 4.0K jún   13  2021 ssh

I started the container with this command:

sudo docker run -it --volume 'gitea_gitea_data:/mnt/gitea_data' debian bash

(gitea_gitea_data is a named volume stored at the location where I executed the previous commands)

Checked the permissions inside too, and stopped the container:

# cd /mnt/gitea_data/
# ls -hal
total 20K
drwxr-xr-x  5 1000 1000 4.0K Oct 29  2021 .
drwxr-xr-x  1 root root 4.0K Aug 19 07:27 ..
drwx------ 13 1000 1000 4.0K Jul 31 16:34 custom
drwx------  5 1000 1000 4.0K Mar 25 23:21 git
drwx------  2 root root 4.0K Jun 13  2021 ssh
# exit

Checked the permissions outside again:

$ ls -hal
total 20K
drwxr-xr-x  5 1000 1000 4.0K okt   29  2021 .
drwxr-xr-x  5 root root 4.0K jún   14  2021 ..
drwx------ 13 1000 1000 4.0K júl   31 18:34 custom
drwx------  5 1000 1000 4.0K márc  26 00:21 git
drwx------  2 root root 4.0K jún   13  2021 ssh

Gitea bug test

Out of curiosity first I tried with the current container (1.16.5).

I checked if the permissions are still ok, before starting the container:

$ ls -hal
total 20K
drwxr-xr-x  5 1000 1000 4.0K okt   29  2021 .
drwxr-xr-x  5 root root 4.0K jún   14  2021 ..
drwx------ 13 1000 1000 4.0K júl   31 18:34 custom
drwx------  5 1000 1000 4.0K márc  26 00:21 git
drwx------  2 root root 4.0K jún   13  2021 ssh

I started the container with docker-compose (if you're interested in the compose file, please let me know):

sudo docker-compose up

After the database and Gitea has finished starting up (~1 minute), I checked the permissions again:

$ ls -hal
total 20K
drwxr-xr-x  5 1000 1000 4.0K okt   29  2021 .
drwxr-xr-x  5 root root 4.0K jún   14  2021 ..
dr-x------ 13 1000 1000 4.0K júl   31 18:34 custom
drwx------  5 1000 1000 4.0K márc  26 00:21 git
drwx------  2 root root 4.0K jún   13  2021 ssh

The permissions have changed.

I also checked inside the container:

$ sudo docker exec -it gitea bash
bash-5.1$ ls -hal /var/lib/gitea/
total 20K
drwxr-xr-x    5 git      git         4.0K Oct 29  2021 .
drwxr-xr-x    1 root     root        4.0K Mar 24 11:49 ..
dr-x------   13 git      git         4.0K Jul 31 18:34 custom
drwx------    5 git      git         4.0K Mar 26 00:21 git
drwx------    2 root     root        4.0K Jun 13  2021 ssh

I also checked this again with 1.17.0.
I set the write permission again:

$ sudo chmod u+w custom/
$ ls -hal
total 20K
drwxr-xr-x  5 1000 1000 4.0K okt   29  2021 .
drwxr-xr-x  5 root root 4.0K jún   14  2021 ..
drwx------ 13 1000 1000 4.0K júl   31 18:34 custom
drwx------  5 1000 1000 4.0K márc  26 00:21 git
drwx------  2 root root 4.0K jún   13  2021 ssh

I changed the container image to start, and started the container:

sudo docker-compose up

When Gitea has started to start up and printed its first error messages, the permissions were changed again:

$ ls -hal
total 20K
drwxr-xr-x  5 1000 1000 4.0K okt   29  2021 .
drwxr-xr-x  5 root root 4.0K jún   14  2021 ..
dr-x------ 13 1000 1000 4.0K júl   31 18:34 custom
drwx------  5 1000 1000 4.0K márc  26 00:21 git
drwx------  2 root root 4.0K jún   13  2021 ssh

Conclusion

The strange behavior is connected to the Gitea container, but it looks it hasn't been introduced with 1.17.0, as the 1.16.5 container does this too.

[lunny]

search all codebase with os.MakeDir and os.MakeDirAll, couldn't find 500 permission. It's confusing.

[wxiaoguang]

search all codebase with os.MakeDir and os.MakeDirAll, couldn't find 500 permission. It's confusing.

In most cases, Gitea uses 0o777 + umask for the mode.

---

@mpeter50 Have you ever set some umask related things?

I can reproduce the behavior by:

func main() {
	syscall.Umask(0o277)
	os.MkdirAll("/tmp/test-dir/test", os.ModePerm) // or 0o755, etc
}
// then you will get: 
// dr-x------ 2 uid gid  64 Aug 19 16:10 test-dir/

Maybe you are using umask = 277 , if you are using a mount point, check the mount's arguments and make it 022 and take a look at it if there are other arguments affecting the modes.

[mpeter50]

@mpeter50 Have you ever set some umask related things?

On the host OS I have the umask set to 022:

$ umask
0022

The container has this too: (for being able to execute this command, I started 1.16.5)

sudo docker exec -it gitea bash
bash-5.1$ umask
0022

The directory where I store the data files of Gitea is stored on the root partition, and according to /etc/fstab it is mounted with defaults,noatime options, or rw,noatime according to mount (/dev/sda2 on / type ext4 (rw,noatime)).

Otherwise I don't remember touching the umask setting on this system.

[wxiaoguang]

Thank you for your time, I think I know the problem now.

I found this:

https://github.com/go-gitea/gitea/blob/main/docker/rootless/usr/local/bin/docker-setup.sh#L8

mkdir -p ${GITEA_CUSTOM} && chmod 0500 ${GITEA_CUSTOM}

😂

[wxiaoguang]

I proposed a PR for it:

• Fix the mode of custom dir to 0700 in docker-rootless #20861

But I haven't fully understand the problem at the moment.

[mpeter50]

Today I updated my instance to version 1.17.2, and everything seems fine.
Thanks again for looking into this!