During git clone, gitea fails with accept4: too many open files; retrying in 1s

[galets]

Description

Description

While pulling a large repository with LFS, Gitea instance stops responding. Website shows "500 Internal Server Error". Log shows (initially):

Oct 13 14:20:34 gitea gitea[75]: 2025/10/13 14:20:34 HTTPRequest [I] router: completed GET /api/internal/repo/galets/documents.git/info/lfs/objects/9634ecdcbb32d105bb617de1bd6e93bc4d78bef0f76a8890f862b3f56ce6056e for 10.17.0.2:0, 200 OK in 94.2ms @ lfs/server.go:83(lfs.DownloadHandler)
Oct 13 14:20:34 gitea gitea[75]: 2025/10/13 14:20:34 HTTPRequest [I] router: completed GET /api/internal/repo/galets/documents.git/info/lfs/objects/97968b00ff273fd1a85e53a793befba5a6e6dd0ad8486998f156b5c73cdbf484 for 10.17.0.2:0, 200 OK in 152.2ms @ lfs/server.go:83(lfs.DownloadHandler)
Oct 13 14:20:35 gitea gitea[75]: 2025/10/13 14:20:35 HTTPRequest [W] router: slow      GET /api/internal/repo/galets/documents.git/info/lfs/objects/8583e862b3ed1b0a6b7519dc237383ae58c1d97aac85acca0ccc0bf04a949e81 for 10.17.0.2:0, elapsed 3475.9ms @ lfs/server.go:83(lfs.DownloadHandler)
Oct 13 14:20:38 gitea gitea[75]: 2025/10/13 14:20:38 HTTPRequest [I] router: completed GET /api/internal/repo/galets/documents.git/info/lfs/objects/8583e862b3ed1b0a6b7519dc237383ae58c1d97aac85acca0ccc0bf04a949e81 for 10.17.0.2:0, 200 OK in 6255.2ms @ lfs/server.go:83(lfs.DownloadHandler)
Oct 13 14:20:38 gitea gitea[75]: 2025/10/13 14:20:38 HTTPRequest [W] router: slow      GET /api/internal/repo/galets/documents.git/info/lfs/objects/983f1289778dfa7964d4953f9312270353f958a14ca87b56c5ef6c6669fd1d25 for 10.17.0.2:0, elapsed 3234.2ms @ lfs/server.go:83(lfs.DownloadHandler)

Then:

Oct 13 14:14:28 gitea gitea[75]: 2025/10/13 14:14:28 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:29 gitea gitea[75]: 2025/10/13 14:14:29 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:30 gitea gitea[75]: 2025/10/13 14:14:30 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:31 gitea gitea[75]: 2025/10/13 14:14:31 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:32 gitea gitea[75]: 2025/10/13 14:14:32 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:33 gitea gitea[75]: 2025/10/13 14:14:33 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:34 gitea gitea[75]: 2025/10/13 14:14:34 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:35 gitea gitea[75]: 2025/10/13 14:14:35 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:36 gitea gitea[75]: 2025/10/13 14:14:36 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:37 gitea gitea[75]: 2025/10/13 14:14:37 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:38 gitea gitea[75]: 2025/10/13 14:14:38 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:39 gitea gitea[75]: 2025/10/13 14:14:39 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:40 gitea gitea[75]: 2025/10/13 14:14:40 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s
Oct 13 14:14:41 gitea gitea[75]: 2025/10/13 14:14:41 modules/log/misc.go:71:(*loggerToWriter).Write() [I] http: Accept error: accept tcp [::]:3000: accept4: too many open files; retrying in 1s

git clone shows:

$ git clone ssh://gitea@*********/galets/documents.git
Cloning into 'documents'...
remote: Enumerating objects: 36549, done.
remote: Counting objects: 100% (36549/36549), done.
remote: Compressing objects: 100% (24795/24795), done.
remote: Total 36549 (delta 10668), reused 36495 (delta 10648), pack-reused 0 (from 0)
Receiving objects: 100% (36549/36549), 10.79 MiB | 12.14 MiB/s, done.
Resolving deltas: 100% (10668/10668), done.
Updating files: 100% (15543/15543), done.
Filtering content:  31% (4266/13761), 12.21 GiB | 17.92 MiB/s4 MiB/s

Once gitea server goes into that condition, it does not self-recover

Also:

$ ulimit -u
578039
$ cat /etc/systemd/system.conf | grep NOFILE
DefaultLimitNOFILE=65535:524288

Gitea Version

v1.24.6

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

Ubuntu 22.04 / Debian 13

How are you running Gitea?

Gitea is running in systemd container (nspawn). Host is running Ubuntu 22.04. Container is running Debian 13.

Database

SQLite

[wxiaoguang]

It seems that there are some resource leaking.

[wxiaoguang]

Propose a PR to try to fix: LFS SSH: Fix missing Close when error occurs and add more tests #35658

If you can collect more information (for example: when error occurs, what are the leaked resources? file description or tcp socket connection), it would be very helpful.

[galets]

I have environment where the issue is reproducible. Can you write me up the steps I need to follow to collect the data you want?

Also, would you like me to deploy the fix you proposed and see if it resolves the issue?

[wxiaoguang]

Some tools like ss or lsof can help to list the open FD or TCP sockets in a process.

For example: ss -anp and lsof -anp $(pidof gitea)

[galets]

I dumped the whole lsof, as there weren't many other processes in the container

(Update: deleted the log attachment)

Please let me know if you need anything else. I will restart gitea in a few hours

[galets]

Also, something occured to me worth mentioning: a lot of exhausted connections belong to web ui process, not workers:

# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 Oct13 ?        00:00:01 /usr/lib/systemd/systemd
root          16       1  0 Oct13 ?        00:00:42 /usr/lib/systemd/systemd-journald
systemd+      42       1  0 Oct13 ?        00:00:01 /usr/lib/systemd/systemd-networkd
root          65       1  0 Oct13 ?        00:00:00 /usr/sbin/cron -f
message+      66       1  0 Oct13 ?        00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root          67       1  0 Oct13 ?        00:00:00 /usr/lib/systemd/systemd-logind
gitea         69       1 21 Oct13 ?        07:05:42 /usr/local/lib/gitea/gitea web --config /usr/local/var/gitea/custom/conf/app.ini
root          87       1  0 Oct13 pts/0    00:00:00 /sbin/agetty -o -- \u --noreset --noclear --keep-baud 115200,57600,38400,9600 - vt220
root          90       1  0 Oct13 ?        00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root         228       1  0 Oct13 ?        00:00:00 /usr/lib/systemd/systemd --user
root         230     228  0 Oct13 ?        00:00:00 (sd-pam)
gitea       2128       1  0 23:22 ?        00:00:00 /usr/lib/systemd/systemd --user
gitea       2130    2128  0 23:22 ?        00:00:00 (sd-pam)
gitea       2203       1  1 23:22 ?        00:00:31 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2230       1  0 23:22 ?        00:00:18 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2257       1  0 23:22 ?        00:00:14 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2286       1  0 23:22 ?        00:00:19 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2311       1  0 23:22 ?        00:00:15 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2333       1  1 23:22 ?        00:00:22 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2359       1  0 23:22 ?        00:00:19 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2385       1  1 23:22 ?        00:00:43 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
root        2442       1  0 23:38 pts/2    00:00:00 /bin/bash -l
root        2444    2442  0 23:38 pts/2    00:00:00 (sd-
root        2573    2442  0 23:59 pts/2    00:00:00 ps -ef

[wxiaoguang]

Hmm, just realized that it's difficult to "catch" a snapshot when the resource leaking occurs.

The reason is that the leaking occurs in the "git hook gitea subcommand" (but not gitea server), when the leaking occurs, git client is likely to exit soon, then the "git hook gitea subcommand" also exits, then everything recovers. It needs a quite good timing to catch the resource leaking in "git hook gitea subcommand" ....

[wxiaoguang]

I dumped the whole lsof, as there weren't many other processes in the container

gitea-lsof.txt.gz

Please let me know if you need anything else. I will restart gitea in a few hours

OK, I think we can know the problem is caused by the gitea subcommand to gitea server connection.

There are plenty of these logs:

gitea       69   81 gitea             gitea 3459u     IPv6           31151935       0t0        TCP localhost:3000->localhost:36762 (ESTABLISHED)

I guess "Fix missing Close when error occurs and add more tests #35658" might be able to fix. You can restart the server :)

[galets]

Would you like me to deploy the commit that fixes is to test?

[wxiaoguang]

Would you like me to deploy the commit that fixes is to test?

If you are able to build your own docker image and deploy, it would be quite helpful. Thank you very much.

(And I am still working on it to try to debug more details)

[galets]

Can you supply exactly the commit hash?

[wxiaoguang]

You can checkout my branch: https://github.com/wxiaoguang/gitea/tree/fix-lfs-ssh

At the moment there is only one commit db70766. I might add more commits later if I find something new.