-
-
Notifications
You must be signed in to change notification settings - Fork 374
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
too many open file handles #351
Comments
This can be worked around via "ulimit -n 64000" however it seems like aptly is keeping a lot of file handles open... |
@avongluck-r1soft I was trying to figure out the problem here... so you are adding packages via CLI aptly while API aptly is running? Is it related to API aptly? |
Also this is inefficient way to add packages (very slow), aptly can do much better job by importing more packages at once, it can do iteration on its own like |
I had a similar issue yesterday - had been uploading debs via REST. The service had been up for a long time (weeks if not months) and there's usually about half-a-dozen uploads per day from our build system. The REST API was accepting the debs (reporting that it uploaded them), but after reporting it published the resulting snapshot, the debs weren't actually available for the clients. In the log, I was getting GPG errors due to open files.
Our setup only ever uploads one package at a time, and a restart fixed the issue. My guess is that the problem is not the bulk upload itself, but the number of open filehandles - and in our case, it had just grown over time. This was all seen with default setting for nofiles (1024 soft 4096 hard) and aptly 0.9.5 |
So aptly should have fd leak somewhere. Thanks I'll keep looking. |
@smira, I was actually just using This is aptly 1.3.0 running on 16.04.4. Thankfully my repos are published to S3 and I had a snapshot of the instance volume from this morning to recover the instance itself. But it would be great to find out why this is happening before I attempt it again. |
@jalmansor just trying to understand what is going on there. as aptly via |
We are having a similar issue. Aptly has been in use for 2 years. We recently started testing using the API (with -no-lock) and are encountering this issue doing snapshot/db cleanups. errors include "too many open file handles" Aptly repos are managed by bash scripts run by Jenkins. At the command line (after stopping aptly server): Our API version is just the same commands with the API equivalent of CLI ( I have added NOTE: Running |
on a side note, I don't think having LevelDB on NFS is really good scenario (not sure about locking and consistency). I'm thinking about what is triggering this open file exhaustion though. Looks to be related to number of |
I tried to reproduce this by creating and dropping 2000 snapshots, but number of files in the db directory stays stable like it should (testing with |
Created 10GB worth of database, 3.5k files, ulimit set to 384, still no failures |
|
Looks like I was able to hit that, but with forced compaction and |
but seems to be |
Summary:
Plan:
|
Looks like the db got corrupted yesterday, jobs failed overnight based on this error: `aptly@fulcrum-1:~$ aptly publish list ERROR: unable to load list of repos: snapshot with uuid 66ed3ea5-2931-4e91-8c8d-8a0f6255de0c not found` Listing snapshots, repos, etc worked fine. |
Note: Re:- "stale NFS file handle" messages |
#882 might help here as well |
Hi @biggreenogre , did you retry with recent code ? Regards |
I have not. I'm afraid I don;t have any experience with GO or how to build it. I also need the fix for the S3/MD5SUM issue. |
I'm hit by the open file handles bug (triggers every 2 to 5 days). And starting today I'm running Will report back in about a month if it worked 🤞 |
how are you running aptly ? via command or API ? for commands, what is your |
I created a limits file to raise the limit for the whole server sudo tee /etc/security/limits.d/20-nprocs.conf << EOF
# raise number of opened file per process for each user to 256k (from default 1024)
* soft nofile 256000
root soft nofile 256000
EOF And I'm running the API with sudo tee /etc/systemd/system/aptly-api.service << EOF
[Unit]
Description=Aptly REST API service
[Service]
User=aptly
WorkingDirectory=/var/lib/aptly
ExecStart=/usr/bin/aptly api serve -listen "localhost:8000" -no-lock
KillSignal=SIGTERM
KillMode=process
TimeoutStopSec=15s
[Install]
WantedBy=multi-user.target
EOF Is that enough? Or do I still need to set and weekly I'm running
|
yes, when aptly is running as systemd service, then LimitNOFILE=256000 needs to be in the service configuration or overlay. the cron environment variable might come from /etc/security/limits.d/ |
Thank you very much! This explains why the "too-many-open-files" bug was triggered in about the same time regardless of which nolimit I set in For future reference, after the restart of the aptly-api systemd-service I get the following for open files $ ls -l /var/lib/aptly/db/*.ldb | wc -l
123 |
I think it is expected that the db files are open. 123 is also not a huge number... Using May I close the issue ? |
I think yes, you can. I'll report back in about a month (or earlier if the bug triggers again) For future reference the number of open files for about 2 hours runtime two new *.ldb files were created $ ls -l /var/lib/aptly/db/*.ldb | wc -l
125 I don't understand $ sudo lsof -p $(pidof aptly)
[sudo] password for builder:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
aptly 7237 aptly cwd DIR 0,36 208 256 /var/lib/aptly
aptly 7237 aptly rtd DIR 253,2 4096 2 /
aptly 7237 aptly txt REG 253,2 37622688 2518976 /usr/bin/aptly
aptly 7237 aptly 0r CHR 1,3 0t0 5 /dev/null
aptly 7237 aptly 1u unix 0xffff8d8a4a38a800 0t0 54916 type=STREAM (CONNECTED)
aptly 7237 aptly 2u unix 0xffff8d8a4a38a800 0t0 54916 type=STREAM (CONNECTED)
aptly 7237 aptly 3u unix 0xffff8d8aa84cec00 0t0 56498 type=DGRAM (CONNECTED)
aptly 7237 aptly 4u a_inode 0,15 0 64 [eventpoll:3,5,7]
aptly 7237 aptly 5r FIFO 0,14 0t0 56497 pipe
aptly 7237 aptly 6w FIFO 0,14 0t0 56497 pipe
aptly 7237 aptly 7u IPv4 54917 0t0 TCP localhost:8000 (LISTEN) I found another way to list the open files of a process here: https://stackoverflow.com/questions/21752067/counting-open-files-per-process and its only 8? 😅 $ sudo ls /proc/$(pidof aptly)/fd/ 2>/dev/null | wc -l
8 |
The file limit now is fixed (I believe), but aptly-api Can't check if the number of open files was stable or growing as earlier as the process is killed.
|
OOM issue is most likely #1323 |
yes, there was a slip which did not free memory on each API call, but this is fixed in the nightly builds. the number of open files should be low, when aptly is not doing anything. the 8 FDs (number 0-7 above) are do you still have issues with open files, or can we close this ? |
thanks for the explanation! Yes I think this issue is resolved! Thanks! |
When adding a large number of deb's in a row, aptly will begin throwing "too many open files" while API mode is running.
(aptly running in API mode with no-lock)
find /tmp/iso/pool -name "_._deb" -exec aptly repo add $2 {} ;
[+] php5-cli_5.5.9+dfsg-1ubuntu4.3_amd64 added Loading packages... [+] php5-common_5.5.9+dfsg-1ubuntu4.3_amd64 added Loading packages... [+] php5-curl_5.5.9+dfsg-1ubuntu4.3_amd64 added Loading packages... [+] php5-gd_5.5.9+dfsg-1ubuntu4.3_amd64 added Loading packages... [+] php5-gmp_5.5.9+dfsg-1ubuntu4.3_amd64 added Loading packages... [+] php5-ldap_5.5.9+dfsg-1ubuntu4.3_amd64 added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-mysql_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/23/57/php5-mysql_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-mysql_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-odbc_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/50/0c/php5-odbc_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-odbc_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-pgsql_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/cb/8e/php5-pgsql_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-pgsql_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-pspell_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/5b/4d/php5-pspell_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-pspell_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-readline_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/ba/ec/php5-readline_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-readline_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-recode_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/76/eb/php5-recode_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-recode_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-snmp_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/19/05/php5-snmp_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-snmp_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-sqlite_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/8c/cc/php5-sqlite_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-sqlite_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-tidy_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/f8/55/php5-tidy_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-tidy_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-xmlrpc_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/ec/c7/php5-xmlrpc_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-xmlrpc_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php5-xsl_5.5.9+dfsg-1ubuntu4.3_amd64.deb into pool: open /repo/.aptly/pool/c5/74/php5-xsl_5.5.9+dfsg-1ubuntu4.3_amd64.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php5-xsl_5.5.9+dfsg-1ubuntu4.3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to import file /tmp/iso/pool/main/p/php5/php-pear_5.5.9+dfsg-1ubuntu4.3_all.deb into pool: open /repo/.aptly/pool/eb/2a/php-pear_5.5.9+dfsg-1ubuntu4.3_all.deb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php5/php-pear_5.5.9+dfsg-1ubuntu4.3_all.deb ERROR: some files failed to be added Loading packages... [!] Unable to save package php5-json_1.3.2-2build1_amd64: open /repo/.aptly/db/018015.ldb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/php-json/php5-json_1.3.2-2build1_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to save package python-pil_2.3.0-1ubuntu3_amd64: open /repo/.aptly/db/018015.ldb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/pillow/python-pil_2.3.0-1ubuntu3_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to save package libpixman-1-0_0.30.2-2ubuntu1_amd64: open /repo/.aptly/db/018015.ldb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/pixman/libpixman-1-0_0.30.2-2ubuntu1_amd64.deb ERROR: some files failed to be added Loading packages... [!] Unable to save package libpkcs11-helper1_1.11-1_amd64: open /repo/.aptly/db/018015.ldb: too many open files [!] Some files were skipped due to errors: /tmp/iso/pool/main/p/pkcs11-helper/libpkcs11-helper1_1.11-1_amd64.deb ERROR: some files failed to be added Loading packages...
The text was updated successfully, but these errors were encountered: