Slowness with 100% CPU utilization #2889

AyhamZz · 2023-03-06T08:15:32Z

⚠️ Please verify that this bug has NOT been raised before.

I checked and didn't find similar issue

🛡️ Security Policy

I agree to have read this project Security Policy

Description

No response

👟 Reproduction steps

I noticed a huge slowness on my Uptime Kuma monitor and noticed that the CPU utilization is 100%, after increasing the CPU cores it still hits 100%, check the following screenshots:

and found the following Docker logs

am using Ubuntu 22.04 LTS

with the following docker version

Uptime kuma is on the latest version and i tried restarting both docker and server

👀 Expected behavior

CPU utilization caused by "node" might be the main reason of the slowness.

😓 Actual Behavior

Slowness

🐻 Uptime-Kuma Version

1.20.2

💻 Operating System and Arch

Ubunto 22.04 LTS

🌐 Browser

Google Chrome Version 110.0.5481.178

🐋 Docker Version

20.10.12

🟩 NodeJS Version

No response

📝 Relevant log output

Trace: Error: aborted
    at PendingOperation.abort (/app/node_modules/tarn/dist/PendingOperation.js:25:21)
    at /app/node_modules/tarn/dist/Pool.js:208:25
    at Array.map (<anonymous>)
    at /app/node_modules/tarn/dist/Pool.js:207:53
    at async Client_SQLite3.destroy (/app/node_modules/knex/lib/client.js:338:9)
    at async RedBeanNode.close (/app/node_modules/redbean-node/dist/redbean-node.js:375:9)
    at async Function.close (/app/server/database.js:435:13)
    at async Object.shutdownFunction [as onShutdown] (/app/server/server.js:1771:5)
    at process.<anonymous> (/app/server/server.js:1794:13)
    at process.emit (node:events:525:35)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: aborted
    at PendingOperation.abort (/app/node_modules/tarn/dist/PendingOperation.js:25:21)
    at /app/node_modules/tarn/dist/Pool.js:208:25
    at Array.map (<anonymous>)
    at /app/node_modules/tarn/dist/Pool.js:207:53
    at async Client_SQLite3.destroy (/app/node_modules/knex/lib/client.js:338:9)
    at async RedBeanNode.close (/app/node_modules/redbean-node/dist/redbean-node.js:375:9)
    at async Function.close (/app/server/database.js:435:13)
    at async Object.shutdownFunction [as onShutdown] (/app/server/server.js:1771:5)
    at process.<anonymous> (/app/server/server.js:1794:13)
    at process.emit (node:events:525:35)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: aborted
    at PendingOperation.abort (/app/node_modules/tarn/dist/PendingOperation.js:25:21)
    at /app/node_modules/tarn/dist/Pool.js:208:25
    at Array.map (<anonymous>)
    at /app/node_modules/tarn/dist/Pool.js:207:53
    at async Client_SQLite3.destroy (/app/node_modules/knex/lib/client.js:338:9)
    at async RedBeanNode.close (/app/node_modules/redbean-node/dist/redbean-node.js:375:9)
    at async Function.close (/app/server/database.js:435:13)
    at async Object.shutdownFunction [as onShutdown] (/app/server/server.js:1771:5)
    at process.<anonymous> (/app/server/server.js:1794:13)
    at process.emit (node:events:525:35)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: aborted
    at PendingOperation.abort (/app/node_modules/tarn/dist/PendingOperation.js:25:21)
    at /app/node_modules/tarn/dist/Pool.js:208:25
    at Array.map (<anonymous>)
    at /app/node_modules/tarn/dist/Pool.js:207:53
    at async Client_SQLite3.destroy (/app/node_modules/knex/lib/client.js:338:9)
    at async RedBeanNode.close (/app/node_modules/redbean-node/dist/redbean-node.js:375:9)
    at async Function.close (/app/server/database.js:435:13)
    at async Object.shutdownFunction [as onShutdown] (/app/server/server.js:1771:5)
    at process.<anonymous> (/app/server/server.js:1794:13)
    at process.emit (node:events:525:35)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: aborted
    at PendingOperation.abort (/app/node_modules/tarn/dist/PendingOperation.js:25:21)
    at /app/node_modules/tarn/dist/Pool.js:208:25
    at Array.map (<anonymous>)
    at /app/node_modules/tarn/dist/Pool.js:207:53
    at async Client_SQLite3.destroy (/app/node_modules/knex/lib/client.js:338:9)
    at async RedBeanNode.close (/app/node_modules/redbean-node/dist/redbean-node.js:375:9)
    at async Function.close (/app/server/database.js:435:13)
    at async Object.shutdownFunction [as onShutdown] (/app/server/server.js:1771:5)
    at process.<anonymous> (/app/server/server.js:1794:13)
    at process.emit (node:events:525:35)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: aborted
    at PendingOperation.abort (/app/node_modules/tarn/dist/PendingOperation.js:25:21)
    at /app/node_modules/tarn/dist/Pool.js:208:25
    at Array.map (<anonymous>)
    at /app/node_modules/tarn/dist/Pool.js:207:53
    at async Client_SQLite3.destroy (/app/node_modules/knex/lib/client.js:338:9)
    at async RedBeanNode.close (/app/node_modules/redbean-node/dist/redbean-node.js:375:9)
    at async Function.close (/app/server/database.js:435:13)
    at async Object.shutdownFunction [as onShutdown] (/app/server/server.js:1771:5)
    at process.<anonymous> (/app/server/server.js:1794:13)
    at process.emit (node:events:525:35)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: aborted
    at PendingOperation.abort (/app/node_modules/tarn/dist/PendingOperation.js:25:21)
    at /app/node_modules/tarn/dist/Pool.js:208:25
    at Array.map (<anonymous>)
    at /app/node_modules/tarn/dist/Pool.js:207:53
    at async Client_SQLite3.destroy (/app/node_modules/knex/lib/client.js:338:9)
    at async RedBeanNode.close (/app/node_modules/redbean-node/dist/redbean-node.js:375:9)
    at async Function.close (/app/server/database.js:435:13)
    at async Object.shutdownFunction [as onShutdown] (/app/server/server.js:1771:5)
    at process.<anonymous> (/app/server/server.js:1794:13)
    at process.emit (node:events:525:35)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
Trace: Error: aborted
    at PendingOperation.abort (/app/node_modules/tarn/dist/PendingOperation.js:25:21)
    at /app/node_modules/tarn/dist/Pool.js:208:25
    at Array.map (<anonymous>)
    at /app/node_modules/tarn/dist/Pool.js:207:53
    at async Client_SQLite3.destroy (/app/node_modules/knex/lib/client.js:338:9)
    at async RedBeanNode.close (/app/node_modules/redbean-node/dist/redbean-node.js:375:9)
    at async Function.close (/app/server/database.js:435:13)
    at async Object.shutdownFunction [as onShutdown] (/app/server/server.js:1771:5)
    at process.<anonymous> (/app/server/server.js:1794:13)
    at process.emit (node:events:525:35)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues

webartifex · 2023-03-08T14:02:15Z

I can confirm @AyhamZz 's observations.

For the past couple of days, Uptime is very, very slow.

For example, when you open it, it takes up to a minute until the monitors are shown.

I do have a recurring maintenance window. Yet, there were no issues when I set it up about 3-4 weeks ago. Must be a more recent version that causes the problems.

I run Uptime as a Docker container on an Ubuntu 22.04 LTS VM.

Since I am not really envolved on a technical level with your (great) project, @louislam , maybe you tell me how I can contribute to find and fix this bug.

louislam · 2023-03-08T14:25:04Z

@webartifex

I recently discovered that if a recurring maintenance ran into a wired state, it will keep generating time slots endlessly which can be slow down Uptime Kuma. My recommendation is you should delete the maintenance first.
If you are using Redis monitor, it also causes performance issue. Redis connections are not cleared #2900

webartifex · 2023-03-08T22:47:28Z

@louislam

I will try that out, disabling maintenance windows.

However, I have to admit that they are quite a useful feature.

I just found that my local SQLite with all the history was about 2 GB, even though I only monitor the last 7 days with about 50 monitors. Seems to confirm your finding that "it will keep generating time slots endlessly".

webartifex · 2023-03-09T01:15:18Z

@louislam

UPDATE: Disabling the maintenance windows works.

Also: Whoever faces the same issue, maybe take a look at the kuma.db file. Some tables may need to be emptied.

AyhamZz · 2023-03-13T05:58:55Z

@louislam

Thank you for your response, regarding the maintenance window I didn't use it until now, but it is worth mentioning that the issue just disappears whenever I clear the logs and export the backup file then import it again using overwrite.

I have a good resources attached to my VM

But for now, it is stable after what I mentioned, thank you again.

webartifex · 2023-03-13T09:23:57Z

@louislam

I saw the number of open tickets. Seems like a lot to do.

Do you plan to fix this maintenance window issue?
It's not urgent for me but would be nice to have in general.
I recommend your tool to a lot of friends starting home labs.

Btw: I bought a 5 Dollar sponsorship. Thanks for your effort.

louislam · 2023-03-13T09:45:08Z

@webartifex It should be fixed in 1.21.0-beta.1, feel free to try if you don't mind to use a beta version.

https://github.com/louislam/uptime-kuma/releases/tag/1.21.0-beta.1

petercharleston · 2023-04-05T11:18:25Z

Hi Louis,

I really like your product.

We have run into the same issues with CPU 100+ for the node command.

I deleted the MX schedules and also upgraded to 1.21.2

Still having very slow or not loading UI.

Adding or editing monitors is very very slow.

we have about 3530 monitors running and I doubled the VM spec today to 32GB RAM and 8Vcpu's which did not make a differrence.

DB size is 2234 MB
I set the monitor history to 7 days, it was on 180 days.

Any advise please?

petercharleston · 2023-04-05T11:57:52Z

update.

Followed advise from AyhamZz

export the backup file then import it again using overwrite.

we lost our status pages but this fixed the issue and shrank the DB very low from 2GB

louislam · 2023-04-05T12:18:19Z

we have about 3530 monitors running and I doubled the VM spec today to 32GB RAM and 8Vcpu's which did not make a differrence.

Unfortunately, 3530 monitors right now is too many for Uptime Kuma.

petercharleston · 2023-04-05T12:38:42Z

sorry that was meant to be 350 monitors.
All working fine now thanks

j-f1 · 2023-04-06T13:52:48Z

If other people run into these sorts of issues, you may be able to help by defining the environment variable NODE_OPTIONS with the value --cpu-prof --diagnostic-dir=/app/data. If you run Uptime Kuma for a short time and reproduce the slowness then exit it gracefully, the .cpuprofile file created in the data directory will provide detailed information about what was going on when it was running slowly. (the resulting profile file can be loaded into Chrome’s dev tools to inspect it)

Zandor300 · 2023-05-15T15:08:59Z

@louislam

Running uptime-kuma 1.21.3.

Experiencing the same slowness on my instance with ~90 monitors, only ping, http and push monitor types.

My db is 5 GB according to the UI and history was on 180 days. Changed it to 30 days but don't know when it will start removing old history?

@j-f1 Tried adding your NODE_OPTIONS to the Docker container but it spat out the following:

-v /root/uptime-kuma-prof:/app/data-prof \
--env NODE_OPTIONS="--cpu-prof --diagnostic-dir=/app/data-prof"

node: --cpu-prof is not allowed in NODE_OPTIONS

AyhamZz added the bug Something isn't working label Mar 6, 2023

This comment was marked as resolved.

Sign in to view

This was referenced Jul 27, 2023

Perf: Fully batch sendImportantHeartbeatList #3463

Closed

Upgraded from 1.19.1 to 1.22.1 and now keeps crashing #3500

Closed

CommanderStorm added the area:core issues describing changes to the core of uptime kuma label Dec 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Slowness with 100% CPU utilization #2889

Slowness with 100% CPU utilization #2889

AyhamZz commented Mar 6, 2023 •

edited by CommanderStorm

Loading

This comment was marked as resolved.

This comment was marked as resolved.

webartifex commented Mar 8, 2023 •

edited

Loading

louislam commented Mar 8, 2023

webartifex commented Mar 8, 2023

webartifex commented Mar 9, 2023

AyhamZz commented Mar 13, 2023

webartifex commented Mar 13, 2023

louislam commented Mar 13, 2023

petercharleston commented Apr 5, 2023

petercharleston commented Apr 5, 2023

louislam commented Apr 5, 2023 •

edited by CommanderStorm

Loading

petercharleston commented Apr 5, 2023

j-f1 commented Apr 6, 2023 •

edited

Loading

Zandor300 commented May 15, 2023

Slowness with 100% CPU utilization #2889

Slowness with 100% CPU utilization #2889

Comments

AyhamZz commented Mar 6, 2023 • edited by CommanderStorm Loading

⚠️ Please verify that this bug has NOT been raised before.

🛡️ Security Policy

Description

👟 Reproduction steps

👀 Expected behavior

😓 Actual Behavior

🐻 Uptime-Kuma Version

💻 Operating System and Arch

🌐 Browser

🐋 Docker Version

🟩 NodeJS Version

This comment was marked as resolved.

This comment was marked as resolved.

webartifex commented Mar 8, 2023 • edited Loading

louislam commented Mar 8, 2023

webartifex commented Mar 8, 2023

webartifex commented Mar 9, 2023

AyhamZz commented Mar 13, 2023

webartifex commented Mar 13, 2023

louislam commented Mar 13, 2023

petercharleston commented Apr 5, 2023

petercharleston commented Apr 5, 2023

louislam commented Apr 5, 2023 • edited by CommanderStorm Loading

petercharleston commented Apr 5, 2023

j-f1 commented Apr 6, 2023 • edited Loading

Zandor300 commented May 15, 2023

AyhamZz commented Mar 6, 2023 •

edited by CommanderStorm

Loading

webartifex commented Mar 8, 2023 •

edited

Loading

louislam commented Apr 5, 2023 •

edited by CommanderStorm

Loading

j-f1 commented Apr 6, 2023 •

edited

Loading