Unusually high memory usage #11879
Comments
|
I have also experienced much higher memory usage after updating to 4.2.1 from 4.1.9 OS: Arch Linux Memory usage keeps increasing for a couple of hours until the system freezes and finally gets killed by the OOM killer. 13 GB memory usage after a few hours. In version 4.1.9 the memory usage was constant and it could run indefinitely without any issues. I have also noticed that every time i select Edit -> Pause all/Resume all, the memory usage jumps about 100 MB. |
|
I reverted to 4.1.9.1, rechecked all files, restarted qb and still had the memory issue, kind of strange. |
|
Everyone in this thread please see https://github.com/qbittorrent/qBittorrent/blob/master/CONTRIBUTING.md. Try to provide more data such as the qBittorrent logs, as well as testing if you can reproduce this on a clean install of qBittorrent (since you all mention this started happening after upgrading/reverting). "Clean installing qBittorrent" means deleting the |
|
I renamed the folders. Then I added a couple of new torrents. But after seeding for about 2 hours I did not notice any increase in memory at all. Toggling Log: qbittorrent.log I renamed the folders again. But restored the original Log: qbittorrent_2.log Not sure how to proceed from here as I don't want to share my |
That log looks a bit strange still, because of the Also, in addition to the backup folder, did you reset the settings as well when testing? Just for completeness, it would be good to be able to rule that out.
That log looks fine, assuming
Understandable. At this point, the memory leak could be due to the differences between I am not sure how you would go about "cleanly" regenerating the @glassez @arvidn Ideas? How can one regenerate the |
|
I added the torrents by directly opening the .torrent files from chrome's download bar. Probably why it says started multiple times. I have 202 torrents and the log didn't mention any error for any of them when restoring. For the second test i had renamed both I ran I can try deleting the |
|
In my experience, the simplest way to diagnose memory leaks is to run a build with symbols and ideally some debug information, and Then, |
|
@mrxdst want to try the suggestion above? I can't reproduce the problem right now, so if you could help that would be great. With your help, we might be able to pinpoint the cause of the issue and perhaps come up with a fix. |
|
I had a go at it. I'm not familiar with c++ development but i think i figured it out. Full test results: From what I can tell, the following appears to use 80% of all memory: (From test 4)
|
Excellent job, thanks for testing. I assume you used the problematic |
|
@FranciscoPombal |
|
I was right. Selecting all torrents and choosing After that i changed the following settings:
It let me keep all torrents resumed, hopefully without any other traffic.
I tried narrowing it down to see if it could be a specific tracker but I was unfortunately unable to get any consistent results. I don't see a way to remove a single tracker from all torrents at once, So I tried a much smaller set of torrents. However it made i much harder to notice the leaks consistently. When using a smaller it didn't leak every time, only sometimes. But with a larger set it always leaked consistently. Also. this might not be a |
|
@arvidn @FranciscoPombal a quick memory pressure analysis report of any use?! (this is with a 4.2.1 test build) |
|
Ok, so looks like the See if you can narrow it down to a specific tracker, or at least a specific tracker type (http, https or udp). Since the leak apparently has to do with openssl, I'm expecting that it only happens with https trackers. |
|
@xavier2k6 P.S. - a more standard file format would have been preferred - I had to spin up my Windows VM to read that report. |
|
I've tried removing all Best bet would probably be to compare trackers to someone else having the same issue, but for now it appears to be only me, so not much use. I'm considering just deleting all trackers from all torrents and starting fresh. Should probably fix it, but then we won't know what exactly caused it. |
It is a report with leaktrack attached to qbittorrent process for the potential of seeing any memory leaks.
qBittorrent was running with 20+ torrents that have seperate re-announce to trackers times, none were forced though & had a combinantion of http/https & udp trackers on each torrent.....
This may be so & the basis of the report is to just show if there is a potential leak problem/serve as a basic guideline...(It may or may not be insightful, but hey - anything is better than nothing) It shows a potential probability......whether that is acceptable within the confines of the code/parameters - I can't say....(so could very well be a false positive so to speak.)
Unfortunately that's the format Debug Diag Analysis uses to generate it's reports. |
|
The leak probability is only a suggestion really. Otherwise, I guess this is a good way of generating such reports on Windows (I don't know of a better solution myself). Maybe you can create a wiki page detailing the process?
If you cannot select a different output type, it is possible at least to use Internet Explorer to Save As... -> webpage, full (or something similar). This will save it as a normal html document. Can you try running qBittorrent with the problematic valgrind --tool=memcheck --verbose --leak-check=full --show-leak-kinds=all --track-origins=yes --log-file=valgrind_output.mem qbittorrentvalgrind --tool=massif --verbose --massif-out-file=massif_output.mass qbittorrent# start with instrumentation off for faster startup
valgrind --tool=callgrind --verbose --log-file=callgrind.log --instr-atstart=no --dump-instr=yes --cache-sim=yes --cacheuse=yes qbittorrent
# from a different terminal, once program has finished starting:
callgrind_control -i on # turn on instrumentation
# after triggering the leaky behavior and seeing memory consumption go up, dump data:
callgrind_control -d qbittorrent
# callgrind will also generate and additional dump on exit |
|
If this issue seems to be related to https trackers, then I can update with what would have caused it on my machine. I found some long term stored torrents I have were using the sites old http tracker that now has an https tracker. These are all on a single private tracker that upgraded to an https tracker sometime since I added the files to qb years ago. I decided to update all the http trackers to https. This was over 125 torrents. All the private keys in the addresses were still the same, the only thing that changed was adding https. So maybe it has something to do with the fact that all these original torrents have never been associated with the https tracker and how the site is handling the requests/announces. Or that the original trackers are no longer available as I do not store any torrents since I migrated to qb and just use qb's built in resume files strictly. All these torrents are really small in size, as they are only stored to provide a ratio boost on the site which gives a bonus for each completed torrent seeded. Its just under 7gb of files over 175 total torrents for this site. I visit the site at the end of each year and sort all their torrents by size and scan the list looking for any torrents from the current year that are smaller in size to reduce the long term storage footprint. I can redownload all the torrents and they will use their old tracker. So I guess thats my plan for now to hopefully solve the issue until a potential resolution is found. I don't believe I had memory issues after making the changes on 4.1.9.1 at the close of 2019 and that the issue did only surface after updating to 4.2.0. Thanks for everyones efforts. |
|
@eigerzoom |
|
I agree, thats why I updated them, but they were on http for an ungodly amount of time anyway so if it solves the memory leak temporarily I will sacrifice a little security for functionality. Maybe simply redownloading all the torrents will override the issue and I can update them all to https again. I will do some testing and report back if it helps. |
|
Currently back on 4.1.9.1. So I removed all the manually updated torrents, restarted qb, and monitored memory usage, it didn't rise above 100mb or so (expected memory usage). I re-added all the torrents directly from trackers website and they all used the newer https tracker. Memory increased dramatically. Restarted qb and memory kept rising up over 1gb. I added an xtra 25 torrents from the site as well. I should add that I store all these torrents in a category/subfolder and are all auto-managed by qb. |
|
Ok so now you just have to repeat the same thing but with the http tracker and compare memory usage. |
|
disk cache is set to 1000 MiB |
Then you should expect qbittorrent to consume 1 GIB of ram just because of that. By the way, that is an excessive value. Just set it to auto. |
|
The disk cache set to auto seems to have solved it on its own, it never used this much memory before though for the record. Thanks. |
|
Back on 4.2.1 with the same behavior. What I do notice is that if I do not increase the disk cache manually public torrent transfers suffer on large swarms and do not achieve quality transfer speeds. Would this have to do with nic settings or possibly a combination of qb settings? Thanks for any insight. |
|
I set up the test like this: I ran the test like this: Results: |
Thank you for this. My analysis:
This means 6 GiB were allocated in total, but at exit only about 1 MiB is potentially leaking. This means the main memory leak we're hunting does not make it all the way to program exit. At some point, that memory is correctly
It shows a peak of about 780 MiB heap usage. Is this consistent with the usage you saw on some sort of task manager when executing?
Anyway, it confirms the leak has to do with either
Output was not as useful as I'd hoped for, since there were no debug symbols for libtorrent. Overall the massif report seems to be the most useful in this case. Thanks again for the effort you're putting into this. |
|
if there's an option in qBitTorrent to disable SSL torrents ( |
|
@mrxdst Excellent work!
I wonder how valid these findings are. IIRC, "Not working" in qBittorrent's status column does not necessarily mean that the tracker is not working or not working correctly. It just means the tracker does not know about that particular torrent you added them to. Some trackers report a "Not found"/"Unregistered torrent" or equivalent, but that is optional and depends on the software they are running. @arvidn Not currently. I grep'd the code base for |
|
Ok here is the patch to enable qbittorrent to log The PKGBUILD can be modified to apply the patch: https://wiki.archlinux.org/index.php/Patching_packages#Applying_patches diff --git a/src/base/bittorrent/session.cpp b/src/base/bittorrent/session.cpp
index 0000dc029..9a468ae65 100644
--- a/src/base/bittorrent/session.cpp
+++ b/src/base/bittorrent/session.cpp
@@ -1064,6 +1064,7 @@ void Session::configureComponents()
void Session::initializeNativeSession()
{
const LTAlertCategory alertMask = lt::alert::error_notification
+ | lt::alert::torrent_log_notification
| lt::alert::file_progress_notification
| lt::alert::ip_block_notification
| lt::alert::peer_notification
@@ -4289,6 +4290,9 @@ void Session::handleAlert(const lt::alert *a)
case lt::external_ip_alert::alert_type:
handleExternalIPAlert(static_cast<const lt::external_ip_alert*>(a));
break;
+ case lt::torrent_log_alert::alert_type:
+ handleTorrentLogAlert(static_cast<const lt::torrent_log_alert*>(a));
+ break;
#if (LIBTORRENT_VERSION_NUM >= 10200)
case lt::alerts_dropped_alert::alert_type:
handleAlertsDroppedAlert(static_cast<const lt::alerts_dropped_alert *>(a));
@@ -4418,6 +4422,11 @@ void Session::handleTorrentDeletedAlert(const lt::torrent_deleted_alert *p)
m_removingTorrents.erase(removingTorrentDataIter);
}
+void Session::handleTorrentLogAlert(const lt::torrent_log_alert *p)
+{
+ LogMsg(tr("torrent_log_alert: '%1'").arg(QString::fromStdString(p->message()), Log::INFO));
+}
+
void Session::handleTorrentDeleteFailedAlert(const lt::torrent_delete_failed_alert *p)
{
const InfoHash infoHash {p->info_hash};
diff --git a/src/base/bittorrent/session.h b/src/base/bittorrent/session.h
index a139e9d95..31c9e2c56 100644
--- a/src/base/bittorrent/session.h
+++ b/src/base/bittorrent/session.h
@@ -579,6 +579,7 @@ namespace BitTorrent
void handleListenFailedAlert(const lt::listen_failed_alert *p);
void handleExternalIPAlert(const lt::external_ip_alert *p);
void handleSessionStatsAlert(const lt::session_stats_alert *p);
+ void handleTorrentLogAlert(const lt::torrent_log_alert *p);
#if (LIBTORRENT_VERSION_NUM >= 10200)
void handleAlertsDroppedAlert(const lt::alerts_dropped_alert *p) const;
#endif
|
|
yes, that will be quite verbose. actually, it occurred to me that there should have been a |
|
@FranciscoPombal I deleted half of the torrents in @arvidn I''ll email you the logs. Maybe they are useful. I'll happily try again if a patch is needed for Also: |
|
the naming of the log files don't seem to be chronological. I did this to stitch them back together: The problem is that the first column is |
|
@mrxdst it looks like you have a lot of duplicate entries for all your trackers. is that the case? I'm not done analyzing the logs yet, but it looks like one of the trackers has more than 50 duplicate entries. |
|
@arvidn |
|
I see. I think I know a bit more about what's going on. I think at least some of this has been addressed already in libtorrent-1.2.4 (which hasn't been released yet, but lives in the All of the "duplicates" I see in the log are caused by you ending up having a lot of listen endpoints (which is one of the main issues in libtorrent-1.2.x that was recently fixed). My logging isn't great right now either, I'll improve that. I'll also look into the issue you pointed out about the common name of the SSL cert failing. It's possible that this failure hasn't seen a lot of testing. |
qBittorrent currently does add both of those notifications to the alert mask. However, nothing is logged for the I tried testing with just one torrent (some linux distro) and adding the problematic trackers list posted above: https://github.com/qbittorrent/qBittorrent/files/4142696/https_trackers.txt This was enough to produce pretty big logs with lots of errors, and may be enough to reproduce the problem in the same way, without having to deal with many torrent s and hundreds of MiBs/GiBs of logs. |
|
@arvidn |
|
@FranciscoPombal that sounds like an ABI incompatibility with boost.system. Make sure everything is built with the same version of C++, including boost.system. |
|
EDIT: turns out this patch was failing for unrelated reasons. In statements like @arvidn Here is the backtrace obtained in gdb:
diff --git a/src/base/bittorrent/session.cpp b/src/base/bittorrent/session.cpp
index 0000dc029..5a537eb69 100644
--- a/src/base/bittorrent/session.cpp
+++ b/src/base/bittorrent/session.cpp
@@ -1064,6 +1064,7 @@ void Session::configureComponents()
void Session::initializeNativeSession()
{
const LTAlertCategory alertMask = lt::alert::error_notification
+ | lt::alert::torrent_log_notification
| lt::alert::file_progress_notification
| lt::alert::ip_block_notification
| lt::alert::peer_notification
@@ -4237,6 +4238,7 @@ void Session::handleAlert(const lt::alert *a)
case lt::torrent_paused_alert::alert_type:
case lt::torrent_resumed_alert::alert_type:
case lt::tracker_error_alert::alert_type:
+ handleTrackerErrorAlert(static_cast<const lt::tracker_error_alert*>(a));
case lt::tracker_reply_alert::alert_type:
case lt::tracker_warning_alert::alert_type:
case lt::fastresume_rejected_alert::alert_type:
@@ -4289,6 +4291,9 @@ void Session::handleAlert(const lt::alert *a)
case lt::external_ip_alert::alert_type:
handleExternalIPAlert(static_cast<const lt::external_ip_alert*>(a));
break;
+ case lt::torrent_log_alert::alert_type:
+ handleTorrentLogAlert(static_cast<const lt::torrent_log_alert*>(a));
+ break;
#if (LIBTORRENT_VERSION_NUM >= 10200)
case lt::alerts_dropped_alert::alert_type:
handleAlertsDroppedAlert(static_cast<const lt::alerts_dropped_alert *>(a));
@@ -4418,6 +4423,16 @@ void Session::handleTorrentDeletedAlert(const lt::torrent_deleted_alert *p)
m_removingTorrents.erase(removingTorrentDataIter);
}
+void Session::handleTorrentLogAlert(const lt::torrent_log_alert *p)
+{
+ LogMsg(tr("torrent_log_alert: '%1'").arg(QString::fromStdString(p->message()), Log::INFO));
+}
+
+void Session::handleTrackerErrorAlert(const lt::tracker_error_alert* p)
+{
+ LogMsg(tr("tracker_error_alert: '%1'").arg(QString::fromStdString(p->message()), Log::INFO));
+}
+
void Session::handleTorrentDeleteFailedAlert(const lt::torrent_delete_failed_alert *p)
{
const InfoHash infoHash {p->info_hash};
diff --git a/src/base/bittorrent/session.h b/src/base/bittorrent/session.h
index a139e9d95..6fdc965f8 100644
--- a/src/base/bittorrent/session.h
+++ b/src/base/bittorrent/session.h
@@ -579,6 +579,8 @@ namespace BitTorrent
void handleListenFailedAlert(const lt::listen_failed_alert *p);
void handleExternalIPAlert(const lt::external_ip_alert *p);
void handleSessionStatsAlert(const lt::session_stats_alert *p);
+ void handleTorrentLogAlert(const lt::torrent_log_alert *p);
+ void handleTrackerErrorAlert(const lt::tracker_error_alert *p);
#if (LIBTORRENT_VERSION_NUM >= 10200)
void handleAlertsDroppedAlert(const lt::alerts_dropped_alert *p) const;
#endifFull gdb backtrace in case it is useful: |
|
I have the same problem, qbittorrent suddenly started eating 500+mb of RAM (was loaded in background I did not do anything with it), disk cache is default at 64mb, at most 3-5 active torrents and everything else is idling, feels like it leaks memory when active/uploading, from 100mb at cold boot and consistently rising up to 500mb and beyond, 4.2.1 on Win7 |
|
Well i hate to say this in the face of my beloved qb maintainers, but this issue summons cryptomining sentiments in my heart, which saddens me. |
This is some kind of bug in a specific version that people are using of the cryptographic library or maybe libtorrent is using it in a way that causes this. It has nothing to do with cryptominers, or are you implying that these well-known and heavily scrutinized open-source projects have sneaked one into your system? The code is out there for you or anyone else to check and even compile yourself if you want. |
|
I know very little about code, and made that statement on pure speculation as a self admitted pebkac. I am self aware however, and realize that code doesn't generally operate on feelings (unless the chips get hot and start bitflipping). I am more than happy to be sorry for bringing it up. Very Disrespectful |
No problem, just keep cool and keep the discussion on topic. |
|
Continued at: #12326 |
qBittorrent version and Operating System
qb 4.2.1 x64
Win10 LTSC x64
What is the problem
Unusually high memory usage
What is the expected behavior
Less
Steps to reproduce
Upgrade to 4.2.1 from 4.1.9.1
I tried to downgrade back to 4.2.0 and found same issue.
Upon trying to revert to 4.1.9.1 ran into the known 'recheck all files' and I did not test.
Generally if i restart the program memory usage is low for some amount of time.
When I check task manager after a few hours or so it will remain over 1gb of memory.
No further testing
Extra info(if any)
Screenshot of task manager
https://i.imgur.com/vUza229.png
The text was updated successfully, but these errors were encountered: