Incredibly high memory usage v 3.2.0.1

[jacobi973]

In v.3.2.0.0 a fix was to the high RAM usage. However, in 3.2.0.1 I am still experiencing high RAM usage with it.

[EzioTheDeadPoet]

Does the app crash or fail at downloading with an error at the end ?
Because that's what was fixed.

All the RAM it uses is necessary for it to be as fast as it is at what it does.

[EzioTheDeadPoet]

Also how many threads does your CPU have?

Because that also plays a factor in how much data WJ processes simultaneously (which impacts RAM).

[jacobi973]

Oh, I misunderstood then for that. Thanks for the quick response. I have 12 cores and 24 logical processors. Threads it shows around 4300. I usually close the wabbajack app around 95% memory utilization and restart it, so I haven't seen the error.

[jacobi973]

I just bumped everything in the settings file for wabbajack to 15 instead of 24. Not exactly sure which each setting does but wanted to see if that would limit the amount of memory it used

[EzioTheDeadPoet]

Those 4300 Threads are software threads that are usually parked. The Logical Processors are what thread count is in terms of the hardware. Meaning you have a 12Core/24Threads CPU.
But your system has only 32GB RAM ?
If that's the case we need to look into improving the default performance settings detection to assign a maximum possible thread count based on the detected amount of available RAM.

[EzioTheDeadPoet]

Also please don't restart the app before it hasn't failed the installation so we can see what actually happens.

[EzioTheDeadPoet]

I just bumped everything in the settings file for wabbajack to 15 instead of 24. Not exactly sure which each setting does but wanted to see if that would limit the amount of memory it used

I personally would recommend a round amount of threads being set there, because then your CPU can use its cache more efficiently (if it assigns 2 WJ threads together per core.)

[jacobi973]

Yeah, 32GB of ram is correct. I will let it just run this time. Just was a bit nervous when it reached such high utilization.

Next Time I get the chance, as well, I will bump it up to 16 for the threads.

[balu100]

The same thing happens to me when downloading. Sadly, I can't provide logs because my PC literally becomes unusable, so I have to restart. The only thing I can find is the last started download.

specs: Windows 11, Wabbajack 3.2.0.1, 32gb ddr5, hdd

[EzioTheDeadPoet]

The same thing happens to me when downloading. Sadly, I can't provide logs because my PC literally becomes unusable, so I have to restart. The only thing I can find is the last started download.

specs: Windows 11, Wabbajack 3.2.0.1, 32gb ddr5, hdd

What CPU do you have ?

But also are you really downloading to an HDD ?
Because then your download speed might be too fast for your HDD to keep up causing the RAM to fill up because the data can't be written fast enough to the main storage.

Asking for the CPU cause depending on the CPU you can have a lot of simultaneous downloads. So maybe we need to optimize the download threads used setting based on what the final storage location media is.

[balu100]

You likely nailed it in your answer, as I have an R7 7700 with 2 GB/s internet.
I wasn't familiar with this kind of problem.
Is there any way to configure it, so it doesn't use more than half of my RAM or something like that?
I'll try to optimize the threads in the meantime.

[EzioTheDeadPoet]

The only way to tweak this would be to tweak the amount of used threads. 😅

[UrbanCMC]

I believe I mentioned this before on Discord, but a newer version of the downloader library we use allows to specify a RAM limit per download.
It may be worthwile to check whether this helps (maybe just as an optional setting for people that have problems?).

[EzioTheDeadPoet]

Hm if you make it a setting I would probably set the default to something like (RAM/2)/DownloadThreads.

[Lennart00]

The default settings led to the same problem for me. It used 24 Gb of Ram while being pretty much stuck trying to do something with the download archive on an HDD with only 4MB of IO usage per second. I would seriously recommend asking the user if he has an SSD on startup and chose different default values for simultaneous operations. This behavior made my Windows crash multiple times already. Explorer freezes and everything stops working properly until a reboot happens. These are not sane defaults if slow drive throughput or low amounts of disk iops can cause a total system lockup IMO.

[EzioTheDeadPoet]

The WJ Version in this issue isn't supported anymore.

[Lennart00]

I commented on the wrong Version im sorry for that. This happened with 3.4.0.0