Welcome to stress-ng Discussions!

[ColinIanKing]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[kocoten1992]

Hello, thanks for the software.

Legend said if kernel is build with -O3 stress-ng would be 200% faster, https://www.phoronix.com/scan.php?page=article&item=linux-kernel-o3&num=9

(nothing to do with stress-ng, thought it would be interesting to share)

[ColinIanKing]

Building the kernel with clang will show that some of the crypto engines in linux (via the af-alg stress-ng stressor) is 2 times (or more) faster too. Also building the kernel with the latest gcc and enabling build options to target your CPU model also help :-)

[rickwu666666]

Hi Colin,
May I know that stress-ng has any suggestion running environment? More specifically is how many physical memory that left on system can run stress-ng test. Or how I config stress-ng to fit my environment? Why I'm asking is because I was run the stress-ng class os and it cause oom-killer to kill some process like "login, sd-pam" ... etc. In this kind of case, system hang and stress-ng can't not finish.
Therefore, can I assuming that, stress-ng is not valid in this kind of test environment if oom-killer start to kill something else instead of strees-ng sub-process.
I also found that if I reduce the timeout from 30 to 10, then it will pass the test. Wondering is that mean it not enough stressful in system to trigger oom-killer in this case? Or I can consider this is an "adjust" for stress-ng to make sure the test can be done, and every stressor been run.
Many thanks.
Rick

command I used:
stress-ng --sequential 0 --class os --timeout 30 --skip-silent --verbose
This is the mem info before I run the stress-ng:
~$ free -h
total used free shared buff/cache available
Mem: 733Mi 150Mi 282Mi 7.0Mi 300Mi 557Mi
Swap: 0B 0B 0B

[ColinIanKing]

I've added a --oom-avoid option to perform pre-allocation low-memory checks to try and proactively stop low-memory scenarios where the OOM killer kills stressors. Maybe this will help.

The oom_score for the main stress-ng process is set so low since it has to keep track of stressors and perform the reaping and stats collection as well as any file clean up of stressors die. The stressors themselves are set higher so they are always candidates to be killed before the main process.

[rickwu666666]

Hi Colin, Thanks for advised! But may I ask that, what's the different between --oomable and --oom-avoid? Seems both of them are trying to prevent system crashed by oom-killer.

[ColinIanKing]

--oom-avoid will check if enough free memory is available before performing any form of memory allocation. It will then free resources in the stressor and continue on. There is an overhead to this memory checking and it will not exercise the very low memory conditions that can sometimes trigger low memory bugs in userspace applications/services and kernel drivers.

--oomable will allow the kernel OOM killer to kill a stressor but the stressor won't be re-created again and hence will die prematurely.

One can use both options as they both try and reduce out of memory OOM kills. --oom-avoid adds overhead in checking and cannot guarantee a stressor won't trigger an OOM kill. --oomable allows stressors to eat memory and for the OOM killer to kill a stressor to never re-start. Normally stressors will be restarted if they are OOM killed.

[ColinIanKing]

Normally running stress-ng on a server is fine. However, it seems that recent desktops now have a systemd out of memory killer that also makes life more complex. When running on a desktop the OOM killer can kill the entire session, which can involve the desktop being killed, which I think is a bug in the way low memory situations are handled.

[rickwu666666]

Hi Colin, Thanks for the elaborated. That was very helpful and clear for us to decide what should we do in the low memory resource environment. I'll try it both to see if my machine make it through.
And yes, we observed that systemd oomd kinda mess up in desktop env. However, I was running on ARM with UC20. So that won't bother me.

[djacobs98]

Hello.

I some questions about the stress-ng-cyclic stressor.

When the test finishes, you get some output that looks like this:
stress-ng: info: [19326] stress-ng-cyclic: sched SCHED_RR: 100000 ns delay, 10000 samples
stress-ng: info: [19326] stress-ng-cyclic: mean: 4959.82 ns, mode: 4900 ns
stress-ng: info: [19326] stress-ng-cyclic: min: 4492 ns, max: 19726 ns, std.dev. 390.60
stress-ng: info: [19326] stress-ng-cyclic: latency percentiles:
stress-ng: info: [19326] stress-ng-cyclic: 25.00%: 4880 ns
...

My question is about the "10000 samples". I know this refer to the size of the data set used to generate the statistics, however, why is it 10000 and can the sample size be changed? I wasn't able to find anything in the options.

[ColinIanKing]

This is currently set to a maximum of 10,000, one can change this in stress-cyclic.c in the source code:

#define MAX_SAMPLES (10000)

Each sample uses 8 bytes of data for a latency timing measurement, so that's ~80K of data. I guess I could easily make it dynamically settable with a new --cyclic-samples option.

[djacobs98]

Ok, thanks for the information.

[ColinIanKing]

I've pushed to the repo a fix for this, I've added a new option --cyclic-samples where one can specific the number of samples to record, from 1 to 10,000,000. Let me know if this addresses your question.

[djacobs98]

Question about cyclic stressor on Ubuntu 22.04.

Previously I tried out the cyclic stressor under Ubuntu 20.04.
$ stress-ng --cyclic 1 --cyclic-dist 250 --cyclic-method clock_ns -t 60s --log-file cyclic-regular.log --verbose

And this ran fine.

But now when I run this same command on 22.04, I get the following error: stress-ng: debug: [12809] invoked with 'stress-ng --cyclic 1 --cyclic-dist 250 --cyclic-method clock_ns -t 60s --log-file cyclic-regular.log --verbose' by user 1000
stress-ng: debug: [12809] stress-ng 0.14.03 g346518caffe5
stress-ng: debug: [12809] system: Linux u 5.15.0-1013-intel-iotg #17-Ubuntu SMP Sun Aug 7 18:17:28 UTC 2022 x86_64
stress-ng: debug: [12809] RAM total: 14.9G, RAM free: 11.0G, swap free: 2.0G
stress-ng: debug: [12809] 4 processors online, 4 processors configured
stress-ng: info: [12809] cyclic stressor needs to be run with CAP_SYS_NICE set SCHED_RR, SCHED_FIFO or SCHED_DEADLINE priorities, skipping this stressor
stress-ng: info: [12809] setting to a 60 second run per stressor
stress-ng: error: [12809] No stress workers invoked (one or more were unsupported)

I did some research into what CAP_SYS_NICe and SCHED_FIFO are and how to set them, but it seems there's a further problem because stress-ng was installed via a snap, meaning the stress-ng binary is inside a read-only filesystem.

Am I missing something here, or do I need to recompile stress-ng with these already set? (I'm still having trouble understanding the setcap program because its manpage is terrible.)
Thanks.

[djacobs98]

Update:
Trying:
sudo setcap cap_sys_nice=ep /snap/bin/stress-ng
doesn't work. Because /snap is a write-only file system.

Copying /snap/bin/stress-ng to my home directory works...sorta. The setcap command succeeds but then the it still uses the snap's directories to run, giving an error that "temp directory '.' must be writeable." So, again, snap gets in my way.

Checking out the stress-ng repo and building it in my user's directory works:

$ sudo setcap cap_sys_nice=ep ./stress-ng
$ sudo chrt --rr 99 ./stress-ng --cyclic 1 --cyclic-dist 250 --cyclic-method clock_ns -t 60s --log-file cyclic-regular.log --verbose

So, something in Snap in 22.04 changed, and now you can't run the cyclic test without extra hoops.

[ColinIanKing]

I'd advise to avoid using snaps. It may be useful to either use the docker build of stress-ng or build it from source using the information provided in the README.md file.

[1392273211]

请问吧，这些工具是go写百度和谷歌查询到的，里面do_sync_write、vfs_write、vfs_write、sys_write、generic_file_aio_write、代表啥意思的

[ColinIanKing]

Translated by Google translate: "Excuse me, these tools are queried by go to write Baidu and Google, which do_sync_write, vfs_write, vfs_write, sys_write, generic_file_aio_write, what do they mean"

I'm not sure what that question is about. If one can provide more context then would be useful.

[shivaraju173ponnagani]

Hi Colin,

Thanks for the Software.

I would like to use stress-ng software for the arm64 android device. I followed the steps for cross-compilation but it is always throwing an error.
$CC=aarch64-linux-gnu-gcc-4.9 STATIC=1 make
CC stress-zero.c
CC stress-zlib.c
CC stress-zombie.c
LD stress-ng
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(libc-start.o):function __libc_start_main: error: undefined reference to '__aarch64_ldadd4_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(assert.o):function __assert_fail_base: error: undefined reference to '__aarch64_swp8_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(dcigettext.o):function _nl_find_msg: error: undefined reference to '__aarch64_cas4_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(dcigettext.o):function _nl_find_msg: error: undefined reference to '__aarch64_swp4_rel'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(dcigettext.o):function _nl_find_msg: error: undefined reference to '__aarch64_cas4_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(dcigettext.o):function _nl_find_msg: error: undefined reference to '__aarch64_swp4_rel'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(dcigettext.o):function _nl_find_msg: error: undefined reference to '__aarch64_swp4_rel'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(dcigettext.o):function _nl_find_msg: error: undefined reference to '__aarch64_swp4_rel'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(loadmsgcat.o):function _nl_load_domain: error: undefined reference to '__aarch64_cas4_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(localealias.o):function _nl_expand_alias: error: undefined reference to '__aarch64_cas4_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(libc_fatal.o):function __libc_message: error: undefined reference to '__aarch64_swp8_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(libc-lowlevellock.o):function __lll_lock_wait_private: error: undefined reference to '__aarch64_swp4_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(libc-lowlevellock.o):function __lll_lock_wait_private: error: undefined reference to '__aarch64_swp4_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function munmap_chunk: error: undefined reference to '__aarch64_ldadd4_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function munmap_chunk: error: undefined reference to '__aarch64_ldadd8_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function mremap_chunk: error: undefined reference to '__aarch64_ldadd8_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function mremap_chunk: error: undefined reference to '__aarch64_cas8_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function malloc_consolidate: error: undefined reference to '__aarch64_swp8_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function arena_get2: error: undefined reference to '__aarch64_cas8_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function arena_get2: error: undefined reference to '__aarch64_cas8_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function _int_free: error: undefined reference to '__aarch64_cas8_rel'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function sysmalloc: error: undefined reference to '__aarch64_ldadd4_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function sysmalloc: error: undefined reference to '__aarch64_ldadd8_acq'
/usr/lib/gcc-cross/aarch64-linux-gnu/4.9/../../../../aarch64-linux-gnu/lib/../lib/libc.a(malloc.o):function sysmalloc: error: undefined reference to '__aarch64_cas8_acq'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:511: stress-ng] Error 1
make[1]: Leaving directory '/home/My-Space/GitHub/stress-ng'
make: *** [Makefile:491: all] Error 2

Target Machine - Linux localhost 4.9.112 #20 SMP PREEMPT Thu Dec 22 05:43:04 EST 2022 aarch64
Host Machine -Linux 5.14.0-1054-oem #61-Ubuntu SMP Fri Oct 14 13:05:50 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Can you please help me out with the android (aarch64 ) setup?

Thanks in advance.

[ColinIanKing]

Maybe adding -mno-outline-atomics to the CFLAGS will help.

[tapanbasak12]

Hi, is there a description of each of the stressors, and how they work somewhere? I found the manual page but they have a very small description. I want to understand the mechanics of the stressors, especially those which put stress into the cpu cache. So between the code and the man page is there a place from where I could get an intermediate level explanation of how the stressors work under the hood?

[ColinIanKing]

I'm afraid that that including all the description in the man page would make it so long that casual users of the tool would be overwhelmed by the technical descriptions, I've had a lot of feedback that the manual is too long in the past. The only real description is in the code itself. I'll look at adding more comments in the code for future releases.

[hhx-cq]

Hello, thanks for the software. Is there a version of stress-ng available for QNX? I didn't see any mention of a version for QNX in the documentation.Many thanks.

[ColinIanKing]

I don't have a QNX licence, so I've not got a copy of QNX to build stress-ng. However, stress-ng does check for various libc and system call interfaces and is intended to be portable, so it possibly can build and run on QNX.

[iDings]

I open a PR to support qnx #410, it's just work, myabe you can have a try @hhx-cq

[CosminGhiorghita]

Hello Colin and community!

First of all, thank you for this great software!

Recently I have faced an unexpected behavior in the version of stress-ng I am currently using (0.13.12). I am starting stress-ng to stress all available cores and I get the following running processes:

root       10062  0.0  0.0  18100  4864 pts/1    SL+  16:31   0:00 /usr/bin/stress-ng --cpu 0 --cpu-method fft --verify --cpu-load 50 --timeout 100
root       10063 36.3  0.0  18100  1788 pts/1    S+   16:31   0:01 /usr/bin/stress-ng --cpu 0 --cpu-method fft --verify --cpu-load 50 --timeout 100
root       10064 36.3  0.0  18100  1788 pts/1    S+   16:31   0:01 /usr/bin/stress-ng --cpu 0 --cpu-method fft --verify --cpu-load 50 --timeout 100
root       10065 36.3  0.0  18100  1788 pts/1    S+   16:31   0:01 /usr/bin/stress-ng --cpu 0 --cpu-method fft --verify --cpu-load 50 --timeout 100
root       10066 36.3  0.0  18100  1788 pts/1    S+   16:31   0:01 /usr/bin/stress-ng --cpu 0 --cpu-method fft --verify --cpu-load 50 --timeout 100
root       10067 36.3  0.0  18100  1788 pts/1    S+   16:31   0:01 /usr/bin/stress-ng --cpu 0 --cpu-method fft --verify --cpu-load 50 --timeout 100
root       10068 36.3  0.0  18100  1788 pts/1    S+   16:31   0:01 /usr/bin/stress-ng --cpu 0 --cpu-method fft --verify --cpu-load 50 --timeout 100

My tool that spawns stress-ng will also try to terminate it if certain action is performed (which will send the SIGTERM signal to the parent process - 10062 in this case). In the past I had no issue with this but in the actual version it seems that SIGTERM is only terminating one of the child process (not all of them along with the parent as I expected). However, using SIGKILL will kill every stress-ng processes but I try to avoid using it.

Therefore, is there any way/flag to close all process when SIGTERM is received?

Thank you very much in advance!

[ColinIanKing]

This definitely works in more recent releases of stress-ng, there has been a lot of re-working with signal handling in the parent process and the way child processes get terminated on various HUP and TERM signals. I suggest using the latest release to see if this fixes your issue.

[Anthero1]

Hi Colin and Community!

I had a question about the --taskset option. From how I understood the code, when I supply a list of CPU's to the --taskset option, it will set the cpu affinity of all stressors to that list of CPU's.
E.g. I have "--taskset 1,2,3 --pipe 3 --cyclic 3" then those 6 stressors will run on CPU1, CPU2, and CPU3 in any combination. I could have 1 pipe and 1 cyclic stressor on each CPU or I could have 3 cyclic on CPU1, 2 pipe on CPU2, and 1 pipe on CPU3, or etc.
Also, correct me if I am wrong, but the stressors are free to change CPU's as long as it is within the provided taskset list. In the previous example, lets say cyclic stressor #1 started on CPU1, it could at any point move to CPU2/3.

I need to test the interrupt latency on each CPU, so I need the capability to lock each individual cyclic stressor to a single CPU. Is there functionality currently in stress-ng to do this that I have missed? Are there plans to implement this sort of option?

Note: I have already considered running stress-ng as well as cyclic-test at the same time to measure interrupt latency, but I was hoping to use the built in cyclic stressor of stress-ng.

[ColinIanKing]

Hi,

Yes, you are correct, the --taskset sets the set of CPUs that the stressors are allowed to run on and the scheduler is free to move stressor processes to what ever CPU is in that taskset.