Performance affected by Config Elements

[octaviand2]

Expected behavior

I am expecting that JMeter performance is not affected by Config Elements when using 100 threads (min)

Actual behavior

When my jmx has config elements - large in size - performance is significantly affected.
I need to understand if this is normal or not - would appreciate insights on how jmeter works.
Perhaps this is a memmory issue?

Steps to reproduce the problem

I am attaching a simplified jmx that can be used.

If the HTTP header manager is enabled, performance is reduced by 75%

• with HTTP header enabled on my laptop I get about 200k TPS
• with HTTP header disabled on my laptop I get about 800k TPS
  Replicate.jmx.zip

JMeter Version

5.5

Java Version

java version "9.0.4"

OS Version

macos

[FSchumacher]

Well, to get a real insight, look at the source :)
That said, I looked at the source (not for the first time) and am always a bit confused by it :)
So, the header element has to be prepared by the thread group, as it doesn't know, whether it will be used, or not. That preparation is eating up a lot of CPU cycles, especially, as the list of headers is quite big.
The iteration over all those entries is stressing the memory system (GC) a bit, so that is consuming CPU cycles, too.
In my shallow profiling attempt for this test plan I saw, that CollectionProperty#recoverRunningVersion crept up in the methods.

[vlsi]

Frankly speaking, I do not understand the reason to have recoverRunningVersion there.

Apparently, the default implementation in AbstractTestElement#recoverRunningVersion iterates over all the properties and calls org.apache.jmeter.testelement.property.JMeterProperty#recoverRunningVersion.
The test case in the attachment has HTTP Header Manager with ~240 headers.

Of course, it takes time to walk over those 240 properties to reset them.
Unfortunataly, recoverRunningVersion is there from the beginning: it was added in 2003 in 2e6f30d#diff-436813e2e3a0903a899ea4c44396e9a24ccd045c90e9de50dda8030ec0c44128R78

[vlsi]

Here are the CPU profiling results:

[vlsi]

The allocation profling shows we allocate a lot of strings when concatenating property names in

jmeter/src/core/src/main/java/org/apache/jmeter/control/IteratingController.java

Line 49 in b6d11d7

JMeterUtils.formatJMeterExportedVariableName(elementName+GenericController.INDEX_VAR_NAME_SUFFIX), iterCount);

Another interesting case is Integer.valueOf(IntegerProperty.getStringValue) in

jmeter/src/core/src/main/java/org/apache/jmeter/control/LoopController.java

Line 90 in b6d11d7

nbLoops = Integer.valueOf(prop.getStringValue());

I think we should just replace it with prop.getIntValue() and avoid Integer for the field type Integer nbLoops.

[vlsi]

Another finding is that we instantiate a new ConcurrentHashMap for every SampleResult:

jmeter/src/core/src/main/java/org/apache/jmeter/samplers/SampleResult.java

Line 221 in a568c86

private final Set<String> files = ConcurrentHashMap.newKeySet(2);

I'm not sure we really need that map.

[octaviand2]

let me also provide more insight on the use case.

I am using custom plugins where the config elements are rather large (this is the reason why in the replicate script I have put so many HTTP headers).

When I run the load tests with these custom plugins, I am seeing high CPU usage and severe performance limitations, especially when the number of threads is more than 150

Moreover, with the script I have provided, I see performance degraded 10 times on my laptop if I have only 5 threads

• 1M TPS with config element disabled
• 100k TPS with config element enabled

Would really appreciate support.

[FSchumacher]

Do you know if the plugin uses a CollectionProperty, as the HeaderManager does?
Can you do some profiling of a run with your workload? For example with something like https://github.com/async-profiler/async-profiler or https://jdk.java.net/jmc/8/ (that is using flightrecorder)

[vlsi]

I agree it would make sense to profile the case with custom plugins, then you could figure out the key bottlenecks.

[octaviand2]

our development has told me that we are not using CollectionProperty but StringProperty.

Also, with the replicate script already provided (standard config/sampler), I am noticing the CPU is very high when the config element is enabled, even if the throughput is much lower - first part is element enabled, second part is with it disabled.

as if JMeter is doing something extra for each sampler executed due to the config element being enabled

[vlsi]

The CPU load does not clarify much. I would suggest profiling at method level

[octaviand2]

profiling with our custom plugins is not really relevant - I think if you find the issue/fix it for the large config elements with the replicate script I had provided, it will fix also our issue.

something is happening in Jmeter that consumes a lot of CPU when there is a large config element and multiple threads - performance is severely affected. As if it's reloaded with each sampler executed.

reducing to the minimum the config element in my case led to improved results - but still CPU is much higher than expected.

would appreciate support - we have use cases where we need to reach 50k+ TPS and I am bit stuck.

[FSchumacher]

That might be true (and I suspect it is), but it would be even better to see those values in a chart. If you want to hide your company name, pixelate it.

[octaviand2]

it's not about company name - R&D does feel comfortable sharing these profiling when we run with our plugins.

they say - and I agree - that the same issue happens with the Replicate script I had provided - so that can be used for investigations

[vlsi]

Do you think you could share the files in private?

For instance, as I run replicate.xml, I get 315'000 / sec in total.
I guess it should be enough to cover 50K TPS.

[octaviand2]

you get 315k/sec with the config element enabled or disabled? can you run a comparison test and also look at the CPU on your laptop?

[vlsi]

I run with header manager enabled.
I have Apple M1 Max, Java 11.0.13.

Header manager on: 315K
Header manager off: 830K

[octaviand2]

I am getting similar results - I have M1 with 16GB. It's a dummy test that shows something is happening probably at each request due to that Config element.

…

On Fri, 5 May 2023 at 19:30, Vladimir Sitnikov ***@***.***> wrote: I run with header manager enabled. I have Apple M1 Max, Java 11.0.13. Header manager on: 315K Header manager off: 830K — Reply to this email directly, view it on GitHub <#5875 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHXCOSXRBRMIUC2HR74RLR3XEUTKDANCNFSM6AAAAAAXUO5VTU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[FSchumacher]

On my machine setup (Dell XPS 13 (i7-8550U), Java 1.8) I get 180K with Header manager.
And yes, the numbers for the given test are bad, so we have to investigate further. But I think it would be better to have more numbers, even if you share them privately, only.
What I don't understand (but I don't have to) is what information would be leaked by pixelating/striking out all company information from those profile numbers.

[octaviand2]

in my view it's not relevant to share this information as the case reproduces with JMeter standard samplers.
and it's definately an issue since the CPU is very intensive while the throughput decreases when the config element is enabled.

would appreciate support - in our case, the more threads we have the worse the performance :(

[vlsi]

@octaviand2 , Pull requests that resolve the issue are welcome

[octaviand2]

Frankly speaking, I do not understand the reason to have recoverRunningVersion there.

Apparently, the default implementation in AbstractTestElement#recoverRunningVersion iterates over all the properties and calls org.apache.jmeter.testelement.property.JMeterProperty#recoverRunningVersion. The test case in the attachment has HTTP Header Manager with ~240 headers.

Of course, it takes time to walk over those 240 properties to reset them. Unfortunataly, recoverRunningVersion is there from the beginning: it was added in 2003 in 2e6f30d#diff-436813e2e3a0903a899ea4c44396e9a24ccd045c90e9de50dda8030ec0c44128R78

apparently this is a huge bottleneck for us - our config elements have 300+ properties and the performance is severely limited by these iterations.

Any clue on the functional reasoning for it - can it be disabled by some configuration?

[vlsi]

You can override recoverRunningVersion with a no-op method in your config element

[vlsi]

One of the immediate actions I suggest is that we skip recoverRunningVersion calls for elements that implement NoThreadClone.

In other words, if an element is safe for concurrent calls, then recoverRunningVersion makes no sense for it as it would reset the values at random when the threads are calling the component.

[vlsi]

I've added recoverRunningVersion implementation to HTTP Header Manager in d13f120, so the benchmark makes both "with header manager" and "without header manager" have the same performance.

Thanks for providing the test case.

---

Rampup 0 (e.g. to make all the threads produce load faster)
Java 17.0.17
Apple M1

Title	Before: requests per second	After: requests per second
HTTP Header enabled	0.49M	1.08M
HTTP Header disabled	1.08M	1.08M