Load test for 3.0.0 : GET https://m32xlarge.gluu.info/oxauth/login FAIL

[yuriyz]

With 5 concurrent users oxauth works very well. If increase load to 100 concurrent users sometimes this exception is thrown.

https://github.com/GluuFederation/oxAuth/blob/master/jmeter/test/authorization_code_flow/Authorization%20Code%20Flow_amazon.jmx

org.apache.http.NoHttpResponseException: m32xlarge.gluu.info:443 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:259)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:209)
at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.receiveResponseHeader(MeasuringConnectionManager.java:212)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:488)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:619)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:379)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.followRedirects(HTTPSamplerBase.java:1514)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.resultProcessing(HTTPSamplerBase.java:1605)
at org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.resultProcessing(HTTPAbstractImpl.java:463)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:459)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1146)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1135)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:465)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:410)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:241)
at java.lang.Thread.run(Unknown Source)

[yurem]

There are not really right values because I've limited CE by 2 CPU and memory. But shows that server can work constantly under this load. I've run Jmeter test with 100 concurrent user with 2000 iterations:

[yurem]

d6b8c6b
sorry old file and new file contains different new lines which add difficulty to compare

[yurem]

In few words I add clear cookies at each iteration because starting from first login attempt in each thread group oxAuth returns empty login form (/oxauth/login) which means that user log in already.

Also I remove keep alive to avoid reusing connection where user log in

[yuriyz]

sorry, I didn't push change with clear cookies. It was on my local pc, thanks for submitting it.

Did you make any configuration changes on the CE? If yes, can you list them here?

[yuriyz]

I just load it with 200 concurrent users and it works well too. I will try even more, lets see when it will start to fail :).

[yurem]

With 500 threads apache requires tweaks:
[Fri Dec 09 15:52:51.500667 2016] [mpm_event:error] [pid 113268:tid 139939495278464] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

[yurem]

After increasing "MaxRequestWorkers" to 400 in /etc/apache2/mods-enabled/mpm_event.conf 500 threads are working fine in test. But I think this "MaxRequestWorkers" update from 150 to 400 is reqiuried only for testing, not for production

[yurem]

According to statistic LDAP is not weak place:

2016-12-09 16:06:01,620 INFO  [xdi.oxauth.service.status.ldap.LdapStatusTimer] (pool-1-thread-9) connectionProvider statistics: LDAPConnectionPoolStatistics(numAvailableConnections=3, maxAvailableConnections=3, numSuccessfulConnectionAttempts=51812, numFailedConnectionAttempts=0, numConnectionsClosedDefunct=0, numConnectionsClosedExpired=0, numConnectionsClosedUnneeded=51809, numSuccessfulCheckouts=704814, numFailedCheckouts=0, numReleasedValid=653005)
2016-12-09 16:06:01,620 INFO  [xdi.oxauth.service.status.ldap.LdapStatusTimer] (pool-1-thread-9) bindConnectionProvider statistics: LDAPConnectionPoolStatistics(numAvailableConnections=1, maxAvailableConnections=3, numSuccessfulConnectionAttempts=1, numFailedConnectionAttempts=0, numConnectionsClosedDefunct=0, numConnectionsClosedExpired=0, numConnectionsClosedUnneeded=0, numSuccessfulCheckouts=0, numFailedCheckouts=0, numReleasedValid=0)
2016-12-09 16:06:01,621 INFO  [xdi.oxauth.service.status.ldap.LdapStatusTimer] (pool-1-thread-9) authConnectionProvider#0 statistics: LDAPConnectionPoolStatistics(numAvailableConnections=30, maxAvailableConnections=1000, numSuccessfulConnectionAttempts=30, numFailedConnectionAttempts=0, numConnectionsClosedDefunct=0, numConnectionsClosedExpired=0, numConnectionsClosedUnneeded=0, numSuccessfulCheckouts=8942, numFailedCheckouts=0, numReleasedValid=8942)
2016-12-09 16:06:01,621 INFO  [xdi.oxauth.service.status.ldap.LdapStatusTimer] (pool-1-thread-9) bindAuthConnectionProvider#0 statistics: LDAPConnectionPoolStatistics(numAvailableConnections=6, maxAvailableConnections=1000, numSuccessfulConnectionAttempts=6, numFailedConnectionAttempts=0, numConnectionsClosedDefunct=0, numConnectionsClosedExpired=0, numConnectionsClosedUnneeded=0, numSuccessfulCheckouts=4470, numFailedCheckouts=0, numReleasedValid=4470)

[yurem]

With 500 thread on my slow VM: