[discuss] Kibana APIs load testing

[dmlemeshko]

We didn't have any performance metrics for Kibana server-side and it is not easy to say if there are any performance increase/decrease from release to release. This effort is to add a performance monitoring tooling for our APIs.

Actual questions:

• Define the testing target: eliminated Kibana server-side only vs Stack load testing (Kibana + ES)
• Define the APIs scope: what do we want to test? simulate end-user flow on API level or test specific APIs, e.g. public ones?
• Define the testing scope: if we simulate end-user flow, do we need to simulate all requests from web client including getting bundles or focus on the main end-points that retrieve data?
• Define environment: what is the cluster we need for testing? How can find the bottleneck?

Setting goals and expectations:

What questions/issues do we want to answer/solve with load testing?

Tools:

Artillery is open-source load-testing framework written in Node.js. Easy to start, scenario can be described in yml file with helpers and js functions. The main downside is that final HTML report combines all stats together and there is no way to see individual stats for specific end-point. Probably, good for a small scenario with 1-2 endpoints, but not very useful for a end-user flow.

Gatling is open-source load- and performance-testing framework based on Scala, Akka and Netty. It is very flexible, but requires writing scenarios using Scala-based DSL, though it is very convenient. It has nice HTML report with several graphs and stats can be filtered out by specific end-point.

[dmlemeshko]

I created a demo scenario that uses sample data and simulates user login, navigating to Discover page and loading default query, navigation to Dashboard page and loading dashboard with 12 panels.

Load scenario with open model:

      atOnceUsers(20), // starting with 20 active users 
      rampUsers(10) during (5 seconds), // linear ramp of 10 users during 5 sec  
      constantUsersPerSec(15) during (20 seconds), // injecting 15 new users per second at a constant rate
      rampUsersPerSec(5) to 10 during (1 minutes) // Injecting from 5 to 10 new users per second during 1 min

Reports:

[pmuellr]

Question: for Kibana alerting, we'd like to start doing some "load" testing, which basically means:

• having some "interesting" number of alerts actively running, with intervals we expect customers to be using (eg, some every 5-10 minutes, some every hour, etc)
• since alerts generally work off of data currently being ingested (because the alerts are often checking some "current" condition), we'd also like some kind of ingestion load to be occurring as well

I think ideally, this sort of "load" would be good to add as a general thing for Kibana load testing, but wondering if we could reuse work already happening here, to start our own. We also aren't quite sure how we'd set up this "load", and so doing some work on the side makes sense anyway, since we don't have anything to contribute ATM.

Is there anything we can start basing our exploration on, available now?

[dmlemeshko]

@pmuellr I think you can build your scenario on top of the existing project, similar to DemoJourney. I'm not sure how much of the existing scenario can be reused, but I can help to figure it out.

[mshustov]

I noticed that in DemoJourney scenario for Kibana Load testing https://github.com/dmlemeshko/kibana-load-testing/blob/master/src/test/scala/org/kibanaLoadTest/DemoJourney.scala
We test HTML page loading time (visit Home, visit Discover, visit Dashboard). There are a couple of problems with this approach:

• we test the same thing since Kibana Platform serves the same HTML file for all the pages
• served HTML page doesn't contain *.js, *.css resources, which are loaded with AJAX request. We might need to adjust the platform logic to include all the resources in the HTML page and measure results with https://gatling.io/docs/current/http/http_protocol/#resource-inferring (Kibana still spends CPU serving files, so it's appropriate). Or we need to state the main goal of Kibana Load Testing is API performance testing, then we can delete these tests at all. For the client side perf testing we can use appropriate tools - https://github.com/GoogleChrome/lighthouse, for example

[dmlemeshko]

If you are interested in APis load testing, please reach out in #kibana-performance slack channel