Node.js-based alternative to jMeter

[hassy]

Hi, I just wanted to share a project I've been working on, Artillery, which is a Node.js-based load-testing tool. It offers several advantages over jMeter:

• It's just an npm package making it very easy to install
• Scripts are written in JSON or YAML rather than XML and are easier to read and write
• HTTP and WebSockets are supported out of the box (Socket.io and AMQP are coming in the next release)
• It can produce self-contained HTML reports which are handy for embedding into Jenkins / other CI tools
• It also outputs stats in JSON for custom scripting / analysis / visualisation

I'm hoping you'll consider using it. I'd be happy to help with writing Artillery load-testing scripts for this project and any other set up and integration tasks.

[mhdawson]

Thanks for the heads up.

Under #5 and #6 the first step is to identify a key Node.js use case that we should be tracking benchmark results.

If there is no existing benchmark that can be re-used/run easily then the next step is to develop one and Artillery could form part of that. I'd suggest you read through those 2 items and see if you can define an important Node.js use case and propose a benchmark that we should be running (either from the ideas there or your own), do a quick search to validate there are no existing benchmarks which cover it. At that point I think we'd have a good candidate which you could develop with Artillery.

In terms of getting more involved in general, feel free to attend the next meeting which is tomorrow and details are here: #29

[hassy]

Thanks @mhdawson. I'd be happy to contribute a benchmark for Ghost (on the list in #6). I can also look at writing one for AcmeAir - it would be interesting to compare it vs the jMeter script.

I'll tune in to the meeting tomorrow.

[gareth-ellis]

It certainly looks interesting.

The main draw back of JMeter as far as I'm concerned is the rampup time. With it being a java application, it does take some time to achieve peak throughput. As such, we currently drop the first half of our run (e.g. we run for 4 minutes, but only consider the last two for calculating throughput). I would imagine there would be some rampup time in node, however I've not really seen any evidence of this being excessive (perhaps as we'd only be able to measure the slowest at ramping up....)

I would be interested in comparing the performance between Artillery and JMeter, I do notice on your github repo you mention up to 1.2K RPS - (Requests per second?). I would expect a single instance of Node running AcmeAir to be achieving at least 2000 requests per second(obviously depending on hardware) - we'd need a way of going beyond 1.2K - perhaps with multiple instances of your driver?

[hassy]

@gareth-ellis I haven't run into situations where Node itself (running Artillery) needed warm up time, although in general for longer tests you'd usually have a warm up phase for the target application.

I'd love to produce some comparative benchmarks between JMeter and Artillery. 1.2k HTTP requests per second is what I got on a mid-tier Digital Ocean VPS in my tests - that number would go up on better hardware (especially on bare metal). Artillery doesn't support clustering out of the box yet, but it's coming in the next release.