Performance tests

[ErisDS]

In my travels around GitHub recently, I saw a repository with some performance tests / benchmarking. I'm damned if I can remember what it was, but I thought I'd open this up for discussion.

I've seen https://github.com/bestiejs/benchmark.js and https://github.com/broofa/jslitmus

Has anyone got any experience working with a performance testing framework, and can anyone come up with good examples of node modules or other repositories on GitHub which have a thorough test suite?

Also while we're at it, anyone got any experience or examples of monitoring/performance testing memory usage? Not to detect leaks, but rather determining what impacts on the overall footprint.

Ideas / thoughts / examples on a postcard please.

[javorszky]

New Relic seems to be the de facto standard of testing stuff...

[ErisDS]

I'm using new relic on one of the installs to see how it copes. But I've definitely seen GitHub repos with built in performance tests similar to our unit/integration/functional tests.

If anyone has used benchmark or jslitmus I'd love to hear about it.

[centminmod]

Not sure if entirely related but performance of sqlite databases dependent on how database schema setup http://stackoverflow.com/questions/1711631/how-do-i-improve-the-performance-of-sqlite?rq=1

Not familiar with sqlite, but interesting for making sure sqlite database for Ghost is optimal

[AnneTheAgile]

I apologize I'm just jumping in here, but have you all looked at Mocha, the node rspec-like test framework that includes support for measuring test run time speeds? If we added some mocha tests then we could get perf tests without a separate tool ?

If I wanted to try that how should I proceed?

References;
[1]Mocha , NodeJS based JS testing
http://visionmedia.github.io/mocha/
Mocha - the fun, simple, flexible JavaScript test framework
Mocha is a feature-rich JavaScript test framework running on node.js and the browser, making asynchronous testing simple and fun.

[2]Mocha, RubyGem , not what I meant
http://gofreerange.com/mocha/docs/
Mocha 1.0.0 A Ruby library for mocking and stubbing.
A unified, simple and readable syntax for both full & partial mocking.

//AnneTheAgile

[halfdan]

@AnneTheAgile Hello there, we are already using mocha for our unit tests. Mocha can’t be used for performance tests as we would like to have them though.

[AnneTheAgile]

hi @halfdan , you know the thing I just love about trying to contribute to projects via Github is everyone is so nice and incredibly responsive. I was reading a few posts, and here is my analysis so far.

1.Though we probably couldn't do 'normal' performance tests without a nice tool, such as New Relic (I met them at Pycon and they seem like a great company and product) , they do indeed have a free version. I am not familiar with the exact meaning of every item in their feature chart, but it seems at least some trivial performance tests could indeed be run for free. Is that something we could hook up?
http://newrelic.com/pricing
New Relic Pricing for Web and Mobile Application Monitoring
The FREE Tier DOES have;
Thread Profiler & X-Ray Sessions
Availability Monitoring
Deployment Analysis, History, & Comparison
App Speed Index

The Free tier does NOT have:
Time Comparisons
Access to Performance Data API
Thread Profile

2.Could we try simple timeout tests to ensure that performance meets certain minimums? That is what I saw on mocha and seems very possible. That's the old-fashioned poor-man's way I've used in other situations. It actually works pretty well as a gross test. You can also do a whole bunch, of course they would be slow, with staggered timeouts to determine the actual time.
https://github.com/pghalliday/grunt-mocha-test/

  it('should support the timeout option', function(done) {
    execScenario('timeoutOption', function(error, stdout, stderr) {
      expect(stdout).to.match(/test/);
      expect(stdout).to.match(/Aborted due to warnings./);
      expect(stderr).to.match(/1 failing/);
      expect(stderr).to.match(/Error: timeout of 500ms exceeded/);
      done();
    });
  });

//edited 2014-04-30Wed.22:31:59@fix bad paste

[ErisDS]

New relic is nice for testing certain things, but it's also a bit over complicated, and I'm not certain that it lets us run tests as part of the build.

Ideally what I'd like is a set of benchmarks for particular parts of the application done using a tailored tool like benchmark or litmus. I'm sure this can be done using mocha, but it seems like we're trying to bend the tool to our will, rather than using something designed for the purpose.

[AnneTheAgile]

@ErisDS , thank you for more detail on what you want. So is this ticket a placeholder so sometime you can buy Benchmark/Litmus [I'm not familiar with either]? I am a big agile fan, so it seems like it would be nice to use tools we already have [mocha] to get a tiny bit of the value and then see if we can use the data effectively?

In the open source spirit, I am not the fastest hacker, but I would really love to try to benchmark a tiny bit of the system with mocha. It seems like you have all the infrastructure for me to try out a bunch of things and then your CI machine will tell me if I got it right. I love that!

In any case, perhaps you also know which parts of the system you'd like to check? Maybe putting that here would also be a good start.

[ErisDS]

I'm not sure what you mean by "sometime you can buy Benchmark/Litmus". They are both free open source tools.

If you wanna have a go with mocha, by all means give it a shot. The thing to look at benchmarking, IMO would be the API?

[AnneTheAgile]

ok, ty @ErisDS ! I see the api page. https://github.com/TryGhost/Ghost/wiki/API-Overview

On google, I see Litmus, but not Benchmark, the word is too common? https://litmus.com/blog/tag/benchmarks

[ErisDS]

Links to both libraries are in the original post here on GitHub.

[AnneTheAgile]

Cool, so I have lots of fun things to chew on.

It looks like Benchmark is by far the better one to look at of the two Github frameworks. It is being updated regularly, has pulls/issues, and almost 1000 stars ( https://github.com/bestiejs/benchmark.js ). In contrast, JsLitmus ( https://github.com/broofa/jslitmus ) was only updated four years ago and has little traction on github.

Also BestieJs's Benchmark references a cool online performance calculator, JsPerf:
http://jsperf.com/
jsPerf — JavaScript performance playground
jsPerf aims to provide an easy way to create and share test cases, comparing the performance of different JavaScript snippets by running benchmarks. For more information, see the FAQ.

Finally, like you, @ErisDS , I also saw an open source repo with performance tests. I was really impressed and investigated it quite a bit at the time. I'll try to see if I can hunt down my notes. It was a good while back - maybe 2 or 3 years ago. I think I might have heard a talk on it.

[AnneTheAgile]

I currently am not able to get my Vagrant working. I am sure it must be user error? :) If you have any thoughts I would appreciate it! TryGhost/Ghost-Vagrant#46

[ErisDS]

Going to close this as it didn't get any traction. We can reopen and revisit in future if there's more demand or need.