Compatibilty/legacy support for Mocha testcases

[jonnor]

Right now we have, across component libraries and projects, many existing tests. Long-term we'd like to migrate the majority of these to fbp-spec, but this is not done in a day....

Most of these tests are written in Mocha+Chai. They usually either set up a NoFlo component by requiring it directly, or use Noflo.ComponentLoader (especially with sub/graphs). Component lifetime is sometimes done per testcase, and sometimes for the whole suite.
We also have some Mocha+Chai tests, which test on the MsgFlo participant level. And we sometimes have tests for plain (JavaScript) code used by NoFlo component, but which is not NoFlo?FBP...

To enable an easy, immediately useful and gradual migration to fbp-spec. It would be great if we could have a way to represent/enumerate, run and evalute plain-old-MochaChai tests as if they were fbp-spec.

[jonnor]

This should be doable to implement by a custom Mocha runner/reporter. It would use the Mocha apis to find and execute tests, collect results.

Challenges are:

• Have to be done with no changes to existing tests (or trivial additions)
• We must be able to do this over a network protocol (WebSocket) for it to be runnable like fbp-spec from Flowhub etc
• The runner is not really an FBP runtime. Even in the special case where there NoFlo inside, we don't really have access to it (unlike fbp-spec).
• The tests are not known beforehand in the client, but must be introspected and transferred across (unlike fbp-spec)
• Assertions of test results happen on the far side, not on the runner/client (unlike fbp-spec)
• We want to be able to also run non-legacy tests in same project, and fbp-spec client / Flowhub supports only one runtime connection

[jonnor]

Possible implementation strategy

TL:DR automatically generate a set of fbp-spec testcases from Mocha, which are executed by a NoFlo fixture, that runs the Mocha testcase and reports the result.
The only new concept here is being able to send testcases from runtime to client.

1. At startup time and/or when testcase files change:

• Analyze the Mocha testcases. Collect all test suites/case ('describe' and 'it' statements).
  Can for instance group everything from one file or top-level describe into one suite
• Convert this data into fbp-spec suite/testcases (in the normal format)
  Each testcase will basically have one fbp-spec assertion (the Mocha testcase should not fail)
• Send these cases over to the client. As an extension to the FBP runtime protocol ?

2. When running tests

• (The fbp-spec client/runner just executes as if these are ordinary fbp-spec cases)
• Determine from fbp-spec test, which Mocha suite/testcase it corresponds too
• Execute the tests with ordinary Mocha semantics. Should use Mocha itself
• Collect the results (exceptions from asstions) of each Mocha testcase (using Mocha reporter interface?)
• Map the results back to fbp-spec testcase it belongs to, report result (ideally as cases are running)
• On failure, we'll send an object with the assertion failure / exception data. On success, send an empty object or just a true?

Ideally we use a fixture graph of NoFlo components to set up and execute all the logic, so that we do not need a special implementation of FBP runtime protocol.

[jonnor]

One little bit tricky thing is to figure out how to multiplex legacy cases and non-legacy cases across the same runtime connection. It could be that the best approach is to let this mocha-legacy-compat tool proxy communication to the real runtime. Especially since this is a feature we want to have/build also for MsgFlo (msgflo/msgflo#28).

We also need to avoid double-detection of the proper fbp-spec cases when those are using the Mocha runner integration. Should be doable by that integraiton detecting when we are in this legacy mode.

[jonnor]

Hopefully using the Mocha third party hooks for UI/interface and reporter, we can do this without dirty hacking:
https://github.com/mochajs/mocha/wiki/Third-party-UIs
https://github.com/mochajs/mocha/wiki/Third-party-reporters

There are a bunch of existing extensions mentioned here, https://github.com/mochajs/mocha/wiki

[jonnor]

Bad news first: It seems that using the UI / interface plugins is may be a dead end.

1. with the hooks I could find (post-require etc), no way to know when all files have been loaded
2. hooks don't fire until Mocha.run() which will start the actual tests, not just load/parse them

This would one would need to use a custom entry point. For node.js, likely an fbp-spec-mochacompat executable of mocha.

Good news:
The code being called by mocha.run() is quite simple, and accessible as public APIs.
https://github.com/mochajs/mocha/blob/master/lib/mocha.js#L466

[jonnor]

Basic functionality is mostly in place now. fbp-spec-mocha will send fbp-spec tests representing the underlying Mocha tests - over FBP runtime protocol, and these can be executed like normal/true fbp-spec tests.

[jonnor]

Still need some tests, bugfixing and documentation.