Testing Hurdles

[dryajov]

Just wanted to document some of the general issues I've ran across while implementing the circuit interop tests

Current issues

• tests are clunky

  • this is mostly perception I guess, but it might be an indication of a possible fundamental issue with the current approach, by this I mean that it has been overall challenging coming up with an "elegant" implementation of these tests
    • is this an issue with current tools limitations?
    • a misuse/misunderstanding of those?
    • a combination of the above?

• current tests are hard to scale and maintain

  • the current basic set of tests across all the different combinations have already surpassed 30 (36 to be exact). These tests mostly consist of spawning a combination of nodes
    • a relay (js, go, browser) and two different nodes (js, go, browser)
    • the tests them self are rather simple
      • connect the nodes and send some data over the established relay
    • the issues are mostly spawning those nodes (and sometimes connecting), which quickly becomes complex
  • future tests will possibly involve more complex setups that will have to be implemented across every combination of relays and nodes, a few I can think of are
    • multihop dialing
    • specific relay addressing
    • additional upcoming implementations (rust, python, etc..)

• tests are slow, the current battery of tests takes around 15 minutes to complete on a typical CI environment

What could be some possible solutions to the outlined issues

• alternative tools that allow writing reusable tests in a less convoluted way?

  • dsl? (suggested by @victorbjelkholm)
  • different testing framework?
  • ???

• separate spawning nodes logic from the tests them self (this is currently difficult with mocha)

• improve speed

  • this is mostly around starting and stopping nodes, there might some really good opportunities to speed things up by just tackling that

The current implementation is in #6, I invite anyone to take a look and leave suggestions on how to improve them. Note that they are still in flux and I'm not requesting a formal review, but a suggestions on how to simplify/improve them overal.

[dryajov]

// cc @victorbjelkholm @diasdavid @richardschneider @mkg20001

[mkg20001]

• improve speed

I've made https://github.com/mkg20001/aegir-keycache which hopefully will fix some of the slowness caused by key generation (demo)