Reusable (and exportable) Test Cases - grouping check and integration stage tests

[CMCDragonkai]

Specification

As we are moving towards greater amounts of integration testing, there are lot of existing unit tests that we would like to run, but only during integration. Particularly right now, we are using tests/bin as both the unit tests for the CLI, but also integration testing in the docker platform, and eventually for windows MatrixAI/Polykey-CLI#11 and macos MatrixAI/Polykey-CLI#12.

Once we standardise conditional testing #434, we will have the ability to use npm test to automatically run all tests, while conditional utilities would use "feature flags" to determine whether the test should run or not.

However this doesn't allow us to specify a select group of tests to run. The conditional testing only allows us to say what tests should not be running. This means conditional testing utilities is primarily useful for unit testing stage, where we test everything, but disable certain unit tests when certain conditions are not true.

Instead for integration tests, we need a better "grouping specification" to say instead that these group of tests we do want to test. Think of this as the "allow list" in comparison to the "disallow list" that conditional testing utilities represent. The "disallow list" here would be applied after the "allow list" is applied first.

Originally we evaluated the usage of jest-runner-groups, but I didn't like it's comment annotation implementation and the lack of defined behaviour on edge cases.

Instead I believe directories are good way of structuring tests as we have historically done, and we should be able to reuse directories for "allow list" grouping that would be useful for the integration testing stage.

This would mean rather than running npm test -- tests/bin under docker integration testing, oen could do instead npm test -- tests/integration/docker, only allowing tests under the tests/integration/docker during the docker platform. At the same time, this means general unit testing would need to ignore the integration directory.

To avoid copying/duplicating all the tests however, we would need to be able to re-use test cases defined in other directories, and this means the ability to export test cases to be lazily executed.

Here's an initial prototype:

async function tests() {
  describe(...);
}

if (require.main == null) {
  void tests();
}

export default tests;

The only issue with this, is that it's very coarse-grained, it just allows one to reimport tests from another file, but one cannot dig into the nested test structure, it's just all side effects. If we wanted to be able to selectively execute individual tests from a different test file, we would need something like:

const tests = {
  groupName: {
    beforeAll: [() => { ... }],
    afterAll: [() => { ... }],
    beforeEach: [() => { ... }],
    afterEach: [() => { ... }],
    testname1: [() => { ... }, ...options],
    testname2: [() => { ... }, ...options],
    'testname3.only': [ ... ],
    'testname4.if': [ ... ],
    groupName1: {
      beforeEach: [() => { ... }],
      afterEach: [() => { ... }],
      testname1: [() => { ... }, ...options]
      testname2: [() => { ... }, ...options]
      testname3: [() => { ... }, ...options]
    }
  }
};

I believe rather than writing the full object structure out, we would want to augment the describe and test functions to be non-imperative, that is the same functions are used, but instead of executing side effects, they end up producing a structure like above that can then be lazily evaluated.

const tests = g.describe('', (g) => {
  g.test('', () => { ... });
  g.test('', () => { ... });
  g.describe('', (g) => {
    g.test('', () => { ... });
  });
});

Where g could be new JestLazy or something.

That way we can re-use testIf and test.only... etc.

Finally after all of this, we can then write tests under tests/integration/docker, and create our own tests specifically for docker, or import tests like:

import tests_ from '../bin/x';

const tests = {
  ...tests_['some description']
};

I think this requires quite some amount of iteration to get it right before we apply it all of the tests.

Additional context

• Packaged executable integration tests - initial integration tests with docker #391 - original packaged executable integration tests
• ci: merge staging to master #411 (comment) - comment about running integration tests based on existing unit tests
• ci: merge staging to master #411 (comment) - switch between exporting tests vs running tests

Maybe our testIf should actually be test.if if we "integrate" into jest's structure. For example test.each and test.only, so why not test.if. See: https://softwarewright.dev/blog/posts/jest-unit-testing/extending-jest.html

Tasks

1. ...
2. ...
3. ...

[CMCDragonkai]

For now if we do want create "allow list" tests, we can continue to use directories, but we just need to copy/duplicate the test cases.

It would require that the regular npm test does not run tests on the integration stage test directories. This can be done by filtering it out in package.json or through a predicate on the describeIf. But I prefer the former, since the latter is meant to be a "disallow list" to be applied after and "allow list".

Alternatively we move tests/ down one directory into tests/unit or tests/check to indicate check stage, compared to tests/integration which is for integration stage. Then npm run test-check would be running jest -- ./tests/unit...

[CMCDragonkai]

@emmacasolin I remember you were looking into a custom runner when we were originally doing the sharding. Now I find https://stackoverflow.com/questions/47722048/getting-a-list-of-tests-run-with-jest-without-running-suites. It seems jest does have a way of just getting all the tests as structural information without running them. Is it possible that we may preserve the existing jest code and somehow extract structure to reuse it?

[emmacasolin]

@emmacasolin I remember you were looking into a custom runner when we were originally doing the sharding. Now I find https://stackoverflow.com/questions/47722048/getting-a-list-of-tests-run-with-jest-without-running-suites. It seems jest does have a way of just getting all the tests as structural information without running them. Is it possible that we may preserve the existing jest code and somehow extract structure to reuse it?

This was for the TestSequencer. I don't believe it gets passed the tests as objects, rather just the paths to test files. There may have also been other information passed such as how long that file took to run and how many tests passed, but from what I remember this was just at the suite level, not the individual test level. https://jestjs.io/docs/next/configuration#testsequencer-string

[CMCDragonkai]

Hmm in that case, the only solution maybe to create alternative describe and test functions that produce an object structure that can be interpreted. Sort of like the free monad and interpeter pattern or command pattern.

[CMCDragonkai]

I think I found a library that can help do this: https://github.com/testdeck/testdeck.

They support writing tests in an OOP style. Appears to support jest. I'm not sure if any other limitations.

It seems their way of re-using tests is to write abstract tests, and then import them into concrete tests by way of class extension.

Is this any different from just writing test functions, then importing them to run within describe and test?

Test nesting is also suggested as one can dynamically create tests. In a way, I'm suggesting that all tests should be written in an abstract way so they can be imported to another test file.

[CMCDragonkai]

With #437 and #436, there will be pending fixes here:

1. Remove all the isTestPlatform utilities because they are no longer necessary once we have grouped test directories.
2. Remove any checks for testPlatform from our tests/utils/exec.ts as any conditional things should be done in the test cases, and not in our generic utilities: Standardise process execution for tests #436 (review)

[CMCDragonkai]

The only practical solution right now is something like:

import parameterisedTest from '..somecommontestmodule..';

describe('', () => {
  test('', parameterisedTest(p1, p2));
});

The idea is that parameterisedTest(p1, p2) should return an () => Promise<void> function to execute as the actual test.

Doing this though, we need to be careful about any side-effects coming from beforeEach and beforeAll and afterEach and afterAll. In fact, it would make sense to ensure that all of our tests are functional, which means if things need to be established before a test, they musts be passed as a pointer into the test function itself, so they are still pointing to the same objects being mutated.

This way we separate the "test structure" from the actual test itself which can be shared among different test files.

describe('', () => {
  const obj = {};
  beforeEach(() => {
    obj.x = 3;
  });
  test('', parameterisedTest(obj));
});