Running Unit Tests in CodeSandbox

[gautamarora]

After some early conversations with @CompuIves in Discord, I'm starting this issue to begin discussions for adding support to run unit tests in CodeSandbox.

What i was thinking was a way to use codesandbox as a tool to support "learn by doing" environment where others can learn JS/React or any other topic by writing code to pass failing tests. There are other coding environments that allow this like tddbin and there are curriculums like nodeschool, js-assesment, es6-katas that exist; my proposal is to provide support for creating such a setup in codesandbox.

I was trying out something simple that would work for me given the current codesandbox setup, and created this: https://codesandbox.io/s/716lx85wk6

Baking in something like this that is native to the codesandbox site would make it really fun for folks to create and complete coding workshops.

Happy to hear thoughts from everyone and work together on it if this is interesting.

[CompuIves]

Nice! I was thinking that we could detect all files that have a .test.js or .spec.js. We could run these files using jest-matchers, which works in the browser.

Then we can show all results in a tab besides Console (under the preview).

What do you think?

[gautamarora]

@CompuIves thats a really good idea for native integration.

In my earlier Test component approach (sandbox here), the tests would need to be written in a new style but in the approach you suggest, we would be able to have tests written and viewed in a more real-world setup.

I was quite easily able to switch using expect from chai to jest-matchers - thanks for that tip!

However, the next hurdle would be to run jest in the browser, something that is still being discussed and not resolved (here, here and here). I tried to stub the features we would need from the jest runner over to something simple and created this sandbox as a demonstration of the end result:
https://codesandbox.io/s/2jxjmoj1lp

In order to see the output in console (as in screenshot above) currently, the preview has to be switched to current module view, which I think can be worked out in the unit test integration feature.

Instead of using Console, we can move over the results to a dedicated Tests tab.

Thoughts?

[CompuIves]

Wow, this is really great @gautamarora. I completely agree with the approach. Let's start with a simple version, like your runner.js, and put the results in a separate Tests tab. I think a good benchmark would be to get the tests in the redux example running.

In order to see the output in console (as in screenshot above) currently, the preview has to be switched to current module view, which I think can be worked out in the unit test integration feature.

Yes. We could first evaluate the sandbox itself, then evaluate all test files by finding modules with the name (or maybe do these two tasks concurrently?). Then aggregate the results and send them to the tab. We communicate with messages between te preview and the editor, using codesandbox-api. An example is here: sending and receiving.

What do you think of the approach?

[gautamarora]

@CompuIves agree with everything above 👍

Exciting updates...

I forked the redux example you shared, imported the tester/runner.js into the test files and the tests in actions and reducers just worked!
Check it out here: https://codesandbox.io/s/wn6o8mmx8l

Only the tests in components do not work, i'll look into why later today.

As next steps I will be figuring out the actual integration of the tester into codesandbox-client code & sandbox processes.

[gautamarora]

Needed to add jest-mock to runner. Tests in components pass as well now!

[CompuIves]

Wow!! This is really cool. So the next thing would be integrated evaluation, right?

For that we'd need to evaluate the code with globals describe, it, test and expect set I reckon. We have manager.evaluateModule https://github.com/CompuIves/codesandbox-client/blob/master/packages/app/src/sandbox/eval/manager.js#L113 that we could use. But we should add functionality to specify globals. I can do this, so you can specify an object like this:

{
  it,
  expect,
  // etc
}

We could start evaluating all tests module after this: https://github.com/CompuIves/codesandbox-client/blob/master/packages/app/src/sandbox/compile.js#L188.

Then let the test runner send the messages which the DevTools can respond to. So the first thing we'd need to do is move runner to a file in the sandbox. Something like sandbox/tests/runner.js.

Would you be interested in opening a PR that we can work on 😄 ?

[gautamarora]

yes, next step is integrated evaluation. Will get started on a PR for us to work on 👍

[gillchristian]

Wow this is really cool! 💪 🚀

I've been working on a couple of projects lately that involved a lot of helpers and utils so I wanted to test them but at the same time I did not want to leave CodeSandbox.

I'd like to help (if needed of course). I'm not really familiar with CodeSandbox internals but this could be a great chance to get into it 😉

[DisasterMan78]

This looks absolutely fantastic.
It would be amazing if the feature could be developed to work with other testing libraries added as dependencies. I came across codesandbox today and am seriously impressed, but had hoped to create a reduced test example to share and get some help with testing issues I'm having - converting my tests would be impractical, and not very helpful for getting help with my actual dev environment.

I'm relying on chai, sinon, enzyme and babel-plugin-rewire, which would seem to complicate things enormously.

Would this even be possible in browser?

I appreciate this may be a long time coming, if ever, but wanted to add my congratulations and encouragement, nonetheless.

[gautamarora]

Thanks for the votes of excitement @gillchristian & @DisasterMan78. I am glad that you find this early work exciting and in a direction that will be useful for you in your projects.

@gillchristian - I'll take a shot at starting the native integration for unit testing in code sandbox based on the notes from Ives above, and create a PR to collaborate on next week. I'm sure there will be enough work in this PR for both of us to learn and contribute to CodeSandbox!

@DisasterMan78 - I do think it will be technically feasible to extend this feature to work with set of other testing libraries in the future 👍

[marcinczenko]

I am getting an error when trying to run tests in components:

Warning: React.createElement: type is invalid -- expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in.

[marcinczenko]

Great work guys!

I like the solution for its simplicity, but I am also wondering if the tests cannot be run more "natively".
For me the most important value of the CodeSandbox is that each sandbox follows the create-react-app pattern, which means you can really develop production code using CodeSandbox. Running tests should IMHO be handled in the same idiomatic way without having to resort to some "hacks".

I do not yet know the architecture of CodeSandbox, but I would imagine that there is something spinning a create-react-app-based build process dedicated to each sandbox (maybe this is not the case, just guessing for the moment). The proposed solution has one disadvantage: now when you export the app, your testing code is different from the one you would normally write. I mean, in the normal flow, your would not have to do:

import { describe, it, expect, jestMock } from "../../tester/runner";

or anything like that, not to mention you should not need the ./../tester/runner in the first place.

Also, when you start doing snapshot testing, I am not sure if you can handle that on purely on the client side in an elegant way.

I hope to find some time to dig into the code, so that I will be better informed and will not ask potentially silly questions. For now, I am just wondering. I am also willing to help, if time allows.

[CompuIves]

I completely agree @marcinczenko, for that reason we want to build the test functionality right into CodeSandbox, so you don't have to do:

import { describe, it, expect, jestMock } from "../../tester/runner";

etc. The given demo by @gautamarora was more of a way to show that it's possible, and as a pointer on how to implement it natively on CodeSandbox 😄