[RFC] Post-Test Assertions Using afterEach in Storybook

[kasperpeulen]

RFC: Post-Test Assertions Using afterEach in Storybook

The proposed afterEach hook in Storybook is designed to support post-test assertions such as accessibility (a11y) checks, automatic snapshots, and other validation tasks that need to occur after each story's play function has executed.

Why Use afterEach for Post-Test Assertions and Not for Cleanup?

In many testing frameworks, the primary purpose of afterEach is for cleanup. However, in Storybook, we recommend performing cleanup in the return callback of the beforeEach hook. This approach offers a convenient way to co-locate the creation and cleanup of state providing a better developer experience (DX).

Vitest also supports return a cleanup function in beforeEach:
https://vitest.dev/api/#beforeeach

Example of Cleanup in beforeEach Return Callback

const meta: Meta<typeof Page> = {
  component: Page,
  // 👇 Set the value of Date for every story in the file
  async beforeEach() {
    MockDate.set('2024-02-14');

    // 👇 Reset the Date after each story
    return () => {
      MockDate.reset();
    };
  },
};

Alternatively, you can perform cleanup at the very beginning of a globally registered beforeEach in preview.ts:

import { Preview } from '@storybook/your-renderer';
import MockDate from 'mockdate';

const preview: Preview = {
  async beforeEach() {
    MockDate.reset();
  },
};

export default preview;

This is often simpler, as you don’t have to think about resetting state at a story level.

Understanding Storybook's Approach to Cleanup

Storybook is a visual testing tool where you can execute tests in an interactive development environment. In command-line interface (CLI) testing frameworks like Jest or Vitest, tests are executed from beginning to end, and hooks such as afterEach and afterAll are used to clean up state created during tests. For example:

import { afterEach } from 'vitest';
import { server } from './mocks/node';
import { cleanup } from '@testing-library/react';

afterEach(() => {
  // Reset any request handlers that were added during the tests
  server.resetHandlers();
  // Clean up the DOM
  cleanup();
});

In an interactive testing tool like Storybook, it's preferable not to execute cleanup callbacks at the end of a test's interactive execution. This allows you to interact with the end state of the test, such as performing additional DOM interactions while the mocked server remains active. Cleanup callbacks are only called when switching stories. Making sure a new story execution, starts with a fresh state.

Note: Vitest's browser mode faces similar challenges and has forked testing library to ensure cleanup is not run at the end of a test:
https://github.com/vitest-dev/vitest-browser-react

Unlike @testing-library/react, vitest-browser-react performs cleanup of the component before the test begins, allowing you to see the rendered result in your UI. If you prefer to disable auto-cleanup, you can import the render function from vitest-browser-react/pure.

To address this challenge more generally, we propose having two distinct stages at the end of the testing lifecycle:

• Post-Test Assertions (afterEach): For tasks like accessibility checks, snapshots, and other validations that should occur after the story's play function has completed.
• Cleanup (Return Callback of beforeEach): For resetting or cleaning up state that was set up in beforeEach, ensuring that each story starts fresh.

Migrating from useEffect

Some addons have implemented post-test validations using Storybook's useEffect API (note: this is different from React's useEffect):

import { useEffect } from '@storybook/preview-api';

const preview = {
  decorators: [
    (storyFn) => {
      useEffect(() => {
        // This also gets executed when the play function has ended
        // e.g., perform some validation on the DOM

        // Cleanup runs when switching stories
        return () => {};
      }, []);

      return storyFn();
    },
  ],
};

However, promises in useEffect cannot be awaited, which poses challenges when executing tests—you can't determine if the test has fully ended. Consequently, useEffect is currently not executed in our Vitest-based test runner (using portable stories). Additionally, the useEffect API can be clunky.

Note that many use cases previously handled by decorators can now be achieved with beforeEach. Many scenarios involving a decorator combined with useEffect can be better handled with afterEach in the future.

Usage Examples

Addon Usage of the afterEach Hook

Addons in Storybook can leverage the afterEach hook by defining it within a preview file, registering post-test assertions for each story run. This allows for seamless integration of accessibility checks, visual snapshots, and other custom assertions.

For example, an addon might register an afterEach hook that performs accessibility tests and captures visual snapshots after each story execution:

// some-addon/preview.ts

const preview: Preview = {
  async afterEach(context) {
    // Accessibility assertions
    await runA11yAssertions(context);

    // Automatic visual snapshots
    await captureSnapshot(context);

    // Any additional post-test assertions can be added here
  },
};

export default preview;

Just like beforeEach and decorators, an array of functions is also accepted:

const preview: Preview = {
  afterEach: [firstAfterEachFn, secondAfterEachFn]
};

The items in the array are executed in reverse order. See order of execution below.

Story-Specific Uses for the afterEach Hook

While afterEach can be used in addons for universal checks (e.g., accessibility, snapshots), it can also be leveraged on a per-story file basis for unique post-test behaviors.

Post-Interaction DOM Validation

Certain stories may require a specific DOM structure or state after the story interactions have completed. The afterEach hook can assert that elements are in the expected state, which is particularly useful for stories where UI conditions need to be validated for each story in the story file.

import { Button } from './button';

export default {
  component: Button,
  afterEach() {
    // Post-interaction DOM validation in afterEach shared between stories
    // e.g., expect(someElement).toHaveTextContent('Expected Text');
  },
};

export const Basic = {
  async play() {
    // Simulate user interactions, e.g., clicking the button
    // Specific assertion for this story only
  },
};

export const OtherStory = {
  async play() {
    // Simulate user interactions, e.g., clicking the button
    // Specific assertion for this story only
  },
};

Order of Execution

Storybook executes hooks and functions in a specific order across various levels, from global presets to story-specific configurations.

Execution Levels and Order

In Storybook, presets and hooks can be defined at different levels:

1. Preview files: Defined in ./storybook/preview.ts or in an addon's preview file, such as ./addons/preview.ts. These preview files apply globally across all stories.
2. Story Meta Configuration: Defined at the story meta level within a specific story file, applying to all exports within that file.
3. Individual Story Configuration: Defined within each individual story, applying only to that specific story.

Detailed Order of Execution

Given this structure, here is the specific order of execution for hooks and functions during a story run:

Level	Order	Hook/Function	When
Preview	1	beforeAll	only happens when storybook starts up or reloads
Preview	2	beforeEach	in every story execution
Story Meta	3	beforeEach	in every story execution
Individual Story	4	beforeEach	in every story execution
Individual Story	5	play	in every story execution
Individual Story	6	afterEach	in every story execution
Story Meta	7	afterEach	in every story execution
Preview	8	afterEach	in every story execution
Individual Story	9	beforeEach return callback	when switching/reloading story
Story Meta	10	beforeEach return callback	when switching/reloading story
Preview	11	beforeEach return callback	when switching/reloading story
Preview	12	beforeAll return callback (only happens when storybook reloads or is finished)	only happens when Storybook reloads, or at the end of a Storybook test run

• Setup Hooks (beforeAll, beforeEach): These are executed from the outermost level (global presets) to the innermost level (individual story).
• play Function: The story's play function is executed after all beforeEach hooks have run.
• Post-Test Assertions (afterEach): These are executed after the play function, starting from the innermost level (individual story) out to the global presets.
• Cleanup: Cleanup occurs by executing the return callbacks of beforeEach and beforeAll in reverse order.

[Sidnioulz]

You've made it clear why such a hook is useful and why cleanup should not be done immediately after tests.

I have two concerns with the proposed API: the naming, and the ability to disable inherited assertions.

On the latter, a11y tests are far from perfect and run into the occasional false positive. Large teams that are given mandatory test assertions by a well meaning dx engineer may run into irrelevant use cases. How does one disable such tests? I'm worried the simplest course of action here is to comment out my own test/story since it would be failing otherwise.

On the naming, reusing a name that has a universal meaning to it to do something new, and, specifically, to NOT do what that name means is a hurdle to learning and transferring existing knowledge about testing.

I'd rather, as a developer, receive an error message when using afterEach that tells me what to do instead to fulfil my expectation. This way I wouldn't put my cleanup functions in afterEach, scramble to figure why it isn't executed and generate support traffic on Discord or GitHub.

And so, if that name was made to error out, it would force your intended use case, which is newish, to find a unique name that best conveys its purpose. This could possibly be a name chosen jointly with vitest-browser, possibly one backed by user research (card sorting can work in this situation).

I would also consider an API that lets me register and unregister assertions. Curious to see if folks would consider it overkill though.

[Sidnioulz]

I mean more like the act of defining an afterEach should cause an error, with clear steps to remediation ("afterEach is not supported in Storybook Test. Instead, return a callback at the end of beforeEach").

[kasperpeulen]

On the latter, a11y tests are far from perfect and run into the occasional false positive. Large teams that are given mandatory test assertions by a well meaning dx engineer may run into irrelevant use cases. How does one disable such tests? I'm worried the simplest course of action here is to comment out my own test/story since it would be failing otherwise.

We plan to support something similar as the a11y addon has now. Being able to disable it for a specific story using parameters:

export const SomeStory = {
  parameters: { a11y: { disable: true } },
  async play() {
    // this will still run but the a11y check in the afterEach won't run
  }
}

Maybe more powerful would be to turn it in a warning, so that CI or chromatic won't fail, but you can still see it in the addon panel what you might want to fix at a later date.

I'd rather, as a developer, receive an error message when using afterEach that tells me what to do instead to fulfil my expectation. This way I wouldn't put my cleanup functions in afterEach, scramble to figure why it isn't executed and generate support traffic on Discord or GitHub.

I understand your concern. And I'm also doubting about that.

But the sceneraio you describe would work a bit differently. afterEach does work the same way as you are used to. It runs after your test. Our opinion is, though, that you should not do cleanup there.

So how do you find out? Well, you clean up something, say your mockdate. And if you interact with your story, you see the date is not mocked anymore in the interactions that you manually triggered.

If you want to circumvent this, and you don't know about the return callback of beforeEach, I think personally, I would put the cleanup in a preview level beforeEach. Which is also the pattern that vitest started using.

[Sidnioulz]

So how do you find out? Well, you clean up something, say your mockdate. And if you interact with your story, you see the date is not mocked anymore in the interactions that you manually triggered.

That could lead to issues in user stories that all lead to different search queries (based on what the mocked content was and the symptoms when it's cleaned up too early), and that could be hard to connect to the specific expectation for afterEach.

So I'm worried it wouldn't be easy for folks to debug or to search for solutions. I feel rather strongly that associating the purpose change with a name change is beneficial, albeit not strictly necessary, because it can help associate a single word to this emerging pattern across tools in the ecosystem, and it can simplify onboarding the community.

[kasperpeulen]

So how do you find out? Well, you clean up something, say your mockdate. And if you interact with your story, you see the date is not mocked anymore in the interactions that you manually triggered.

I understand your point, but let me play the opposition for a bit.

So strictly speaking, how we are gonna implement it, is the same way as how vitest afterEach work.
And in Vitest UI browser mode, you will have the same problem, if you cleanup in afterEach, it messes up interacting with the test after the test has executed.

The main difference is that storybook has an "after" hook (the return cb of beforeEach) that is not executed at the end of the test, but only while switching stories.

So in some sense, semantically and expectation-wise, the same thing happens in storybook and vitest. The difference is more how we treat the return cb of beforeEach. And because we treat it that way, we recommend it a new pattern where cleanups happen there and not in afterEach.

I do still understand your point though. Some people will start using afterEach as cleanup in storybook. And won't know there is a better way to do cleanup in storybook.

The reason I'm still leaning to afterEach is that, say without any other testing history, it is quite a good name for this hook.
It happens right after the play function. It is symmetrical with beforeEach which we also support. And having another name for something that happens right after the story, is also weird, and unconventional. And what would that name be?

Another compromise might be, to only support it on a addon/preview level. And have an error in a story and meta level. So that it can not be misued for cleanup there. And I think it is quite rare you need it on a meta level anyway.

[Sidnioulz]

Your point makes sense. I also agree that afterEach is the correct name. I'm just curious how well it works considering the history and prior mental models. This could be a good target for user research.

Though, I have to say, I believe either extremity (using that name, or using a different name altogether and preventing this one from being used) make more sense to me than a compromise, because a compromise will neither lean into whats historically expected nor into what Vitest browser is pushing.