getByRole("progressbar") is pretty slow

[ValentinH]

• React Testing Library version: 10.0.4 (I'm using /react not /dom but I'm posting here as getByRole is implemented here)
• node version: 🤷 (codesandbox)
• npm (or yarn) version: 🤷 (codesandbox)

I've noticed that some of my tests were pretty slow to run (more than 2 seconds). After investigating, I've noticed that the following pattern was to source of the slowness:

await waitForElementToBeRemoved(() => screen.getByRole('progressbar'))

If I add a data-testid to my loader and switch to:

await waitForElementToBeRemoved(() => screen.getByTestId('loader'))

it's much faster.

Relevant code or config:

I'm sometimes displayed a CircularProgress from material-ui while fetching some data and I use this pattern to wait for it to be removed.

await waitForElementToBeRemoved(() => screen.getByRole('progressbar'))

example of implementation

if(isLoading) {
  return <CircularProgress className={classes.loader} data-testid="loader" />
}

What happened:

Here are 2 tests outputs where I only change the pattern mentioned above.

getByRole

getByTestId:

Reproduction:

I've create the following Sandbox: https://codesandbox.io/s/musing-gauss-e1ynf?file=/src/Demo.js

The numbers are much smaller but when running the tests several times we can see that the getByRole is slower when used on the CircularProgress.

Problem description:

I think that

await waitForElementToBeRemoved(() => screen.getByRole('progressbar'))

is the best pattern to test that a loader is displayed on screen. However, the slowness prevents us from using in all our tests as it quickly gets slow.

[eps1lon]

getByRole can get quite expensive in large trees since it currently checks every element while ByTestId is a simple querySelector. It also includes a lot more a11y related checks. For example it won't return inaccessible elements.

Since you actually remove the element from the DOM you can ignore the additional a11y tree checks and go with getByRole('progressbar', { hidden: false }).

On slow machines these additional checks can starve your CPU and you should increase the polling interval with waitForElementToBeRemoved(() => {}, { interval: 200 }) (try out which interval works best).

And last but not least: What version of jest or, more specifically, jsdom are you using? In later jsdom versions (or actual browsers) getByRole should be a lot faster.

For us: I think we should make the perf impact of hidden in ByRole clearer in the docs. Also mention too low polling rate on slow machines.

[ValentinH]

Thanks for the quick reply!

I've just tested the { hidden: false } and it doesn't have an impact in my example unfortunately. 😞
The thing is that in the sandbox example, the tree is not so big. What I don't get, it the difference when using CircularProgress and directly using its output div + svg.

I'm using version 25.2.7 of Jest.

[eps1lon]

I've just tested the { hidden: false } and it doesn't have an impact in my example unfortunately.

Have you tried increasing the interval? Would be helpful if you could provide a full repro. Otherwise it's unclear what makes this test so slow.

What I don't get, it the difference when using CircularProgress and directly using its output div + svg.

It's unclear where this difference is coming from. Could just be the CircularProgress doing more things and slowing the test down because of it.

[ValentinH]

Increasing the interval didn't help either.

I'm preparing a demo with my real use case :)

[ValentinH]

I've updated the demo with a more complex form and here's the result:

Sandbox: https://codesandbox.io/s/musing-gauss-e1ynf?file=/src/DemoForm.js

I've removed the CircularProgress so that we don't focus on it.

[eps1lon]

Great. I think this is somewhat expected with the current implementation. ByTestId takes 130ms while ByRole takes 200ms. This is 50% slower.

Might be nice to experiment with different implementations that don't loop over all nodes but convert the role directly to a selector and use that directly.

[ValentinH]

I've dug in the (transpiled) code and try to see where it was spending the most time. Surprisingly, this is not in queryAllByRole, but at this line:

dom-testing-library/src/query-helpers.js

Line 62 in ca1b608

throw getConfig().getElementError(

Especially, the call to getMissingError(). Before this call every operations takes a few milliseconds. getMissingError takes almost 1 second on my example!

[ValentinH]

Digging even more, I'm landing on this line: https://github.com/testing-library/dom-testing-library/blob/master/src/queries/role.js#L116

However, I'm using 7.3.0 and seeing that you changed the implementation in 7.4.0. Thus, I need to check if it has changed the behaviour. 🙂

[ValentinH]

Still the same behavior on 7.5.1.

I'm now checking this line:

dom-testing-library/src/role-helpers.js

Line 134 in 429de04

return hidden === false ? isInaccessible(element) === false : true

isInaccessible gets called 30 times when getByRole rejects and each call last around 30 ms.

[ValentinH]

Funnily, if I do getByRole('progressbar', { hidden: true }) it solves my issue as it does not call isInaccessible. 😄

[kentcdodds]

Hi @ValentinH! Thanks for all this digging.

I believe that isInaccessible has historically been some bottleneck code. I'm not sure of the best way to speed it up, but if anyone has ideas that would be pretty sweet :)

[ValentinH]

Yes, I'm trying to see what part of it is the slowest part :)

[ValentinH]

Not a big surprise, but most of the time is spent on

window.getComputedStyle(element).display

[ValentinH]

Using the same caching mechanism than in

dom-testing-library/src/queries/role.js

Lines 42 to 49 in 752ff66

	const subtreeIsInaccessibleCache = new WeakMap()
	function cachedIsSubtreeInaccessible(element) {
	if (!subtreeIsInaccessibleCache.has(element)) {
	subtreeIsInaccessibleCache.set(element, isSubtreeInaccessible(element))
	}
	return subtreeIsInaccessibleCache.get(element)
	}

makes my tests almost 2 times faster.

They are still 5-6 times slower than getByTestId though.

One idea I have is: do we really need to build the whole "role" error message when using waitElementToBeRemoved() ? Indeed, we are only checking the error name in it:

dom-testing-library/src/wait-for-element-to-be-removed.js

Line 36 in 752ff66

if (error.name === 'TestingLibraryElementError') {

[ValentinH]

I could make it even faster by using the same cache idea but for the window.getComputedStyle:

const getComputedStyleCache = new WeakMap()
function cachedGetComputedStyle(element) {
  if (!getComputedStyleCache.has(element)) {
    getComputedStyleCache.set(element, window.getComputedStyle(element))
  }

  return getComputedStyleCache.get(element)
}

As this is used both in isInaccessible and isSubtreeInaccessible, we cache even more.

[eps1lon]

Funnily, if I do getByRole('progressbar', { hidden: true }) it solves my issue as it does not call isInaccessible.

Oh right. I had it backwards.

But I think it would be a good idea to skip the error helpers when running in a waitFor*. That would speed things up.

[ValentinH]

I don't see a way to skip the error in this case without explicitly passing a (new) option to getByRole(). Do you have an idea?

[drewschrauf]

I ran into this today while trying to debug why a mocked network request was so slow. I'll try to put together a minimal reproduction but the Cliffs Notes is:

My app code was making a network request and on success would navigate to another route. I was waiting for the new route to load with screen.findByRole('heading', {name: 'xxx'}). Using console.time around the fetch call was showing ~1900ms elapsed. A similar request that would 400 was taking around ~22ms to display the error message. Swapping to screen.findByText('xxx') brought the elapsed time for the former down to ~25ms. I took a look at the profiler and it seems that *ByRole is starving my CPU. I tried both {hidden: true} and {interval: 500} with no improvement. This is in a toy app with a pretty tiny DOM.

The only real difference between the tests seems to be the route change. I'll try to put together a minimal reproduction this weekend to narrow it down.

[kentcdodds]

If I understand correctly were determining the accessibility of everything before matching. Is that right? If so... What if we find the match first and then determine if it's accessible?

[eps1lon]

Is that right? If so... What if we find the match first and then determine if it's accessible?

Pretty sure this is what we do already.

[eps1lon]

I don't see a way to skip the error in this case without explicitly passing a (new) option to getByRole(). Do you have an idea?

We could flip a global variable and then do something like runWithDiagnosticsDisabled(callback) inside waitFor* methods.

[ValentinH]

Is that right? If so... What if we find the match first and then determine if it's accessible?

Pretty sure this is what we do already.

What I understand is that we are searching for all the roles in the container tree in order to explain why the role that was requested was not found.

[ValentinH]

@eps1lon is there already a similar pattern in the existing codebase, if yes I can work on a PR to implement the light error message when run within a waitFor*. 🙂

Actually, I think it should only be for waitForElementToBeRemoved(), for the other ones, the full error message makes sense.

[ValentinH]

I guess I can use the globalObj in https://github.com/testing-library/dom-testing-library/blob/master/src/helpers.js#L1
and create 2 functions to read /write the flag: toggleDiagnosticsInErrors() and shouldPrintDiagnosticsInErrors(). What do you think?

[kentcdodds]

I think we really need to be careful here. I want to give it a little bit of extra thought...

[ValentinH]

Yes this would be quite hacky and just a workaround for a deeper issue: isInaccessible being slow.

[ValentinH]

If you decide that we can go this way, I'm ready to make a PR for the 2 following things:

• replaced the existing cachedIsSubtreeInaccessible() by the cachedGetComputedStyle() I mentioned above. As we cache more, it's even faster.
• implement the "global" shouldPrintDiagnosticsInErrors() check in the getMissingError() of Role queries when run within waitForElementToBeRemoved().

[kentcdodds]

Ok, so I think the simplest solution is to default hidden to true and make the default configurable. I know the arguments against this that @eps1lon will have (we've been through this before) but I'm convinced that opting into this kind of a performance drain in exchange for the confidence is better than having to opt out of it. I worry that people will switch from ByRole to a suboptimal query due to performance issues and I don't think that's wise.

Because this is a breaking change, we should do this in two phases.

1. Make it configurable and release that as a minor version
2. Change the default and release that as a major version (maybe batched with other major changes in we can think of any).

Anyone want to work on phase 1?

[tommarien]

@eps1lon I've setup the repro repo, i've kept as much as possible in because i did not want to remove the problem in some way ( you have a pending collaborator access invite). I've left where to look in the readme ;)

[tommarien]

@eps1lon Any update on this issue ?

[kentcdodds]

🎉 This issue has been resolved in version 7.6.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[domarmstrong]

This doesn't actually improve the performance of getByRole though does it? Just when using with waitFor. I was not using waitFor and getByRole was still unusably slow in a complex form component.

[eps1lon]

I was not using waitFor and getByRole was still unusably slow in a complex form component.

Please open a new issue (since this issue was concerned about ByRole in waitFor) and fill out the template. Having a repro like the one @tommarien provided helps a lot.

[tommarien]

I've just update testing-library/dom in the reproduction repo and see times go from 280ms on average to 38ms ;), so thanks a lot 👍 . If next release of testing-libs dom dependency to > 7.6.0 than everyone can benefit from this :). @eps1lon Thanks and i'll remove the repro repo today

[ctoppel]

I am also seeing very slow times in a large complex DOM (>800ms/getByRole). @domarmstrong Did you open a new issue already?

[kentcdodds]

Hi @ctoppel,

I'll just echo what @eps1lon said earlier:

Please open a new issue (since this issue was concerned about ByRole in waitFor) and fill out the template. Having a repro like the one @tommarien provided helps a lot.

Thanks!

[dbertella]

Hey guys I don't know if anybody else opened a new issue but today I had this exact issue, one of my test suite started to have systematic timeouts after updating to new version of testing library and updating my queries to getByRole
I run a bit of profiling and those are my findings

Let's say I have an Input component similar to this

<Input
  {...el.input}
  placeholder="My El"
  data-testid="MyEl"
/>

In my tests I'm running this:

console.log('pre getByTestId', performance.now())
screen.getByTestId('MyEl')
console.log('after getByTestId', performance.now())
console.log('pre getByRole', performance.now())
screen.getByRole('textbox', { name: /My El/ })
console.log('after getByRole', performance.now())
console.log('pre getByText', performance.now())
screen.getByText(/My El/)
console.log('after getByText', performance.now())

Those are my results before without configure({ defaultHidden: true })

pre getByTestId 10829.295586
after getByTestId 10835.550812
pre getByRole 10836.638695
after getByRole 12147.583377
pre getByText 12148.72231
after getByText 12153.527319

Those are the results after configure({ defaultHidden: true })

pre getByTestId 6450.390944
after getByTestId 6454.832352
pre getByRole 6457.684077
after getByRole 7683.902226
pre getByText 7693.771436
after getByText 7698.411979

In general for me getByRole is slower it's 1 / 2 sec vs 5ms with the other queries

[atsikov]

eps1lon/dom-accessibility-api#323
This PR improves getByRole performance by up to 20% by removing one extra getComputedStyle call

[franksimon90]

Hey guys I don't know if anybody else opened a new issue but today I had this exact issue, one of my test suite started to have systematic timeouts after updating to new version of testing library and updating my queries to getByRole
I run a bit of profiling and those are my findings

Let's say I have an Input component similar to this

<Input
  {...el.input}
  placeholder="My El"
  data-testid="MyEl"
/>

In my tests I'm running this:

console.log('pre getByTestId', performance.now())
screen.getByTestId('MyEl')
console.log('after getByTestId', performance.now())
console.log('pre getByRole', performance.now())
screen.getByRole('textbox', { name: /My El/ })
console.log('after getByRole', performance.now())
console.log('pre getByText', performance.now())
screen.getByText(/My El/)
console.log('after getByText', performance.now())

Those are my results before without configure({ defaultHidden: true })

pre getByTestId 10829.295586
after getByTestId 10835.550812
pre getByRole 10836.638695
after getByRole 12147.583377
pre getByText 12148.72231
after getByText 12153.527319

Those are the results after configure({ defaultHidden: true })

pre getByTestId 6450.390944
after getByTestId 6454.832352
pre getByRole 6457.684077
after getByRole 7683.902226
pre getByText 7693.771436
after getByText 7698.411979

In general for me getByRole is slower it's 1 / 2 sec vs 5ms with the other queries

Same for me, more than 2 sec in some cases, depends of test, and component implementation. Even if I don't consider it really important, difference is there, it's a fact.

I'm just worried about this sentence in the documentation clearly stating that getByRole ...should be your top preference for just about everything. There's not much you can't get with this (if you can't, it's possible your UI is inaccessible).

I have been more than once in the position of seen MRs get rejected, or at least, heavily commented, about wrong query usage on RTL, without any real performance consideration (mainly because some developers don't take the time to test this). It's possible to make this difference noted in the doc, at least until its get improved @kentcdodds? Can save some time of internal discussions for some teams.

[kentcdodds]

I personally think a PR to the docs is warranted. Something like: "NOTE: some have experienced poor performance with the role query, learn more: link"

[arahansen]

I personally think a PR to the docs is warranted

Were the docs ever updated to reflect performance impact of byRole? If not, I would be open to adding these docs

[franksimon90]

@arahansen I didn't take the task at the end, because I'm not sure about how useful is going to be. Even if benchmarks here have some point, after a few more talks about it, turns out that small sized teams or alone individuals are probably going to assume that the doc is talking about a huge performance impact (even if you used the right wording there, or even if you provide the link), and they're going to do it maybe without any real performance assessment on the test stages. That means, final result could be people opting-out of getByRole (and all that implies) for not a really outstanding practical reason.

[Doeke]

I came here after realizing there's a huge performance difference between getByRole and other query methods. Using getByText immediately makes my tests run 8x faster. I understand the reasons for recommending getByRole but that much additional time spent waiting for tests adds up fast for a large project. I think the docs should be up front about this so people can make their informed decisions about what method to use.

[atsikov]

For those who use styled-components a lot, such wrapper could improve getByRole performance by leaving only a few css rules used by @testing-library and thus significantly reducing <style> size

import { ReactNode } from 'react'
import { StyleSheetManager, StylisPlugin } from 'styled-components'

const ALLOWED_RULES = ['display', 'visibility', 'pointer-events']

const simplifiedStylesPlugin: StylisPlugin = (context, content) => {
  if (context === 1) {
    if (!ALLOWED_RULES.some(rule => content.toString().startsWith(`${rule}:`))) {
      return ''
    }
  }

  return undefined
}

export const withSimplifiedStyles = (component: ReactNode) => (
  <StyleSheetManager stylisPlugins={[simplifiedStylesPlugin]} disableVendorPrefixes>
    {component}
  </StyleSheetManager>
)