axe-core-npm and axe-puppeteer silently hung when testing a page

[BogdanCerovac]

Product: cli & puppeteer
I've stumbled upon a case when axe-core-npm (triggered via CLI) and axe-puppeteer silently hangs and I have to manually kill them.
I've copied source HTML to local file and it stucks there as well. I've been using latest versions of both.

Same page tested with browser extension went through but caused some warnings and errors in the console.

Unfortunately I can't reveal the URL/provide HTML because of NDA, but wonder if I can do some more debugging locally (I've tried --verbose in the CLI and I didn't find any options for verbose output via axe-puppeteer, maybe that would be a great start if it's possible).

I was hoping to find a mechanism in axe documentation that would timeout hung analyzes, but couldn't find any - so I decided to make a temporary solution. Please correct me if I am wrong - is there a way to timeout and throw an error if analyze is hung?

Simple temporary fix for the Puppeteer scenario - using Promise.any - that could maybe be useful for somebody.
It isn't perfect and it can introduce memory leaks if axe will hung multiple times, but it saved my problem when only a single URL was problematic.

/* pseudo code, but quite similar to what I used */

let globalTimeoutHandles= {}; // used to save all interval handles with URL as key

// helper function that resolves promise after some time
function timeoutPromise(globalTimeoutHandles, url){
    const axeTimeout = 30000; // ms
    return new Promise((resolve, reject) => {
        globalTimeoutHandles[url] = setTimeout(
            () => {
                console.log("Giving up on AXE for '"+url+"', as it didn't return results within " + axeTimeout + " ms. Url saved for manual checks...");
                resolve(false);
            },
            axeTimeout
        );
    })
}

// wrap axe analyze to a promise, to make it easier to call
async function axePromise(page){
    const results = await new AxePuppeteer(page).analyze();
    return new Promise(async (resolve, reject) => {
        resolve(results);
    });
}

// finally use the pattern with Promise.any (requires newer NodeJS) - this will resolve whole Promise if any of the promises in array resolves and with this we are implementing manual timeout of axe in cases when it hangs...
axeResults = await Promise.any([timeoutPromise(globalTimeoutHandles, url), axePromise(page)]);

// clear interval for that URL
clearInterval(globalTimeoutHandles[url]);

Desired behavior would be to timeout with throwing an error, I think. That would at least make it possible for me to use in try&catch and handle situation manually. My temporary fix could maybe do the cleanup if it could also "kill" the hung analyze...

Expectation: axe should not hung, it should report an error, at least timeout

Actual: It silently hangs. In the CLI output (axe --timer --verbose --timeout=120 https://....) last lines were:

[0224/213812.841:INFO:CONSOLE(3)] "Uncaught TypeError: Cannot read properties of undefined (reading 'expandOnHover')", source: https://www..... (3)
axe page load time: 4.141s
[0224/213813.435:INFO:CONSOLE(11390)] "Canvas2D: Multiple readback operations using getImageData are faster with the willReadFrequently attribute set to true. See: https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-will-read-frequently", source:  (11390)

Motivation: I think this can be a problem in CD/CI pipelines and also when checking things in bulk. Axe should timeout with an error that could be caught. I would also appreciate some info about the cause being displayed, so that I could fix the webpage / inform the owner about problems, if possible...

Running axe-core 4.6.3 in chrome-headless
-@axe-core/puppeteer: 4.5.2
- puppeteer: 18.2.1,
- Node version: Node.js v18.14.2
- Platform:  Windows

[BogdanCerovac]

I've searched other issues here and wasn't able to find similar problems, even tried the solution mentioned in #230 as it is about "stucked CLi", didn't help...

[BogdanCerovac]

And this is the minimum code that I used to reproduce the hung axe locally (or with URL);

const { AxePuppeteer } = require('@axe-core/puppeteer');
const puppeteer = require('puppeteer');
const fs = require('fs');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.setBypassCSP(true);

  const contentHtml = fs.readFileSync('example.html', 'utf8');
  await page.setContent(contentHtml);

  const results = await new AxePuppeteer(page).analyze();
  console.log(results); /* <<<<< didn't come so far >>>>>>> */

  await page.close();
  await browser.close();
})();

(nothing special here, I guess)

[BogdanCerovac]

Chrome extension (axe-core 4.6.3) on same page worked, but triggered a console error (I don't know if it may be related):

content.bundle.js:2 Uncaught (in promise) No response received from frame within timeout of 5000ms.

Digging into minified bundle and this seems to be the source of the console message;

 post: async(e,t,n)=>{
  n(await (0,
    r.Z)(e, "axe-frame-messenger", t, {
        timeout: "axe.ping" === t.topic ? 5e3 : 6e4
    }))
 }

[straker]

Apologies for not responding to this earlier, seems I missed it. I think I coincidentally fixed this issue in axe-core just the other day. Hopefully that means in the next release of axe-core this issue should be fixed. If you wanted to try it out you should be able to install axe-core@next and use that as the axeSource argument when calling AxePuppeteer

const { AxePuppeteer } = require('@axe-core/puppeteer');
const puppeteer = require('puppeteer');
const fs = require('fs');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.setBypassCSP(true);

  const contentHtml = fs.readFileSync('example.html', 'utf8');
  await page.setContent(contentHtml);

  const axeSource = fs.readFileSync(require.resolve('axe-core'));
  const results = await new AxePuppeteer(page, axeSource).analyze();
  console.log(results);

  await page.close();
  await browser.close();
})();

[BogdanCerovac]

No problem, thank you. Will check the axe-core@next version soon and get back to you.

[BogdanCerovac]

Tried now, seems like puppeteer can't serialize the function;

C:\Proj\a11y-tool-site-elements\axe-bug\possible-fix-a\node_modules\puppeteer-core\lib\cjs\puppeteer\util\Function.js:57
            throw new Error('Passed function cannot be serialized!');
                  ^

Error: Passed function cannot be serialized!
    at stringifyFunction (C:\Proj\a11y-tool-site-elements\axe-bug\possible-fix-a\node_modules\puppeteer-core\lib\cjs\puppeteer\util\Function.js:57:19)
    at ExecutionContext._ExecutionContext_evaluate (C:\Proj\a11y-tool-site-elements\axe-bug\possible-fix-a\node_modules\puppeteer-core\lib\cjs\puppeteer\common\ExecutionContext.js:238:73)
    at ExecutionContext.evaluate (C:\Proj\a11y-tool-site-elements\axe-bug\possible-fix-a\node_modules\puppeteer-core\lib\cjs\puppeteer\common\ExecutionContext.js:143:113)
    at IsolatedWorld.evaluate (C:\Proj\a11y-tool-site-elements\axe-bug\possible-fix-a\node_modules\puppeteer-core\lib\cjs\puppeteer\common\IsolatedWorld.js:125:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v18.14.2

Used code from your example (#675 (comment)) as runtime...

I've tried with different versions, stable and next, and got same problem with all.

Last dependency tree:

  "dependencies": {
    "@axe-core/puppeteer": "^4.6.1-alpha.365",
    "axe-core": "^4.6.3-canary.f029271",
    "puppeteer": "^19.7.3"
  }

Any ideas? Can you please share your dependency tree with versions?

[straker]

This is working for me

const { AxePuppeteer } = require('@axe-core/puppeteer');
const puppeteer = require('puppeteer');
const { promises: fs } = require('fs');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  const axeSource = await fs.readFile(require.resolve('axe-core'), 'utf8');

  await page.setContent(`
    <html lang="en">
    <head>
      <title>Test Page</title>
    </head>
    <body>
      <main>
        <h1>Hello World</h1>
        <button id="empty-button"></button>
        <iframe title="iframe" srcdoc="<button id='iframe-empty-button'></button>"></iframe>
      </main>
    </body>
    </html>
  `);

  const results = await new AxePuppeteer(page, axeSource).analyze();
  console.log(results.violations);
  // console.log(results.violations[0].nodes);

  await page.close();
  await browser.close();
})();

And here's my dependencies:

"devDependencies": {
  "@axe-core/puppeteer": "^4.6.0",
  "axe-core": "^4.6.3-canary.f029271",
  "puppeteer": "^18.2.1"
}

[straker]

Might be I forgot the utf8 in the first code example when reading axe-core.

[BogdanCerovac]

Just tested with exactly the same dependencies and code and your example worked fine.

But when testing with my HTML it is the same - things are just stucked, no output whatsoever.

/* I've used this to use local HTML file: */
const contentHtml = await fs.readFile('example.html', 'utf8');
await page.setContent(contentHtml);
console.log(contentHtml); /* This worked well*/ 

const results = await new AxePuppeteer(page, axeSource).analyze();
/* <<<<<<<<<<<<<<<<<<< things stucked here, even if I wait for 20 minutes - no automatic timeout*/

Any tips on how to debug what is happening in analyze()? To find out where it gets stucked?

I am quite sure that HTML I want to test is the cause. It has a lot of iframes with videos and is quite complex and not valid, but would anticipate that analyze() would give up within some reasonable / configured timeout instead of getting blocked with no output...

[straker]

I'd start with opening the page in headed mode (await puppeteer.launch({ headless: false })) and see if the browser gives any error message or logs. If that doesn't produce anything, then you can open the page in headed mode without running analyze, add a script tag to the page that loads axe (<script src="https://unpkg.com/axe-core@next"></script>) and then in the browsers devtool console run axe (await axe.run()). If that passes then it's probably a problem with one of the iframes.

If the example.html file isn't proprietary, you could also upload it to this issue and I could take a look at it (no zip files though).

[BogdanCerovac]

Hi thanks for advice and patience. Unfortunately I can not share the HTML or URL (NDA...) and I know that it makes things difficult. Thanks for understanding.

I've tried with headed mode and yes, there are a lot of errors and warnings in the console, but none of them comes from axe. Most of them are CSP related and some are just JS complaining because of missing elements that are supposed to have events on them (undefined...).

Then I tried injecting the axe on the page and running it manually as you suggested and it worked without problems. Got the results back and all was good (testEngine: {name: 'axe-core', version: '4.6.3-canary.235e632'})

So I wondered if I can maybe just remove all iframes and run the axe-puppeteer as originally planned. It worked. Axe returned results as desired. I've just used this code:

  const selector = 'iframe';
  await page.$$eval(selector, els => els.forEach(el => el.remove()));
  const results = await new AxePuppeteer(page, axeSource).analyze();
  console.log(results.violations);

I wanted to know if problem lies in specific iframes - I suspected the cookie consent provider as their solution is quite complex... It wasn't the cause.

After some try and fail I removed all Google maps iframes ( const selector = 'iframe[src*="https://www.google.com/map"]';) and axe-puppeteer worked fine this time.

So obviously the problem lies in Google maps iframes for this site. Site has at least 20 of them, lazy loaded, but still.

I would still expect that axe would not be blocked in such cases though. Or that it would timeout as mentioned in my bypass (#675 (comment)).

Will try to make a simple test case that will re-create the situation with iframes and won't be proprietary...

[BogdanCerovac]

@straker - sorry for long delay. Found similar issue on a random site I can share here and maybe you can check it out.

https://www.rastoder.si/

Tried with axe-core 4.7.0 in Puppeteer and it often fails the same way - as before mentioned problem.

Using axe extension using same core version (4.7.0) seem to work, but browser console reports caught (in promise) No response received from frame within timeout of 5000ms..

Single iframe on the page is again Google maps (with loading="lazy"), so it seems to me that it is the same issue...

[straker]

@BogdanCerovac Thanks for the link. I'll check it out and see if I can find anything.

[straker]

Alright. I found the issue (at least with the page you gave me). There's an iframe on the page with loading="lazy" that isn't loaded until the page is scrolled. Puppeteer will hang indefinitely if you try to interact with a non-loaded iframe. I'm not sure the best approach at the moment to fix that, but at the very least if you scroll the page to the bottom and then analyze it seems to work.

[BogdanCerovac]

Thank you @straker. This worked OK. I've just made a simple check before scrolling to the bottom;

/* autoscroll from StackOverflow (actually https://github.com/chenxiaochun/blog/issues/38) */

async function autoScroll(page){
    await page.evaluate(async () => {
        await new Promise((resolve) => {
            var totalHeight = 0;
            var distance = 100;
            var timer = setInterval(() => {
                var scrollHeight = document.body.scrollHeight;
                window.scrollBy(0, distance);
                totalHeight += distance;

                if(totalHeight >= scrollHeight - window.innerHeight){
                    clearInterval(timer);
                    resolve();
                }
            }, 100);
        });
    });
}

/* simple condition */
const hasLazyIframes = await page.evaluate(`(()=> [...document.querySelectorAll('iframe[loading="lazy"]')].length > 0)();`);
if(hasLazyIframes){ //https://github.com/dequelabs/axe-core-npm/issues/675
    console.log("Lazy iframes detected, we need to scroll to bottom before audit");
    await autoScroll(page); // if we have iframes we need to scroll to the bottom for axe to work as some are loaded lazily
}