[CI] Enable audio tests in Chrome #23665

cwoffenden · 2025-02-13T12:44:58Z

Step one of a fix for #23131 (the second part in #23695).

Enabled Chrome's FakeAudioOutputStream for the CI machines and bypassed the need for user interaction. The Chrome tests are now enabled with @requires_sound_hardware.

Many attempts at replicating the same functionally were tried with Firefox (details below) but audio would never run so its EMTEST_LACKS_SOUND_HARDWARE was left unchanged.

sbc100

nice!

Does the test pass locally for you now?

cwoffenden · 2025-02-13T18:04:02Z

Does the test pass locally for you now?

Yes, running with the browser flags documented earlier (or having interaction). I'll test all of the parameterised tests tomorrow, I didn't check all variants. MEMORY64 will fail without one of my other PRs, so I'll comment that one out for the time being.

cwoffenden · 2025-02-14T09:40:57Z

Does the test pass locally for you now?

Now checked, and yes, all the parameterised test_audio_worklet tests apart from the _memory64 variant work locally (and now with added asserts that one fails early instead of timing out).

I'll work my way through the other browser tests next week now. It would be useful to have #23659 merged (which doesn't affect anything other than my code) so I can move all the browser audio tests to use the same pattern.

~~One question, does EMTEST_LACKS_SOUND_HARDWARE signify that the test runner actually lacks audio hardware or that the audio won't run because of the interactive requirement?~~ EDIT: the running VMs lack sound support.

emscripten/.circleci/config.yml

Line 309 in 6fd5a1f

EMTEST_LACKS_SOUND_HARDWARE: "1"

I removed @requires_sound_hardware and added the required flags for Chrome and FF. I'll check the results of the logs.

EDIT: answering myself, it looks like there's no virtualised audio, given the assert:

https://app.circleci.com/pipelines/github/emscripten-core/emscripten/40519/workflows/41693480-35f3-44b1-8348-a29330847654/jobs/899772?invite=true#step-108-391780_124

I'll add code to enumerate the hardware (if JS allows this).

More notes:

Multiple attempts at enabling Debian's dummy sound device with modprobe failed. ❌
Chrome's source has a FakeAudioOutputStream, so I'm hoping--use-fake-device-for-media-stream enables it. ✅
Firefox also has a fake audio device.

cwoffenden · 2025-02-14T19:06:59Z

The latest flags for Chrome I tried appear to work, with the AW tests for regular Chrome passing:

https://app.circleci.com/pipelines/github/emscripten-core/emscripten/40523/workflows/bca5a785-e04e-4bf1-8187-01a87982d059/jobs/899865?invite=true#step-108-24450_168

For Firefox it's failing as expected since I've not added its fake audio flags (media.navigator.streams.fake from what I can make out), and the wasm64/2GB tests are failing (needing my MEMORY64 PR).

sbc100 · 2025-02-14T21:01:35Z

.circleci/config.yml

          command: |
-            export EMTEST_BROWSER="/usr/bin/google-chrome $CHROME_FLAGS_BASE $CHROME_FLAGS_HEADLESS $CHROME_FLAGS_WASM $CHROME_FLAGS_NOCACHE"
+            export EMTEST_BROWSER="/usr/bin/google-chrome $CHROME_FLAGS_BASE $CHROME_FLAGS_HEADLESS $CHROME_FLAGS_WASM $CHROME_FLAGS_NOCACHE $CHROME_FLAGS_AUDIO"


If we make this change should we also remove the EMTEST_LACKS_SOUND_HARDWARE setting above?

I'm kind of inclined to make this change as its one separate PR, e.g. [CI] Enable audio testing on chrome?

I didn't have much time today so it was mostly experimenting and coming back to it, but:

I'll remove theEMTEST_LACKS_SOUND_HARDWARE ✅

I'll rename to match its new purpose (I wasn't sure what I'd achieve when starting) ✅

I'll disable all the wasm64 and 2GB tests with audio for now

Is it fine to always enable the dummy sound hardware and bypass the media interaction? I can't think what this would break (and where real sound hardware is present its used as a preference, at least from my manual testing).

I imagine enabling dummy sounds hardware and bypass the interaction check should be fine for all out tests..

I limited removing EMTEST_LACKS_SOUND_HARDWARE to Chrome (FF I can't get to work, documented in the comments).

cwoffenden · 2025-02-15T11:02:29Z

Hmm, FF still broken. To do:

~~look at media.audio_loopback_dev~~ <-- doesn't exist
EMTEST_LACKS_SOUND_HARDWARE for FF
(Set up a local VM with no sound hardware) ✅

sbc100 · 2025-02-15T20:19:55Z

test/webaudio/audioworklet.c

+#ifdef TEST_AND_EXIT
+  // We're in the test harness and exiting is via main_thread_tls_access()
+  emscripten_exit_with_live_runtime();
+#endif


I was thinking or a separate PR to enable audio testing CI.

PR1: enable audio testing in CI
PR2: fix audio worklet test so they fail correctly

Sure, NP. I’ll see how I get on with FF on Monday but I might split it for Chrome first too (I set up a VM with the same Debian and FF and so far, even with no audio device, it doesn’t fail; left to try is headless).

After getting an audio-less Debian VM set-up with all the fallback sound devices removed, I finally managed to recreate what Firefox does. It fails here playing manually, or the test times out otherwise:

emscripten/test/webaudio/audioworklet.c

Line 60 in f9a781b

audioContext.resume();

The promise calls neither the success nor failure route, it just silently fails and the audio remains permanently suspended. I'll call this a Firefox bug and keep the EMTEST_LACKS_SOUND_HARDWARE (I'm satisfied that I've spent enough time going down all avenues and what remains is creating a repro in minimal JS for Firefox).

I'll just enable audio testing in CI for Chrome for now.

sbc100 · 2025-02-18T18:44:52Z

test/test_browser.py

  def test_audio_worklet_modularize(self, args):
-    self.btest_exit('webaudio/audioworklet.c', args=['-sAUDIO_WORKLET', '-sWASM_WORKERS', '-sMODULARIZE=1', '-sEXPORT_NAME=MyModule', '--shell-file', test_file('shell_that_launches_modularize.html')] + args)
+    self.btest_exit('webaudio/audioworklet.c', args=['-sAUDIO_WORKLET', '-sWASM_WORKERS', '-sMODULARIZE=1', '-sEXPORT_NAME=MyModule', '--shell-file', test_file('shell_that_launches_modularize.html'), '-DTEST_AND_EXIT'] + args)


Revert the -DTEST_AND_EXIT part of this change

I'm just doing so (and moving it to the other PR). 😀

sbc100

Great!

Please update the PR description too

cwoffenden · 2025-02-18T19:04:26Z

Tested locally, this doesn't break anything, whereas #23695 breaks the CI as expected.

sbc100

LGTM with update PR description. Do you want to also remove the "Draft" status of this PR?

cwoffenden · 2025-02-18T20:12:22Z

Funny, wasm64 tests like test_sdl2_mixer_wav_dash_l are failing now audio is enabled (since they were flagged as needing audio). I'll disable those tomorrow for now.

sbc100 · 2025-02-18T20:15:42Z

Ah.. good to know. I can take a look at browser64.test_sdl2_mixer_wav_dash_l. If you find any more of those failing tests can you open bugs for them?

cwoffenden · 2025-02-18T20:18:02Z

There seems to be quite a few. I'll make a list tomorrow and mark them as @no_wasm64.

cwoffenden · 2025-02-18T20:29:06Z

They're parameterised so it's only test_sdl2_mixer_wav and test_sdl2_mixer_music. I marked them not to run for now just to verify. I'll see how the CI finished and create bugs tomorrow (I really am done for the day...!).

sbc100 · 2025-02-18T20:30:29Z

test/test_browser.py

@@ -3244,6 +3245,7 @@ def test_sdl2_mixer_wav(self, flags):
    # TODO: need to source freepats.cfg and a midi file
    # 'mod': (['mid'],    'MIX_INIT_MID', 'midi.mid'),
  })
+  @no_wasm64('TODO: SDL2 audio memory64')


I just submitted a PR on the SDL side: libsdl-org/SDL#12332

Can you use that URL as the string argument to no_wasm64 here.

…udio works) (#23719) Now that the audio tests are running in CI (see #23665) this existing AudioWorklet/futex test can be moved from `interactive` to `browser`: https://github.com/emscripten-core/emscripten/blob/32832575e0d200afcccd1d892ff3555baf54bc83/test/webaudio/audioworklet_emscripten_futex_wake.cpp It's had the bare minimum to enable it to compile but is otherwise broken and therefore disabled. The issue is here #22962. A fix is [already done](https://github.com/cwoffenden/emscripten/tree/cw-audio-spinlock-fix) with more advanced lock tests, but needs this to land first to get things in the right order. To confirm the test is broken, comment out the `@disabled` in line 5518 and run with: ``` test/runner browser.test_audio_worklet_emscripten_futex_wake ``` It will fail with: ``` test.js:1003 Uncaught RuntimeError: Aborted(Assertion failed: emscripten_is_main_browser_thread(), at: /emsdk/emscripten/system/lib/pthread/emscripten_futex_wait.c,26,futex_wait_main_browser_thread) at abort (test.js:1003:11) at ___assert_fail (test.js:1945:7) at test.wasm:0x10a8 at test.wasm:0xf34 at test.wasm:0xa78 at WasmAudioWorkletProcessor.process (test.aw.js:113:34) ```

…tests) (#23659) This is splits out the test changes from #23508 adding an interactive parameter test: ``` test/runner interactive.test_audio_worklet_params_mixing ``` This test is also added to the browser tests now #23665 has merged, offering a single test that touches all the various structs and their offsets. When run as part of the `browser` tests it plays for 1 second, for interactive without the `TEST_AND_EXIT` macro it will play continuously. It also tidies a little the shared code (and related tests) in the process. Interactive tests had `_2gb` and `_4gb` options added (which will fail until #23508 lands, but they're not part of the CI). All features of these tests as interactive now work fully on Chrome, Firefox and Safari (Safari has issues with looping, but it doesn't affect the spirit of the test).

cwoffenden force-pushed the cw-audio-fix-tests branch from 7c2c85f to 065759d Compare February 13, 2025 12:45

cwoffenden mentioned this pull request Feb 13, 2025

[AUDIO_WORKLET] tests in the CI don't appear to be run (in test_browser) #23131

Closed

cwoffenden marked this pull request as draft February 13, 2025 13:37

sbc100 reviewed Feb 13, 2025

View reviewed changes

sbc100 reviewed Feb 14, 2025

View reviewed changes

cwoffenden added 10 commits February 15, 2025 10:46

Minor: Firefox now supports wasm64

42e032e

The AW test should now fail, waiting on audio playback

c858dea

Confirmed failure, now flagging as needing audio

29451a4

Correctly halt for btest_exit()

b7fb25a

Attempt to run CI with autoplay enabled

a9132c8

Retrying without wasm64

92c8b5c

Try adding a dummy sound device...

a42c6d4

Try with a full path...

4b80dd2

Try with Chrome flags for audio...

9bcf181

Try dummy audio for FF; remove LACKS_SOUND_HARDWARE

f9a781b

cwoffenden force-pushed the cw-audio-fix-tests branch from 5483e24 to f9a781b Compare February 15, 2025 09:46

cwoffenden changed the title ~~[AUDIO_WORKLET] Tests no longer silently fail~~ [CI] Enable audio tests on Chrome and Firefox Feb 15, 2025

sbc100 reviewed Feb 15, 2025

View reviewed changes

cwoffenden added 2 commits February 18, 2025 19:28

Removed FF flags (they don't work)

e8ca2c4

Flagged tests as requiring sound hardware

9fd83f7

cwoffenden changed the title ~~[CI] Enable audio tests on Chrome and Firefox~~ [CI] Enable audio tests on Chrome Feb 18, 2025

cwoffenden changed the title ~~[CI] Enable audio tests on Chrome~~ [CI] Enable audio tests in Chrome Feb 18, 2025

cwoffenden mentioned this pull request Feb 18, 2025

[test] AudioWorklet tests now correctly run and exit #23695

Merged

Revert back test (changes moved to emscripten-core#23695)

9e63502

sbc100 reviewed Feb 18, 2025

View reviewed changes

sbc100 approved these changes Feb 18, 2025

View reviewed changes

Finish moving test code to other PR

e9f37db

sbc100 approved these changes Feb 18, 2025

View reviewed changes

cwoffenden marked this pull request as ready for review February 18, 2025 19:10

SDL audio tests with wasm64 disabled for now

5d27dc8

sbc100 reviewed Feb 18, 2025

View reviewed changes

Changed todo to URL of PR

9008192

sbc100 approved these changes Feb 18, 2025

View reviewed changes

sbc100 enabled auto-merge (squash) February 18, 2025 21:27

sbc100 merged commit 5858c64 into emscripten-core:main Feb 18, 2025
29 checks passed

cwoffenden deleted the cw-audio-fix-tests branch February 18, 2025 22:19

This was referenced Feb 19, 2025

[CI] Enable audio tests in Firefox #23701

Open

[test] Add audio worklet parameter tests (and tidy other interactive tests) #23659

Merged

[test] Existing AudioWorklet futex test moved to browser tests (now audio works) #23719

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Enable audio tests in Chrome #23665

[CI] Enable audio tests in Chrome #23665

cwoffenden commented Feb 13, 2025 •

edited

Loading

sbc100 left a comment

cwoffenden commented Feb 13, 2025

cwoffenden commented Feb 14, 2025 •

edited

Loading

cwoffenden commented Feb 14, 2025 •

edited

Loading

sbc100 Feb 14, 2025

cwoffenden Feb 14, 2025 •

edited

Loading

sbc100 Feb 14, 2025

cwoffenden Feb 18, 2025

cwoffenden commented Feb 15, 2025 •

edited

Loading

sbc100 Feb 15, 2025

cwoffenden Feb 15, 2025

cwoffenden Feb 17, 2025 •

edited

Loading

sbc100 Feb 18, 2025

cwoffenden Feb 18, 2025

sbc100 left a comment

cwoffenden commented Feb 18, 2025

sbc100 left a comment

cwoffenden commented Feb 18, 2025

sbc100 commented Feb 18, 2025

cwoffenden commented Feb 18, 2025

cwoffenden commented Feb 18, 2025

sbc100 Feb 18, 2025

cwoffenden Feb 18, 2025

[CI] Enable audio tests in Chrome #23665

[CI] Enable audio tests in Chrome #23665

Conversation

cwoffenden commented Feb 13, 2025 • edited Loading

sbc100 left a comment

Choose a reason for hiding this comment

cwoffenden commented Feb 13, 2025

cwoffenden commented Feb 14, 2025 • edited Loading

cwoffenden commented Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

cwoffenden Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwoffenden commented Feb 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwoffenden Feb 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbc100 left a comment

Choose a reason for hiding this comment

cwoffenden commented Feb 18, 2025

sbc100 left a comment

Choose a reason for hiding this comment

cwoffenden commented Feb 18, 2025

sbc100 commented Feb 18, 2025

cwoffenden commented Feb 18, 2025

cwoffenden commented Feb 18, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwoffenden commented Feb 13, 2025 •

edited

Loading

cwoffenden commented Feb 14, 2025 •

edited

Loading

cwoffenden commented Feb 14, 2025 •

edited

Loading

cwoffenden Feb 14, 2025 •

edited

Loading

cwoffenden commented Feb 15, 2025 •

edited

Loading

cwoffenden Feb 17, 2025 •

edited

Loading