Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot use gain node in safari #6144

Open
codingedgar opened this issue Jan 24, 2024 · 9 comments
Open

Cannot use gain node in safari #6144

codingedgar opened this issue Jan 24, 2024 · 9 comments
Labels
Browser issue If there is an underlying issue with the browser that hls.js is running on, this tag should be used. browser: Safari

Comments

@codingedgar
Copy link

What do you want to do with Hls.js?

Hi, thank you so much all your hard work!

I've been debugging an issue for several days and cannot yet come to a fix.

When I use GainNode to control the volume, it works perfectly on Chrome, but not in Safari.

Does someone know why I cannot use Safari + Hls.js + GainNode?

What have you tried so far?

<!DOCTYPE html>
<html>

<body>
  <audio controls>
  </audio>
  <div class="button-bar">
    <button class="play-btn">
      Play
    </button>
    <button class="exp-ramp-minus">
      Exponential ramp gain to 0 in 2 seconds
    </button>
  </div>
</body>
<script src="https://cdn.jsdelivr.net/npm/hls.js@1"></script>
<script>
  const audioEl = document.querySelector("audio");
  const expRampMinus = document.querySelector(".exp-ramp-minus");
  const playBtn = document.querySelector(".play-btn");

  let audioCtx;
  let gainNode;

  audioEl.addEventListener("play", () => {
    if (!audioCtx) {
      audioCtx = new AudioContext();
      const source = new MediaElementAudioSourceNode(audioCtx, {
        mediaElement: audioEl,
      });
      gainNode = new GainNode(audioCtx, { gain: 0.5 });
      source.connect(gainNode);
      gainNode.connect(audioCtx.destination);
    }
  });

  playBtn.addEventListener("click", () => {

    if (Hls.isSupported()) {
      audioEl.removeAttribute('src');
      audioEl.currentTime = 0;

      var hls = new Hls();

      hls.once(Hls.Events.MANIFEST_PARSED, () => {
        audioEl.play();
      });

      hls.loadSource("https://test-streams.mux.dev/x36xhzz/x36xhzz.m3u8");
      hls.attachMedia(audioEl);
    }
  });

  expRampMinus.onclick = () => {
    gainNode.gain.setValueAtTime(
      gainNode.gain.value,
      audioCtx.currentTime
    );
    gainNode.gain.exponentialRampToValueAtTime(
      0.0001,
      audioCtx.currentTime + 2
    );
  };

</script>

</html>

Given this HTML, if you click play and then Exponential ramp gain to 0 in 2 seconds, the volume decreases in 2 seconds.

However, it does not work in Safari.

It seems Hls.js is the only reason the NodeGain interface does not work, if I give the audio source a normal audio (like this example https://mdn.github.io/webaudio-examples/audio-param/), it works fine.

I've tried to create the gain node in all possible places to see if that was the issue (as creating context must happen after interacting with the page in safari but not necessary in chrome, for example).

I also read some of the Hls code to see if the problem was another MediaElementAudioSourceNode, AudioContext, or GainNode, but it seems that it is not mentioned in the source code at all.
@codingedgar codingedgar added Needs Triage If there is a suspected stream issue, apply this label to triage if it is something we should fix. Question labels Jan 24, 2024
@robwalch
Copy link
Collaborator 

      audioEl.removeAttribute('src');
      audioEl.currentTime = 0;

You shouldn't have to set currentTime back to 0 before attaching it to the player, but you might want to call load() on it to make sure the src is cleared (be warned that some UAs might attempt to load the page into the element (src="" or ".")).

I'm not sure what the issue is here, but it sounds browser specific. I'm not sure what the gain buttons are supposed to do exactly in these examples so hard to tell if they are working or not.

Maybe attaching HTMLMediaElements with MSE based sources doesn't work in some browsers. Also note that latest Safari will use ManagedMediaSource on iPad and Mac unless you set preferManagedMediaSource to false in the config.

@codingedgar
Copy link
Author

You shouldn't have to set currentTime back to 0 before attaching it to the player, but you might want to call load() on it to make sure the src is cleared (be warned that some UAs might attempt to load the page into the element (src="" or ".")).

Absolutely, just some remanent of many changes, it has no effect in this example to leave it or remove it.

I'm not sure what the issue is here, but it sounds browser specific. I'm not sure what the gain buttons are supposed to do exactly in these examples so hard to tell if they are working or not.

The primary use case is to substitute the volume to soften pause/stop volume and remove clicking (can read more here: http://alemangui.github.io/ramp-to-value)

Maybe attaching HTMLMediaElements with MSE based sources doesn't work in some browsers. Also note that latest Safari will use ManagedMediaSource on iPad and Mac unless you set preferManagedMediaSource to false in the config.

I am not sure I follow what you mean here; do you mean GainNode stops working when hls uses Media Source Extensions?

just to clarify, safari + gain node = works, safari + hls = works, it is just trying to use safari + hls + gain node = streaming works fine, but gain node has no effect resulting in me not being able to set the volume programmatically and causing clicking.

I think that maybe HLS is routing the output to someplace that cannot be controlled with the gain node in Safari for some security reason (this is a shot in the dark I have absolutely no idea why this is happening)

@robwalch
Copy link
Collaborator

Have you tried delaying binding the GainNode to the audioContext (or getting the audioContext from the element) until after the MediaSource is open? That would be on hls.js "MEDIA_ATTACHED".

@codingedgar
Copy link
Author 

<!DOCTYPE html>
<html>

<body>
  <audio controls>
  </audio>
  <div class="button-bar">
    <button class="play-btn">
      Play
    </button>
    <button class="exp-ramp-minus">
      Exponential ramp gain to 0 in 2 seconds
    </button>
  </div>
</body>
<script src="https://cdn.jsdelivr.net/npm/hls.js@1"></script>
<script>
  const audioEl = document.querySelector("audio");
  const expRampMinus = document.querySelector(".exp-ramp-minus");
  const playBtn = document.querySelector(".play-btn");

  let audioCtx;
  let gainNode;

  // audioEl.addEventListener("play", () => {
  //   if (!audioCtx) {
  //     audioCtx = new AudioContext();
  //     // const source = new MediaElementAudioSourceNode(audioCtx, {
  //     //   mediaElement: audioEl,
  //     // });
  //     // const source =  audioCtx.createConstantSource()
  //     // gainNode = new GainNode(audioCtx, { gain: 0.5 });
  //     // source.connect(gainNode);
  //     // gainNode.connect(audioCtx.destination);
  //   }
  // });

  playBtn.addEventListener("click", () => {

    if (Hls.isSupported()) {
      audioEl.load();

      var hls = new Hls();

      // hls.once(Hls.Events.MANIFEST_PARSED, () => {
      //   audioEl.play();
      // });

      hls.once(Hls.Events.MEDIA_ATTACHED, (name, { media, mediaSource }) => {
        try {
          console.log(media);
          console.log(mediaSource);
          audioCtx = new AudioContext();
          // const source = audioCtx.createMediaStreamSource(media)
          // const source = audioCtx.createMediaStreamSource(mediaSource)
          // const source = audioCtx.createMediaElementSource(audioEl);
          const source = audioCtx.createMediaElementSource(media);
          gainNode = audioCtx.createGain();
          source.connect(gainNode);
          gainNode.connect(audioCtx.destination);
          // audioEl.play();
          media.play();
        } catch (e) {
          console.error(e);
        }
      });

      hls.loadSource("https://test-streams.mux.dev/x36xhzz/x36xhzz.m3u8");
      hls.attachMedia(audioEl);
    }
  });

  expRampMinus.onclick = () => {
    gainNode.gain.value = 0;
    gainNode.gain.setValueAtTime(
      gainNode.gain.value,
      audioCtx.currentTime
    );
    gainNode.gain.exponentialRampToValueAtTime(
      0.0001,
      audioCtx.currentTime + 2
    );
  };

</script>

</html>

like this? still doesn't work, not sure how to get the audioContext from the element

@robwalch
Copy link
Collaborator

HLS.js doesn't route sound output. It buffers media to a MediaSource attached to the media element for decoding and playback (MSE playback). If WebAudio is not working for playback when using an MSE player in a particular browser you should report the issue to the browser vendor.

@robwalch robwalch added Browser issue If there is an underlying issue with the browser that hls.js is running on, this tag should be used. browser: Safari and removed Question Needs Triage If there is a suspected stream issue, apply this label to triage if it is something we should fix. labels Jan 25, 2024
@codingedgar
Copy link
Author

HLS.js doesn't route sound output. It buffers media to a MediaSource attached to the media element for decoding and playback (MSE playback). If WebAudio is not working for playback when using an MSE player in a particular browser you should report the issue to the browser vendor.

hls works fine (i think) i just cannot router the mediasource -> gainnode -> audio context to be able to control the gain node.

maybe i just do not know how to compose the gain node with the media source? should it be different?

@robwalch
Copy link
Collaborator

like this? still doesn't work, not sure how to get the audioContext from the element

That's it. You are doing it right. media is the audio element. Looks like an issue specific to Safari (since it works in Chrome).

@marf
Copy link

marf commented Oct 25, 2024

Hi, we have the same issue. It seems a bug with Safari. Can someone confirm?

@ddurham2
Copy link

ddurham2 commented Dec 10, 2024 •
edited
Loading

FWIW, on iOS, whether in Safari or Chrome, I have been unable to make the basic prescribed use of AudioContext.createMediaElementSource that you see everywhere with any filter (gain or otherwise) when the audio is playing from a streaming source (i.e. without a Content-Length header because it's live). The exact same code works fine if the URL is just to a static file (having the Content-Length header). Yes the CORS headers are set up. The streaming and static sources are from the same server.

The specific behavior is that audio plays unaltered in iOS, but correctly is processed by filters elswhere. Also, no information in the console.

I'm not sure why iOS would care. Nor do I find this limitation documented in apple's docs.

The same internal cause/limitation may exist if hls.js is in play?... which is ultimately where I'm trying to head too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Browser issue If there is an underlying issue with the browser that hls.js is running on, this tag should be used. browser: Safari
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@robwalch @marf @ddurham2 @codingedgar