Aggressive echos during conference calls (mostly on iOS)

[papmodern]

We recently moved our voice conference features on LiveKit. In our first test with a group of five people, we realized strong and loud echo happens mostly for iOS users. Obviously all the participants are not using headphone, but we had a perfect consistency with Agora and we changed it because of the unstable flutter plugin and lack of web support.

I tried to turn echoCancellation on to fix the issue but nothing changed and it seems the option is already turned on by default.
In the documentation it reads Attempt to use echoCancellation option (if supported by the platform). So is there something I need to do to enable it in iOS?

I would appreciate if you guide me to fix this issue

P.S
During implementation I found more inconsistence behaviors in other parts which I had to handle with a dirty workaround. I will list them here in case of informing

• For the first time after connecting to the room, mute or unmute microphone doesn't notify room listeners
• On iOS, enabling and disabling microphone track is inconsistent. If I disable it, it would not enable anymore. Even by disposing the room.

[davidzhao]

@papmodern can you provide an sample clip of the echo that you are seeing?

Which version of flutter SDK are you using?

[papmodern]

This is the sample clips you asked. In the first one participants try to mute the mic to prevent the echo.

RPReplay_Final1672135868.MP4

In this one, we kept the mic open to show it reaches worse situation

RPReplay_Final1672138424.MP4

Participants are in different places. None of them are using headphone.

Versions:
Flutter: 3.3.10
livekit_client: ^1.1.11

P.S
Do not focus on mic icons, for now they are not synching currently between livekit and our server

[davidzhao]

Thank you for the detailed info. we'll investigate.

[cloudwebrtc]

hey @papmodern, are the participating phones in the same room (or close together)? If not, maybe the category of the AudioSession is not set to VoiceChat, so the built-in echo cancellation is not enabled. cc @hiroshihorie

[davidzhao]

Participants are in different places I believe this means they are not close together.

Good call on checking if it's VoiceChat or VideoChat. @papmodern do you explicitly set the AudioSession category? or are you using defaults?

[papmodern]

Hey @davidzhao @cloudwebrtc
I could find this link for setting category
https://docs.livekit.io/client/publish/#advanced-track-management

But it doesn't provide flutter example code. I couldn't find any related function or property in dart api reference

[Parfyonator]

Hello @cloudwebrtc
I'm experiencing exactly the same problem. And it looks like it is only iOS problem. I was unable to reproduce this with Android. An interesting thing is that on initial connection everything works fine, but when one of the participants mutes and then unmutes himself the other participants starts hearing echo of himself going from the other participant (that toggled mute/unmute).

[Parfyonator]

Also the issue with inconsistency in enabling/disabling audio track, mentioned by @papmodern, may by related to this issue since I've experienced the same problem while connecting to a room with iphone and not being able to publish my audio stream. Only several reconnections helped. Interestingly the delay between room connection and audio publishing helped (audio is published after few hundred milliseconds). This issue is reproducible with the example app in this repository.

P.S.: livekit server version is of the latest verison and is passing all the tests

[davidzhao]

@Parfyonator thank you for the additional details. To be clear, are you seeing this when running on iOS with Flutter-web? i.e. in Safari? It seems that @papmodern is seeing it with a Flutter-native app on iOS.

[Parfyonator]

@davidzhao No, I'm using Flutter-native. But the error logs provided in the mentioned issue are very similar to mine. And the issue behavior is the same. That's why I've mentioned it.

[Parfyonator]

Hi @davidzhao @cloudwebrtc. I'm not sure if it would help in any way but I was trying different versions of livekit client. The last version I've tested was 1.1.0 and the issue was still present. But when I've switched to configuration I had prior to upgrading to Flutter 3, the problem went away.

dart: 2.16.2
flutter: 2.10.5
livekit_client: 0.5.9

[papmodern]

@davidzhao Can you please share us any details about the issue? Is the source of the problem clear?
Is there any estimation for fix it? or any workaround to enable this integrated echo cancelation of iOS?

[davidzhao]

@papmodern we need to investigate. It's not clear where the issue is originating from.

[Parfyonator]

@davidzhao @cloudwebrtc, hi. Were you able to reproduce the issue? Maybe you need some additional info to debug?

[hiroshihorie]

Hi @Parfyonator thanks for the following hint:

An interesting thing is that on initial connection everything works fine, but when one of the participants mutes and then unmutes himself the other participants starts hearing echo of himself going from the other participant (that toggled mute/unmute).

I will continue to investigate this one.

[hiroshihorie]

Hi, can you try the audio-mute-unmute-fixes branch by updating your pubspec.yaml:

  livekit_client:
     git:
       url: https://github.com/livekit/client-sdk-flutter.git
       ref: audio-mute-unmute-fixes

Make sure to get / upgrade packages after updating pubspec.yaml file.
Please let me know if it makes a difference.

[Parfyonator]

@hiroshihorie Hi.
Thank you for update. I was able to test it. It looks like the echo is not appearing any more but another thing popped out.
I didn't notice before that during the call both speakers are used to play audio. After mute/unmute toggle only the top speaker plays audio, while the bottom one is muted. So now it's very hard to hear other participants.

UPD: Interestingly, when a user taps on the speaker icon and selects Speaker (even it is marked as selected) everything comes back to normal.

[hiroshihorie]

@Parfyonator Ok thanks for the info.

[papmodern]

@davidzhao @hiroshihorie @cloudwebrtc

So new comments from my side.
I didn't update the livekit version and we are still using livekit_client: ^1.1.11
We had a couple of tests with our latest output.
In the first room we created, surprisingly the echo problem was totally gone and the quality of the voices where awesome (one person with iOS 16.1.1 could manage to make strong echos though!)
None of us where using headphones and even some people where in a big area with persisted natural reverbs. But still the voice quality was superb. However on iOS devices the behavior of mic mute and unmute was totally corrupted. User voices where hearable after calling setMicrophoneEnabled(false) and It seems the voice was duplicating after setting mic enabled.

But the rooms we created after this one, everything returned back to the previous state. Echo came back and mic control was fine.

I can only consider you where doing some change on server side during the first room or randomly the room was created with another profile!

in our pubspec.lock:
webview_flutter: 2.8.0
livekit_client: 1.1.11

We are preparing to publish a new update which already delayed over a month and this inconsistent behavior stops us again. I would appreciate if you make it clear why this inconsistency happens and when we can expect a stable version which fixes voice problems for good.

[hiroshihorie]

@papmodern Hi, I'm working on resolving the issues. It would be great to know if the audio-mute-unmute-fixes branch resolves your echo and mute/unmute issues. You can temporarily try the branch with the method I described above.

There is another issue reported regarding speaker/ receiver which I am looking into.

[papmodern]

@hiroshihorie
I've tested the new branch you mentioned. echo is totally gone, voice quality is perfect

But as @Parfyonator also mentioned, playback source toggles between speaker and ear speaker during mute and unmute the microphone!

[hiroshihorie]

@papmodern Thanks for the feedback, I'm working on the remaining issues.

[papmodern]

Any update?

[cloudwebrtc]

hi @papmodern, Can you test the latest main branch?

[papmodern]

@cloudwebrtc @hiroshihorie
Still inconsistent quality and behavior. Sometimes the session starts, the mics supposed to be off and muted, but they are publishing the microphone.
Still the source of voice swaps between inear and main speaker while muting and unmuting the mic

[cloudwebrtc]

ah, this is confusing, I'll keep checking this issue.

[cloudwebrtc]

just reproduced the issue of the inconsistent switch operation for audio output, so, this should be a simple bug.

[cloudwebrtc]

Still the source of voice swaps between inear and main speaker while muting and unmuting the mic

This may be a tricky problem, due to the mic privacy indicator fix, when we are in non-PlayAndRecord mode, the sound cannot be switched between the earpiece and the speaker, because the sound can only be output from the speaker in AVAudioSessionCategoryPlayback mode,
This directly leads to a result. When we are using the earpiece for two-way conversation (PlayAndRecord), after setting track.mute(true); (turning into Playback), the sound cannot continue to be output from the earpiece, and it will suddenly become louder and output from the speaker. This results in a very poor experience.

So there is a trade-off

1. Make the mic indicator light on, even if we have muted the local mic, the advantage is that the sound can still be switched from the earpiece, speaker, and external Bluetooth headset
2. Let the mic indicator light turn off after track.mute. At this time, the sound needs to be fixed on the speaker, and it will be invalid if it is cut into the earpiece

[cloudwebrtc]

@papmodern
This is a fix for option 2, #243 The result is that no matter mute or umute mic, the sound will be output from the speaker,

[papmodern]

@cloudwebrtc thanks for the commit. I will add it to our next test build. So is echo problem one of these poor qualities you are mentioning?
Because low quality but understandable voice of a human works well for us

[cloudwebrtc]

The echo issue should be fixed by @hiroshihorie.

[Peng-Qian]

@cloudwebrtc, would it be feasible to implement Solution 1: "Make the mic indicator light on, even if we have muted the local mic, the advantage is that the sound can still be switched from the earpiece, speaker, and external Bluetooth headset"

I would argue that the light's on/off status might not be as disruptive to the user experience as the lack of a crucial function: the ability to switch between the speaker and earpiece. Notably, most video call applications support switching to the earpiece during a call, a feature that significantly enhances user experience.