AVPlayer audio does not mix with Room audio.

[podkovyrin]

If you have AVPlayer that plays some video with sound it will become muted once video call established. And after that nothing helps: I've tried almost all combinations of AVAudioSession's AVAudioSessionCategoryOptions, AVAudioSessionSetActiveOptions, tried to setActive:NO and then YES back again before and after call connection.
It also doesn't matter in which position iPhone's sound switch is.
There is only thing that restores sound in player and keeps it going in the call is leaving application (with home button) and open it back.

[piyushtank]

@podkovyrin Playing audio+video with AVPlayer when a video call is established is not currently supported by Video SDK. We are discussing internally on supporting this feature, I will update the ticket when we have more information on this.

[ceaglest]

Hi @podkovyrin,

As @ptankTwilio mentioned we have been discussing this feature request internally.

The problem with using AVPlayer or AVAudioPlayer while connected to a Room is that the media engine is configured for echo cancellation and doesn't know about your player's audio content. Because of this, the echo canceller is fighting with the audio from your player. This is definitely a solvable problem, but I'd like to understand more about your use case first.

If you have AVPlayer that plays some video with sound it will become muted once video call established.

At the moment, the audio pipeline is engaged before the first frame of video / audio flows from the remote Participant. Is the goal to seamlessly playback content from AVPlayer until the Room audio / video content starts and then cross fade between the two? If so, would you be okay with not being able to share or record audio until AVPlayer is stopped?

The reason why I ask is because we are considering two solutions to the problem.

1. Disable audio from the media engine entirely. Re-enable when the pre-roll (AVPlayer) content has completed.

2. Operate the media engine in a playback only audio mode. Switch to playback and recording once the pre-roll (AVPlayer) content has completed. Playback only support has also been requested in this ticket: Support for playback only audio #72.

I'm looking forward to hearing your feedback @podkovyrin. If anyone else has interesting use cases, and would benefit from these APIs please share them on this ticket.

Best,
Chris Eagleston

[DabKick]

Hi @ceaglest,

We are trying a similar project in our company. In addition to the feature requested by @podkovyrin, we would like to duck the volume of the AVPlayer when someone talks on the Twilio Room audio/video channel and then resume the volume of AVPlayer when no one is talking. Is that possible?

Thanks,
Balaji

[podkovyrin]

Hi @ceaglest,

Thanks for your detailed response!
Our use case is to play a video and having a call simultaneously and sound should be mixed from a video call and AVPlayer. That's why any switching between audio stream from a call and sound from a video is not we're trying to accomplish.
I don't know the nature why that workaround (with leaving-entering app) works, but it does pretty well - there is no echo or noise in the audio from call.

[ceaglest]

@podkovyrin okay, so sounds like the AVPlayer content will play the entire time you are connected to a Room. If thats the case does your Client need to share audio or just receive it from other Participants?

Hi @DabKick. Once we add playback only support this should work, assuming you don't need to record audio while the AVPlayer is active. With 1.0.0-beta13 you could use the getStats API to determine the current volume of a TVIAudioTrack.

Check out the docs for: getStatsWithBlock: and TVIAudioTrackStats.

We are also considering adding audio levels to TVIAudioTrack, so you don't have to fetch full stats just to know this information.

Best,
Chris

[podkovyrin]

@ceaglest

sounds like the AVPlayer content will play the entire time you are connected to a Room.

Yes, you're right.

does your Client need to share audio or just receive it from other Participants?

No, the call itself should transfer only audio from microphone, in other words it should be the same simple video call like in example.

[Amrit1901]

@ceaglest

Once we add playback only support this should work, assuming you don't need to record audio while the AVPlayer is active

So does this mean that the new SDK version won't support recording a livestream and play media with AVPlayer at the same time? Is it something possible in the current version. We did some testing and it seems like the AVPlayer object always blocks an ongoing livestream's audio. Is this because the audio track gets "damaged" and we have to create one again?

[ceaglest]

Hi @Amrit1901,

Thank you for the feedback. It sounds like it may be possible to support both AVPlayer playback, and Room Media playback / recording at the same time, but we will need to investigate this issue further. From my experience with WebRTC it is common to disable audio from the media engine while using AVPlayer content, but I'm open to the possibility of having both coexist.

My apologies for the lack of progress on this issue. We know that this is an important use case, and will be focusing more effort on it once we ship our first General Availability release at the end of the month.

Thanks,
Chris Eagleston

[ceaglest]

Hello,

Now that we've shipped our first GA release I've started looking into AVPlayer interoperability. Hopefully I will have more updates to share soon.

Best,
Chris Eagleston

[ceaglest]

Hi,

I've been doing some experimentation with AVPlayer and it is indeed possible to use it alongside a TVIRoom, all while sending & receiving audio. The solution involves using the TVIAudioController (CallKit) category methods to ensure that the application has manual control over the AVAudioSession.

I've completed an internal PoC that works reliably, and requires little or no changes to the SDK itself. I'll converse with my team members, and we will follow up with new sample code demonstrating the use of AVPlayer alongside our Video SDK.

Best,
Chris

[piyushtank]

Hi,

We have written a sample app to demonstrate how to use AVPlayer in your app with TwilioVideo. As @ceaglest mentioned earlier, the app is using TVIAudioController (CallKit) category methods to ensure that the application has manual control over AVAudioSession.

Please try it out and let us know if you have any questions.

Sample App: https://github.com/twilio/video-quickstart-objc#examples

Thanks,
Piyush

[MrBendel]

Running into the same issue. I get really inconsistent behavior, sometimes the audio between users drops out and sometimes the audio from video plays through the earpiece not the speaker.

I followed the code in the example app. In best case, I get audio between participants and audio from the video, but as soon as the video changes, everything gets screwed up again.

I've tried the following to no avail. It doesn't always respect the output to speaker.

[_audioSession setCategory:AVAudioSessionCategoryPlayAndRecord withOptions:AVAudioSessionCategoryOptionDefaultToSpeaker error:nil];

[piyushtank]

@MrBendel During the call setup, Video SDK changes the AVAudioSesssion's category and mode. I am suspecting, the AVAudioSession properties you are setting in your app gets overwritten by the SDK. So, rather setting properties on AVAudioSessin, I would recommend using TVIAudioController APIs instead. If you want to enforce speaker audio route you can set audio output using TVIAudioController API, here is the code snippet to do so -

 [[TVIAudioController sharedController] setAudioOutput:TVIAudioOutputVideoChatSpeaker];

Please let me know if you have any questions.

[MrBendel]

Thank you for the reply! I am trying that exact snippet currently – but the movie player always seems to default to the earpiece not the speaker.

[piyushtank]

@MrBendel In our AVPlayerExample sample app, if you modify the setupAudioSession to following you should the AVPlayer audio flowing through the speaker always. Also, can you check if the speaker volume is high while trying this out? -

- (void)setupAudioSession {
    // In this example we don't want TwilioVideo to dynamically configure and activate / deactivate the AVAudioSession.
    // Instead we will setup audio once, and deal with activation and de-activation manually.
    [[TVIAudioController sharedController] configureAudioSession:TVIAudioOutputVideoChatDefault];

    // This is similar to when CallKit is used, but instead we will activate AVAudioSession ourselves.
    NSError *error = nil;
    [[AVAudioSession sharedInstance] setActive:YES error:&error];
    if (error) {
        [self logMessage:[NSString stringWithFormat:@"Couldn't activate AVAudioSession. %@", error]];
    }

    [[TVIAudioController sharedController] setAudioOutput:TVIAudioOutputVideoChatSpeaker];
    [[TVIAudioController sharedController] startAudio];
}

Also, I have created the branch where AVPlayer and remote participant's audio gets played on the speaker.

Let me know if you have any question.

[MrBendel]

Thanks – I'm doing the same logic. It appears to work the first time – but any subsequent videos played don't play through the speaker. Your demo only ever sets up one video player. My problem is any videos that are created and played after the first one ends only ever play through the earpiece.

[piyushtank]

@MrBendel Is it possible for you to modify the sample code and share a branch to demonstrate the problem?
It could be the case that AVPlayer is changing the AVAudioSession properties. One thing I would try is setting [[TVIAudioController sharedController] setAudioOutput:TVIAudioOutputVideoChatSpeaker]; before the every AVPlayer's play? Also, can you provide the details about the iOS version and the device you are trying it?

[MrBendel]

Oddly enough, calling [[TVIAudioController sharedController] setAudioOutput:TVIAudioOutputVideoChatSpeaker]; whenever a video plays causes the video to always play through earphone, not speaker. 😭

Also when this happens, if you listen through the earphone, the video is really quiet and there is a lot of static noise.

We're testing on iOS 11.1.

[cipriancaba]

We are doing a video streaming app where multiple participants watch the same video stream.. Once we start playing the video, all the participants audio gets muted.. Please provide a solution for this scenario

[piyushtank]

@cipriancaba Thanks for reaching out. I am suspecting, when you start the video streaming, it reconfigures the AVAudioSession which results in no participant audio. We are working on providing better Audio APIs with our Video SDK, however, the problem you are observing can be worked around following way -

Before starting the AVPlayer call resetAudioSession

Once the AVPlayer is started call setupAudioSession

I would recommend you to try AVPlayerExample sample app? It does not play the streaming audio and the remote participant's audio at the same time, however, it demonstrates usage of audio APIs. Also, it is very easy to play the audio from video streaming audio and remote participant's audio at the same time by changing some code.

Let me know if you have any question.

[thebrandontyler]

I have a similar question related to audio. I'm trying to play a few sound effects when users tap buttons, etc during the video call. I don't really need the remote participant to hear the sound effect, but the audio for the sound effect is very low (the video audio seems to be ducking the sound effect audio). Is there any way to play these sound effects at full volume? I don't necessarily want to manually override and force the video call audio to be ONLY played through the speaker. I want all sounds to be played at an equal volume on speaker, headphones, etc.

UPDATE: I tried TVIAudioController.shared().stopAudio, and then I tried to play the sound effect and then startAudio again after the sound effect plays, but that didn't work.

[cipriancaba]

Hey @ptankTwilio My usecase is a little more specific since I am using the react native implementation.. So, we have the videoplayer found here
This sets the audio session to:

      [[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayback error:nil];

Now, I've updated the react native twilio implementation to call setupAudioSession and resetAudioSession as you mentioned.. We call resetAudioSession before playing the video and setupAudioSession after the video started playing.. However, the audio from the twilio streams is still getting muted
Here are the implementation details (relevant changes are in the RCTTWVideoModule.m)

I've been trying to figure this out all day, but I'm not sure what I'm missing.. The reset and setup methods are being called correctly as I've confirmed this via NSLog

Thank you very much for your help, really appreciated

[piyushtank]

@cipriancaba Thanks for reaching out. I will try it out on our AVPlayerExample and get back to you soon.

We have also released 2.0.0-preview8 which deprecates the TVIAudioController has the new TVIDefaultAudioDevice. The TVIDefaultAudioDevice gives you more control configure AVAudioSession. I would recommend trying it out. See 2.0.0-preview8 changelogs for more details.

[cipriancaba]

@ptankTwilio Would love to update to 2.0 but the current react-native implementation only supports 1.3.8

[piyushtank]

@cipriancaba Unfortunately with our existing APIs watching a video (audio+video) and playing remote participant's audio, side by side, is not supported used case. The behavior is undefined when two AudioUnits (one in TwilioVideo SDK and another from video content) are running side by side at the same time.

We are working on 2.x APIs where developers can provide their own audio device to have full control over the audio.

I apologize for any inconvenience caused by this.

[cipriancaba]

Hey @piyushtank, I've updated the react native wrapper to 2.0.0-alpha9

I've looked over the changes to TVIDefaultAudioDevice but honestly I am not sure what would be the way to achieve what I need (supporting multiple sounds at the same time)

I've tried the default behaviour, the one from example 1 and example 2 in the changelog for preview8
https://www.twilio.com/docs/api/video/changelog-twilio-video-ios-version-2x
Example 1

TVIDefaultAudioDevice *audioDevice = [TVIDefaultAudioDevice audioDevice];

//...connect to a Room with audioDevice. By default, the audio route will be configured to the speaker.

room.audioDevice.block =  ^ {
    // We will execute `kDefaultAVAudioSessionConfigurationBlock` first.
    kDefaultAVAudioSessionConfigurationBlock();

    // Overwrite the audio route
    AVAudioSession *session = [AVAudioSession sharedInstance];
    NSError *error = nil;
    if (![session setMode:AVAudioSessionModeVoiceChat error:&error]) {
        NSLog(@"AVAudiosession setMode %@",error);
    }

    if (![session overrideOutputAudioPort:AVAudioSessionPortOverrideNone error:&error]) {
        NSLog(@"AVAudiosession overrideOutputAudioPort %@",error);
    }
};
room.audioDevice.block();

Example 2

TVIDefaultAudioDevice *audioDevice = [TVIDefaultAudioDevice audioDeviceWithBlock:^ {

    // Execute the `kDefaultAVAudioSessionConfigurationBlock` first.
    kDefaultAVAudioSessionConfigurationBlock();

    // Overwrite the category to `playback`
    AVAudioSession *session = [AVAudioSession sharedInstance];
    NSError *error = nil;
    if (![session setCategory:AVAudioSessionCategoryPlayback
                         mode:AVAudioSessionModeVideoChat
                      options:AVAudioSessionCategoryOptionAllowBluetooth
                        error:&error]) {
        NSLog(@"AVAudioSession setCategory:options:mode:error: %@",error);
    }
}];

TVIConnectOptions *connectOptions = [TVIConnectOptions optionsWithToken:token
                                                                  block:^(TVIConnectOptionsBuilder *builder) {
                                                                builder.audioDevice = audioDevice;
                                                            }];

TVIRoom *room = [TwilioVideo connectWithOptions: connectOptions]

Still no luck and with example 1 the sound is not working not even before starting the video.

The code I used can be mostly found in this class over here: https://github.com/cipriancaba/react-native-twilio-video-webrtc/blob/update-to-twilio-sdk-2.0.0/ios/RCTTWVideoModule.m

I would really really appreciate if you could help me with a configuration for TVIDefaultAudioDevice that can support the sound from the twilio participants and AVPlayer at the same time.

[ceaglest]

Hi folks,

There are definitely a number of issues with using AVPlayer alongside an Audio Unit graph, like Twilio Video (and other WebRTC based products) operate for full duplex real-time audio.

We have been looking for solutions to this problem, and so far the best we could do is get access to the audio from AVPlayer, and mix it with audio being played back from Twilio Video. There is a work in progress PR which does this - #325

I'm posting it here, because some might be interested. The implementation is a mix of Objective-C and Swift, with nearly all of the audio code being written in Objective-C or straight C (for RT safety).

Regards,
Chris

[James3432]

Hi @ceaglest ,

I've been following this issue for some time, and am really happy to hear about these developments - thanks! I'd be really grateful if you could advise whether you expect this to work for my use-case:

• I have an app which sends audio + video to a room, but does not need to play audio or video from other participants (so it's like a one-way video call)
• The app needs to play its own audio (ie. AVPlayer) locally, whilst continuing to participate in the Twilio room as above
• Originally, this didn't work with any Twilio solution. When the v2 update came, I was able to get it working (using the ExampleAVAudioEngine), but the AVPlayer seems as if echo cancellation is being applied to it. Your first comment in this thread suggests that's what is happening.

The reason I ask is I'll be looking to update the react native wrapper with this PR, but wouldn't want to attempt that if there's a simpler solution. I had hoped I wouldn't have problems since I'm not mixing room audio with AVPlayer audio, but I guess sending audio to the room still requires the engine to handle both.
(Thanks @cipriancaba for previous work on the react native wrapper. Did you get it working for your scenario in the end?)

Thanks very much for your work on this. This is critical to the functionality of my app, so I'd be very grateful for any help you can give.
James

[ceaglest]

Hi @James3432,

Is it accurate to say that you need to share the microphone, share video, and play audio + video from an AVPlayer? This could work, but with the limitations mentioned in my PR.

... but the AVPlayer seems as if echo cancellation is being applied to it.

Yep, this has been my experience.

The reason I ask is I'll be looking to update the react native wrapper with this PR, but wouldn't want to attempt that if there's a simpler solution.

I don't think you need the whole solution if you're not sending the AVPlayer audio to the Room, but you would still want the audio device for playback + mic recording, and the AVPlayer setup with MTAudioProcessingTap.

Thanks very much for your work on this. This is critical to the functionality of my app, so I'd be very grateful for any help you can give.

You are welcome. Unfortunately I will be out of the office for the rest of the week, but when I return I will see when we can schedule the work to complete the open PR.

Best,
Chris

[James3432]

Thanks @ceaglest

Is it accurate to say that you need to share the microphone, share video, and play audio + video from an AVPlayer?

Yes, except I don't need to play video from the AVPlayer (just audio).

Any chance you could give just a bit more detail on how I could use the open PR to achieve this?

Thanks,
James

[ceaglest]

Hi James,

Any chance you could give just a bit more detail on how I could use the open PR to achieve this?

Sure. I just got back into the office today, so I will take another look at the PR and get back to you.

Thanks,
Chris

[James3432]

Hi @ceaglest ,

Any update on this issue or the PR?

If you know of any way to disable the echo cancellation or auto-gain control altogether, that could also be a solution.

Thanks,
James

[ceaglest]

Hi @James3432,

Unfortunately, I haven't had a chance to take another look at this PR since February. I do apologize, and I'll note that others have also asked if/when it will get merged. I will discuss with the iOS Video team how much effort would be involved in taking the PR to the finish line.

If you know of any way to disable the echo cancellation or auto-gain control altogether, that could also be a solution.

One thing you could try relatively easily, if you haven't already, would be to drop in ExampleCoreAudioDevice, which does not use echo cancellation. It's possible that this would fare better along side a vanilla AVPlayer instance then using TVIDefaultAudioDevice, but I have not tried this myself since echo cancellation still needed any time the Client is publishing (recording) audio.

Edit: I see that your use case still requires publishing audio. The problem you will face is the published microphone audio will include some of the AVPlayer audio picked up by the microphone.

Best,
Chris

[reinhardholl]

I am experiencing a slightly different issue. We have a sequence of videos playing using expo-video. The call would join fine but as soon as the video changes mid call all call audio is lost. Our videos are muted but our twilio call still loses all audio.

Any ideas?

[piyushtank]

@reinhardholl - Can you elaborate when you said "video changes mid call" what exactly do you do? In general, audio and video should work independently.

[reinhardholl]

@piyushtank Sorry for the confusion. We play a series of videos usingexpo-video during our call. Then the expo-av video changes, our Twilio call loses audio.. The Twilio video is fine.