Remote video track sometimes distorted on Pixel 3

[shaunpanjabi]

Description

When the Pixel is the Remote Participant the video track on the local participants screen sometimes gets distorted.

Steps to Reproduce

1. Enter at twilio room from Device A (Pixel 3 or 3XL) and Device B(Any other device)
2. Publish video tracks
3. Observe that on Device B video is sometimes distorted

Expected Behavior

Video should not be distorted

Actual Behavior

Video looks normal on Device A's side however Remote Track published on Device B's side is distorted. Device B's.

Video can start out normally, then becomes distorted after a few seconds, and can revert back to being normal.

Reproduces how Often

Hard to say the specific percentage, but it has happened several times on several Pixel devices.

Versions

All relevant version information for issue.

Video Android SDK

5.1.0

Android API

Android Q

Android Device

Pixel 3, Pixel 3XL

[aaalaniz]

Hey @shaunpanjabi

Can you provide a Room SID and device logs with logging turned up when the issue occurs? To turn up logging execute the following snippet anytime before connecting to a Room.

Video.setLogLevel(LogLevel.ALL);
Video.setModuleLogLevel(LogModule.WEBRTC, LogLevel.ALL);

Thanks!

[shaunpanjabi]

Room SID: RM1b541cb7b437c97baf99264b953f2e25
logs.txt
The issue happened toward the end of the logs I believe. This is the logs from the Pixel device side.

[aaalaniz]

Hey @shaunpanjabi

I noticed that in this room, both devices are using H.264. I'm curious, if you prefer VP8 do you still experience this issue?

Thanks!

[shaunpanjabi]

@aaalaniz Yes just tried VP8 and still seeing this issue.

[puneeta-medopad]

Hey @aaalaniz,

I am facing the similar issue on the Pixel devices (Pixel 3 XL, Pixel 4 and Pixel 4 XL) all of them are running on Android 10 but it doesn't happen on the Pixel 3 device which is running on Android 9. I am also able to reproduce the same issue on Twilio video Quickstart app. Simply run the app on any of those 3 devices mentioned above. The video frames appears to be very laggy on Pixel 4 and 4XL but on Pixel 3 XL, the video appears to be fine on the device but on the remote participant side, the video seems to be completely distorted.

Attached logs here:
Twilio_Video_Call_Logs.zip

Out of curiosity, I tested my app on Samsung Galaxy S10 (running on Android 10) and it works very well.

Here is the screenshot from Twilio quick start on Pixel 4 and 4XL:

Here is the screenshot on the remote participant side (on the web) when sending the video from Pixel 3XL:

Please let me know if you need further information. Thanks!

[paynerc]

@shaunpanjabi,

Sorry for the late response on this issue as I needed to get a Pixel 3 for testing. After reproducing this issue locally and further investigation this issue turns out to exist in the hardware VP8 encoder on the Pixel 3 devices. There is a WebRTC issue filed for this exact issue: https://bugs.chromium.org/p/webrtc/issues/detail?id=11337.

The immediate workaround that you can take is to disable the hardware VP8 encoder on the affected devices. To do this you would need to call:

MediaCodecVideoEncoder.disableVp8HwCodec();

before joining a room. While this isn’t the most efficient and long term solution, this will get you unblocked with sharing video from a Pixel 3 device until WebRTC has a chance to triage and come up with a patch that can be applied prior to encoding the video frames in the hardware encoder.

Let me know if you are still experiencing issues after disabling the VP8 hardware encoder.

Ryan

[muhanics]

@paynerc I've been having this exact same issue and your solution solved it for me, I have noticed the video quality from the Pixel 3 is now significantly lower - do you have any recommendations for improving this (while using the workaround)?

Note: I'm actually using the Twilio React Video App repo and have done the following (which should work on the same concept as your solution):

if(mobileModel === 'Pixel 3'){ //using a plugin to detect mobile model
  connectionOptions.preferredVideoCodecs = ['H264'];
}else{
  connectionOptions.preferredVideoCodecs = [{ codec: 'VP8', simulcast: true }];
}

[rbarbish]

Any updates on this issue @paynerc ? Could we come up with a more comprehensive solution i.e. A complete list of all the models we need to call "MediaCodecVideoEncoder.disableVp8HwCodec();" on? Also what plugin are you using to detect model @muhan-fixter?

[muhanics]

@rbarbish The plugin is called react-device-detect.

https://www.npmjs.com/package/react-device-detect

[rbarbish]

In my Android code I'm adding the following to the OnCreate of the RoomActivity.java file:
if (Build.MODEL.contains("Pixel 3")) { MediaCodecVideoEncoder.disableVp8HwCodec(); }

It seems to resolve the screen tearing issue but the quality is quite diminished, if anyone has a better solution please let me know...

[puneeta-medopad]

I think it might not be the perfect solution but still the best solution I am able to find to solve this problem by using com.twilio.video.CameraCapturer instead of com.twilio.video.Camera2Capturer. Here is the code I am using:

class CameraCapturerManager(context: Context, private val cameraSourceProvider: CameraSourceProvider) {

    lateinit var cameraCapturer: CameraCapturer

    val videoCapturer: VideoCapturer
        get() = cameraCapturer

    init {
        initialiseCameraCapturer(context)
    }

    private fun initialiseCameraCapturer(context: Context) {
        cameraCapturer = CameraCapturer(context, cameraSourceProvider.getAvailableSource(), CameraCapturerListener())
    }

    fun switchCamera() {
        cameraCapturer.switchCamera()
    }
}

class CameraSourceProvider @Inject constructor() {

    fun getAvailableSource(): CameraSource {
        return when {
            CameraCapturer.isSourceAvailable(CameraSource.FRONT_CAMERA) -> CameraSource.FRONT_CAMERA
            else -> CameraSource.BACK_CAMERA
        }
    }
}

class CameraCapturerListener : CameraCapturer.Listener {

    override fun onError(errorCode: Int) {
        Timber.e(RuntimeException(getErrorDescription(errorCode)), "Error while capturing camera")
    }

    override fun onFirstFrameAvailable() {
        Timber.i("CameraCapturerListener - onFirstFrameAvailable")
    }

    override fun onCameraSwitched() {
        Timber.i("CameraCapturerListener - onCameraSwitched")
    }

    private fun getErrorDescription(errorCode: Int): String {
        return when (errorCode) {
            CameraCapturer.ERROR_CAMERA_FREEZE -> "Camera freeze"
            CameraCapturer.ERROR_CAMERA_SERVER_STOPPED -> "Camera server stopped"
            CameraCapturer.ERROR_UNSUPPORTED_SOURCE -> "Unsupported camera source"
            CameraCapturer.ERROR_CAMERA_PERMISSION_NOT_GRANTED -> "Camera permission not granted"
            CameraCapturer.ERROR_CAMERA_SWITCH_FAILED -> "Switch camera failed"
            else -> "Unknown Error"
        }
    }
}

Using this solution you don't need to put any model name checks or anything. Also, it seems that there might be a problem in com.twilio.video.Camera2Capturer implementation as the old implementation still works fine.

Please let us know if you still see any issues or any new findings that you might want to share with all of us.

[koudai-kikuchi-coconala]

@puneeta-medopad's answer worked perfectly on my Pixel4 device.

I just changed a part of codes in quickstartKotlin, then it worked!

// CameraCapturerCompat.kt

    init {
//        if (Camera2Capturer.isSupported(context) && isLollipopApiSupported()) {
//            cameraManager = context.getSystemService(Context.CAMERA_SERVICE) as CameraManager?;
//            setCameraPairs(context)
//            camera2Capturer = Camera2Capturer(context,
//                    getCameraId(cameraSource),
//                    camera2Listener)
//        } else {
//            camera1Capturer = CameraCapturer(context, cameraSource)
//        }
        camera1Capturer = CameraCapturer(context, cameraSource)
    }

[rbarbish]

@koudai-kikuchi-coconala - Tried the same thing in CameraCapturerCompat.java for my Pixel 3a and resulted in the same distorted video. Disabling Camera2Capturer in favor of CameraCapturer does not work for me. Only thing that has worked is MediaCodecVideoEncoder.disableVp8HwCodec(); in the OnCreate

[fcorbi]

i am encountering the same bug of @koudai-kikuchi-coconala and @puneeta-medopad on pixel4 XL and disabling the camera2Capturer seems to be the only way.
Additionally MediaCodecVideoEncoder.disableVp8HwCodec() is useless in my case.

So definitely i think we are experiencing 2 different problems between pixel 3 (solved by MediaCodecVideoEncoder.disableVp8HwCodec();) and pixel 4 XL (solved disabling the Camera2Capturer)

The problem is also in quickstartKotlin but not in AdvancedCameraCapturerActivity

Furthermore i experience the low fps (became 0) with Camera2Capturer when i frame a source of light

Maybe this could be a problem with the AF/AE flags maybe?

On the other side, it works perfectly on Samsung Galaxy S10, so the smartphone is the key to debug this.

[dannnnthemannnn]

I'm on a pixel 4XL and I don't have any android code to update. It seems that setting the preferred codec to H264 doesn't help and I don't have the ability (as far as I can tell) to disable to codec from javascript. Does seem like two different issues to me possibly?

[dannnnthemannnn]

Looks like the prefered codec above was working but it was still messing up on pixel 4 due to sizing issues? I changed my connection settings to give width and height instead of a range and it starting working (albeit with very bad quality with H264).

{
width: 1280,
height: 720,
frameRate: 24,
facingMode: "user",
};

[dylanrjames]

It seems like the comments in this thread each have slightly different behavior around this issue, and my experience with this is slightly different as well, so just wanted to pitch in my 2 cents here.

I've pretty extensively tested this on a Pixel 4 XL, but have tested most of the things here on a Pixel 3 and a Pixel 4a and seen the same behavior across all three.

We had seen this in calls seemingly randomly, but not frequently enough to reproduce/test changes reliably. I found that if you twist the phone around vigorously as you answer a call and join a room, you can cause this issue pretty reliably. It looks like when you're first connecting, the video streams from these devices are a little pixely/low bitrate for the first 5-10 seconds of the call until it settles on a decent quality. If you're spinning the phone and drastically changing the video contents while it's "adjusting" like this, it seems to pretty consistently cause the striped video being reported here.

I will point out that we have seen issues where this behavior occurs in the middle of calls even when the video stream is almost entirely "still" and unchanging. The behavior seems almost entirely the same, but I'm not sure what causes the mid-stream problems. I'm not sure if my vigorous phone twisting during the beginning of the call is exercising a different "cause" into the same issue, but the results seem the same.

If I leave the phone still when this happens, it sometimes recovers itself after ~10 seconds, but not always. It seems to resolve itself pretty reliably with my twisting-while-connecting case. In my randomly-mid-call experience, it seems to resolve itself less reliably but hard to say since I have less occurrences/data on those cases. In all cases, if we use our "mute video" button (which unpublishes the local participants video track), wait a few seconds, and re-enable the video, it seems to almost always fix the issue, though we may see a few frames of the striped video when we first re-enable the video again.

The WebRTC bug mentioned by @paynerc seems closely related but I'm getting slightly different behavior than what they have described. For one, theirs seems to be consistent and permanent with certain resolutions, where as mine happens seemingly randomly, and often times fixes itself. I'm not sure how Twilio picks a video resolution, and I don't see anywhere where I can request certain resolutions - according to that issue, if we could force some multiple of 16, we may have a better workaround. I do wonder if maybe my rapid twisting case is happening while Twilio is trying to pick a bitrate/resolution based on network conditions or something and this just-so-happens to cause it to pick something that isn't a multiple of 16 etc. I also wonder if maybe network "quality" changes are causing Twilio to pick a different resolution or something mid-call and that sometimes this causes the issue... and maybe the network getting better is what sometimes fixes it?

I was hoping some of the simpler workarounds in this thread would help me but I got pretty different behavior with each solution than the people posting it. I've been pretty brutal on trying to reproduce this, and have a good way to reproduce the issue, so I've got a good chunk of data on this but I'm not convinced there isn't some reason why people get different behavior.

First, I tried using CameraCapturer over Camera2Capturer. Initially this looked like a solution to me, but I eventually did reproduce the issue with this change. It seemed to make it harder to reproduce, but it didn't fix it entirely for me. I would be a little cautious with using this solution as it "seemed" like a fix to me before eventually breaking.

There are also solutions about disabling the HW encoder. Our application actually prefers H264, so I initially tried just disabling the H264 hardware encoder, but what's weird is that this seemed to break H264 decoding entirely. This meant that if some other device was in a call with a Pixel 3/4, the Pixel would report it couldn't find a codec and would be unable to decode the other devices video and could never see them. I'm not sure if this is a bug with the devices, a bug with Twilio's MediaCodecVideoEncoder implementation, or a limitation of H264 hardware encoders.

After switching these devices to prefer VP8 AND disable the VP8 hardware encoder, everything seems fine. Disabling the VP8 encoder doesn't seem to break its decoder, but I can't tell if it ends up forcing it to decode in software instead of hardware or something. This is workable, but it is far from ideal as we have to both switch codecs AND disable the HW encoder. Like others have mentioned, there does seem to be a quality drop in doing this.

Other people have reported this hardware encoder change working on some devices and not others, but it seems to have consistently fixed my issue on a Pixel 3, Pixel 4 XL, and a Pixel 4a. I was working on this fix right around the time Android 11 dropped, so my data is on Android 11. I don't know if it's related at all, but around the time Android 11 dropped, we seemed to be getting this issue more frequently, but that's entirely anecdotal and could just be coincidence. I believe we've seen the same issue on 10, and others have reported the issue on 10, but I don't know if all of my findings apply there as well or not.

TL;DR: I was pretty easily able to reproduce the issue by twisting my phone vigorously and spinning the camera around at the very beginning of calls. Using VP8 and disabling the hardware encoder seems to fix this issue, but this doesn't work if you want to use H264 and it reduces quality.

[aaalaniz]

Hey everyone!

We recently released Video Android 6.0.0 which includes a number of betterments and API updates. However, this issue still occurs on Pixel 3 devices when the hardware encoder attempts to encode a frame with a resolution where either the width or the height is not divisible by 16.

We have reached out to the WebRTC team about the status of this issue as they claim the issue was fixed in the Android media stack in RQ1A.200917.001. However, in our testing, we were able to reproduce the same issue with a Pixel 3 running RQ1A.201205.003.

Updated Workaround

As part of investigating the impact of this issue we discovered a need to provide an updated workaround. The previous recommended workaround MediaCodecVideoEncoder.disableVp8HwCodec(); is not available in Video Android 6.0. Reference the following snippet to work around this issue in your application.

videoTrack?.videoSource?.setVideoProcessor(object : VideoProcessor {
    var videoSink: VideoSink?
    override fun onCapturerStarted(success: Boolean) {}
    override fun onCapturerStopped() {}
    override fun onFrameCaptured(
        frame: VideoFrame?,
        parameters: VideoProcessor.FrameAdaptationParameters?
    ) {
        parameters?.let {
            if (!it.drop) {
                var scaledWidth = it.scaleWidth
                var scaledHeight = it.scaleHeight
                if (frame?.rotation == 270 || frame?.rotation == 90) {
                    scaledHeight = scaledHeight / 16 * 16
                } else {
                    scaledWidth = scaledWidth / 16 * 16
                }
                val adaptedBuffer = frame?.buffer?.cropAndScale(it.cropX, it.cropY,
                    it.cropWidth, it.cropHeight,
                    scaledWidth, scaledHeight)
                adaptedBuffer?.run {
                    onFrameCaptured(VideoFrame(this, frame.rotation, it.timestampNs))
                }
                frame?.release()
            }
        }
    }
    override fun onFrameCaptured(frame: VideoFrame?) {
        videoSink?.onFrame(frame)
    }
    override fun setSink(sink: VideoSink?) {
        videoSink = sink
    }
})

This snippet will update the scaled width or height to a number divisible by 16 and avoid the issue with the hardware encoder.

If this workaround does not suit your needs then let us know.

Thank you!

[VanKhulup]

@aaalaniz We managed to test it for some time on Pixel 4, seems like this works for our case. We gonna do more regression testing to be 100% sure, but initially looks good. Only one question - is this necessary to requireNotNull(sink) ? During clearing related viewmodels I'm calling release() on video track, which leads to videoSource.dispose(), and, consequently, setting the videoSink to null. And this code crashes then. Would this affect much to have property videoSink nullable instead of lateInit nonnull?

[aaalaniz]

Hey @VanKhulup

Good catch. Marking the videoSink as nullable is fine. I have updated the snippet accordingly.

Thanks!

[VanKhulup]

Hey @aaalaniz ! I've found a case where this solution is crashing the app. It's happening when a device is using legacy CameraCapturer. CameraCapturer is producing frames with NV21 buffer. When calling cropAndScale for this buffer, the refCount is not incrementing, and so the frame(and it's underlying buffer) is first "released" within the video processor onFrameCaptured method, and right after that the release() is called on this same frame from within Camera1Session.listenForBytebufferPreview method, and since it's refCount is already equals 0, the app is crashing.

When the Camera2Capturer is used, the underlying buffer for frames is TextureBuffer, and it's cropAndScale method is implemented in a way that the method is actually calling retain() before returning new buffer. And so the double release() call actually works properly.

Maybe it's a bug in the implementation of the NV21 buffer?

[kjanderson2]

For the issue that is being seen on Pixel 3 and Pixel 3a with distortion, has anybody gotten a comprehensive (or slightly MORE comprehensive) list of devices that this issue is affecting. We need to fix it asap in our code and can not currently update to a major fix version for Twilio. So far we have seen this issue on:

• Pixel 3
• Pixel 3a
• Pixel 3xl

Has anybody seen it on any additional devices that are non-pixels. Samsungs? OnePlus? Any others?

Our current fix is to call MediaCodecVideoEncoder.disableVp8HwCodec(); only on devices that we have confirmed to show the issue.

Note: we are not using Camera2Capturer and thus to dot see the separate Pixel 4/4xl issue.

UPDATE: We tested with a device farm and only were able to find the issue on Pixel 3 family devices, so this is the full extent of our workaround (hopefully it's helpful to somebody else):
if (Build.MODEL.contains("Pixel 3")) { MediaCodecVideoEncoder.disableVp8HwCodec() }

[kjanderson2]

Updated Workaround

As part of investigating the impact of this issue we discovered a need to provide an updated workaround. The previous recommended workaround MediaCodecVideoEncoder.disableVp8HwCodec(); is not available in Video Android 6.0. Reference the following snippet to work around this issue in your application.

@aaalaniz
Hi there,
We did end up utilizing this workaround in our code once we upgraded our Twilio Video version. We are now on Twilio Video version 6.4.0. However, we are now experiencing a crash after a few minutes of this fix running. We haven't isolated the exact log that is telling us what is going on but there are two potentially helpful logs

2022-04-07 17:02:37.388 12076-13225/pro.streem.now E/tvi.webrtc.Logging: HardwareVideoEncoder: Dropped frame, encoder queue full
and
2022-04-07 15:57:18.218 27335-27694/pro.streem.now I/OpenGLRenderer: Davey! duration=1065ms; Flags=0, IntendedVsync=11379235528535, Vsync=11380068861835, OldestInputEvent=9223372036854775807, NewestInputEvent=0, HandleInputStart=11380075409340, AnimationStart=11380075411215, PerformTraversalsStart=11380295901237, DrawStart=11380296890144, SyncQueued=11380296993321, SyncStart=11380308194520, IssueDrawCommandsStart=11380308532957, SwapBuffers=11380310947489, FrameCompleted=11380312437072, DequeueBufferDuration=780053, QueueBufferDuration=328021, GpuCompleted=11373342856118,

Has anybody else experienced crashes running the newly proposed workaround after a few minutes? Is there potentially another workaround we can try?

[broccolibird]

In case anyone runs into the same issue, we had been using the workaround noted in this comment: #470 (comment) and saw that it caused a memory leak in our application (as noted in the response by @kjanderson2 ).

To fix this, we added a call to release the new frame created with the adapted buffer:

adaptedBuffer?.run {
    VideoFrame(this, frame.rotation, it.timestampNs).let { newFrame ->
        onFrameCaptured(newFrame)
        newFrame.release()
    }
 }

I hope that helps someone else!