Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Video goes black screen after selecting blur background #2038

Open
4 tasks done
dathacky opened this issue Feb 18, 2022 · 8 comments
Open
4 tasks done

Video goes black screen after selecting blur background #2038

dathacky opened this issue Feb 18, 2022 · 8 comments
Labels
feature-request New feature or request Triaged video-ml Machine learning features on video stream like background blur

Comments

@dathacky
Copy link

dathacky commented Feb 18, 2022 •
edited
Loading

What happened and what did you expect to happen?

After selecting blur background, I noticed the video went black before rendering it made our customer unhappy, is there any way to fix this?

Meeting-Demo.mp4

Have you reviewed our existing documentation?

Reproduction steps

  1. Join chime demo
  2. Choose Blur Backgroud 10%
  3. Choose Blur Background 40% -> black screen

Amazon Chime SDK for JavaScript version

2.27.0

What browsers are you seeing the problem on?

Chrome

Browser version

Version 98.0.4758.102

Meeting and Attendee ID Information.

No response

Browser console logs


VM76:2 2022-02-18T02:21:12.886Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getRTCPeerConnectionStats null
VM76:2 2022-02-18T02:21:13.887Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getRTCPeerConnectionStats null
VM76:2 2022-02-18T02:21:14.298Z [INFO] SDK - local clock skew estimate=20ms from ping-pong time=101ms
VM76:2 2022-02-18T02:21:14.887Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getRTCPeerConnectionStats null
VM76:2 [DEMO] clicking video filter Background Blur 10% CPU
VM76:2 [DEMO] Switching to: a143b3800185a195bde09066998bb1524184cbf716640bad6b029039744e508c
VM76:2 [DEMO] Choosing video filter Background Blur 10% CPU
VM76:2 2022-02-18T02:21:15.302Z [INFO] BackgroundBlurProcessor - processor is  supported
VM76:2 2022-02-18T02:21:15.304Z [INFO] BackgroundBlurProcessor - Using browser-provided background blur
VM76:2 2022-02-18T02:21:15.305Z [INFO] BackgroundBlurProcessor - blur strength set to 15
VM76:2 2022-02-18T02:21:15.305Z [INFO] BackgroundBlurProcessor - background blur amount set to 0
VM76:2 2022-02-18T02:21:15.305Z [INFO] BackgroundBlurProcessor - BackgroundBlur processor successfully created
VM76:2 2022-02-18T02:21:15.305Z [INFO] BackgroundBlurProcessor - BackgroundBlur spec: {
  "paths": {
    "worker": "https://static.sdkassets.chime.aws/bgblur/workers/worker.js?assetGroup=sdk-2.26&sdk=2.26.0&ua=chrome-98",
    "wasm": "https://static.sdkassets.chime.aws/bgblur/wasm/_cwt-wasm.wasm?assetGroup=sdk-2.26&sdk=2.26.0&ua=chrome-98",
    "simd": "https://static.sdkassets.chime.aws/bgblur/wasm/_cwt-wasm-simd.wasm?assetGroup=sdk-2.26&sdk=2.26.0&ua=chrome-98"
  },
  "model": {
    "path": "https://static.sdkassets.chime.aws/bgblur/models/selfie_segmentation_landscape.tflite?assetGroup=sdk-2.26&sdk=2.26.0&ua=chrome-98",
    "input": {
      "height": 144,
      "width": 256,
      "range": [
        0,
        1
      ],
      "channels": 3
    },
    "output": {
      "height": 144,
      "width": 256,
      "range": [
        0,
        1
      ],
      "channels": 1
    }
  },
  "assetGroup": "sdk-2.26"
}
VM76:2 2022-02-18T02:21:15.305Z [INFO] BackgroundBlurProcessor - BackgroundBlur options: {
  "filterCPUUtilization": 10,
  "blurStrength": 15,
  "logger": {
    "name": "BackgroundBlurProcessor",
    "level": 1
  },
  "reportingPeriodMillis": 1000
}
VM76:2 2022-02-18T02:21:15.305Z [INFO] BackgroundBlurProcessor - start initializing the processor
VM76:2 2022-02-18T02:21:15.889Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getRTCPeerConnectionStats null
VM76:2 2022-02-18T02:21:16.428Z [INFO] BackgroundBlurProcessor - received initialize message: {
  "msg": "initialize",
  "payload": true
}
VM76:2 2022-02-18T02:21:16.429Z [INFO] BackgroundBlurProcessor - successfully initialized the background blur worker
2adbbb16-2e74-4065-ba31-f5e824a20e19:4 INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
printChar @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
b @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
$func1355 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0x2050b7
$func640 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0xa95fb
$func713 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0xb4f66
$func405 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0x4ba07
$func1190 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0x1af170
$func1231 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0x1b90b9
(anonymous) @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
N @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
(anonymous) @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
(anonymous) @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
r.onload @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
load (async)
u @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
(anonymous) @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
o @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
$v @ _cwt-wasm-simd-9d918e502f01731a.wasm:0x1962be
i._cwt_model_load_model @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
loadModel @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
(anonymous) @ 2adbbb16-2e74-4065-ba31-f5e824a20e19:4
2adbbb16-2e74-4065-ba31-f5e824a20e19:4 INFO: adding 0 filters
2adbbb16-2e74-4065-ba31-f5e824a20e19:4 INFO: model successfully loaded
VM76:2 2022-02-18T02:21:16.469Z [INFO] BackgroundBlurProcessor - received load model message: {
  "msg": "loadModel",
  "payload": 2
}
VM76:2 2022-02-18T02:21:16.469Z [INFO] BackgroundBlurProcessor - successfully loaded background blur worker segmentation model
VM76:2 2022-02-18T02:21:16.469Z [INFO] BackgroundBlurProcessor - successfully initialized the background blur processor
VM76:2 2022-02-18T02:21:16.470Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/chooseVideoInputDevice with transform device
VM76:2 2022-02-18T02:21:16.470Z [INFO] SDK - Choosing video transform device [object Object]
VM76:2 2022-02-18T02:21:16.470Z [INFO] SDK - video transform device uses previous stream
VM76:2 2022-02-18T02:21:16.472Z [INFO] SDK - processing pipeline input stream settings {"aspectRatio":1.7777777777777777,"deviceId":"a143b3800185a195bde09066998bb1524184cbf716640bad6b029039744e508c","frameRate":15,"groupId":"0e0ae47ddb218df004d4c42b4ffdd7157704fc2cd8e56318aa0d0ed085efc21e","height":540,"resizeMode":"crop-and-scale","width":960}
VM76:2 2022-02-18T02:21:16.509Z [INFO] BackgroundBlurProcessor - frame counter setting frame rate to 15
VM76:2 2022-02-18T02:21:16.509Z [INFO] BackgroundBlurProcessor - background blur source width: 960
VM76:2 2022-02-18T02:21:16.509Z [INFO] BackgroundBlurProcessor - background blur source height: 540
VM76:2 2022-02-18T02:21:16.509Z [INFO] BackgroundBlurProcessor - background blur amount set to 15
VM76:2 2022-02-18T02:21:16.537Z [INFO] SDK - successfully replaced video track
VM76:2 video tile updated: {
  "tileId": 1,
  "localTile": true,
  "localTileStarted": false,
  "isContent": false,
  "active": true,
  "paused": false,
  "poorConnection": false,
  "boundAttendeeId": "f5832c8b-1b17-eb24-d9a6-8aaa99173036",
  "boundExternalUserId": "9d3dac59#dat",
  "boundVideoStream": {},
  "boundVideoElement": {},
  "nameplate": null,
  "videoStreamContentWidth": 960,
  "videoStreamContentHeight": 540,
  "videoElementCSSWidthPixels": 1050,
  "videoElementCSSHeightPixels": 591,
  "devicePixelRatio": 2,
  "videoElementPhysicalWidthPixels": 2100,
  "videoElementPhysicalHeightPixels": 1182,
  "streamId": null
}
VM76:2 2022-02-18T02:21:16.539Z [INFO] SDK - binding video tile 1 to video-27
VM76:2 2022-02-18T02:21:16.539Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/bindVideoElement {"tileId":1,"videoElementId":"video-27"}
VM76:2 setting nameplate to dat
VM76:2 setting attendeeId to f5832c8b-1b17-eb24-d9a6-8aaa99173036
VM76:2 2022-02-18T02:21:16.539Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/hasStartedLocalVideoTile null -> true
VM76:2 2022-02-18T02:21:16.539Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/hasStartedLocalVideoTile null -> true
VM76:2 2022-02-18T02:21:16.539Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getLocalVideoTile
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/hasStartedLocalVideoTile null -> true
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getLocalVideoTile
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/hasStartedLocalVideoTile null -> true
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/hasStartedLocalVideoTile null -> true
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getLocalVideoTile
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:16.540Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:16.653Z [INFO] SDK - video transform device processing started
VM76:2 2022-02-18T02:21:16.886Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getRTCPeerConnectionStats null
VM76:2 [DEMO] background filter CPU utilization high: 39%
VM76:2 2022-02-18T02:21:17.886Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getRTCPeerConnectionStats null
VM76:2 [DEMO] background filter CPU utilization high: 37%
VM76:2 [DEMO] clicking video filter Background Blur 30% CPU
VM76:2 [DEMO] Switching to: a143b3800185a195bde09066998bb1524184cbf716640bad6b029039744e508c
VM76:2 2022-02-18T02:21:18.619Z [INFO] BackgroundBlurProcessor - background blur frame process destroyed
VM76:2 [DEMO] Choosing video filter Background Blur 30% CPU
VM76:2 2022-02-18T02:21:18.626Z [INFO] SDK - video transform device processing stopped
VM76:2 2022-02-18T02:21:18.643Z [INFO] BackgroundBlurProcessor - processor is  supported
VM76:2 2022-02-18T02:21:18.643Z [INFO] BackgroundBlurProcessor - Using browser-provided background blur
VM76:2 2022-02-18T02:21:18.643Z [INFO] BackgroundBlurProcessor - blur strength set to 15
VM76:2 2022-02-18T02:21:18.643Z [INFO] BackgroundBlurProcessor - background blur amount set to 0
VM76:2 2022-02-18T02:21:18.643Z [INFO] BackgroundBlurProcessor - BackgroundBlur processor successfully created
VM76:2 2022-02-18T02:21:18.643Z [INFO] BackgroundBlurProcessor - BackgroundBlur spec: {
  "paths": {
    "worker": "https://static.sdkassets.chime.aws/bgblur/workers/worker.js?assetGroup=sdk-2.26&sdk=2.26.0&ua=chrome-98",
    "wasm": "https://static.sdkassets.chime.aws/bgblur/wasm/_cwt-wasm.wasm?assetGroup=sdk-2.26&sdk=2.26.0&ua=chrome-98",
    "simd": "https://static.sdkassets.chime.aws/bgblur/wasm/_cwt-wasm-simd.wasm?assetGroup=sdk-2.26&sdk=2.26.0&ua=chrome-98"
  },
  "model": {
    "path": "https://static.sdkassets.chime.aws/bgblur/models/selfie_segmentation_landscape.tflite?assetGroup=sdk-2.26&sdk=2.26.0&ua=chrome-98",
    "input": {
      "height": 144,
      "width": 256,
      "range": [
        0,
        1
      ],
      "channels": 3
    },
    "output": {
      "height": 144,
      "width": 256,
      "range": [
        0,
        1
      ],
      "channels": 1
    }
  },
  "assetGroup": "sdk-2.26"
}
VM76:2 2022-02-18T02:21:18.643Z [INFO] BackgroundBlurProcessor - BackgroundBlur options: {
  "filterCPUUtilization": 30,
  "blurStrength": 15,
  "logger": {
    "name": "BackgroundBlurProcessor",
    "level": 1
  },
  "reportingPeriodMillis": 1000
}
VM76:2 2022-02-18T02:21:18.643Z [INFO] BackgroundBlurProcessor - start initializing the processor
2adbbb16-2e74-4065-ba31-f5e824a20e19:4 unknown operation close
VM76:2 2022-02-18T02:21:18.887Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getRTCPeerConnectionStats null
VM76:2 2022-02-18T02:21:19.036Z [INFO] BackgroundBlurProcessor - received initialize message: {
  "msg": "initialize",
  "payload": true
}
VM76:2 2022-02-18T02:21:19.037Z [INFO] BackgroundBlurProcessor - successfully initialized the background blur worker
bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4 INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
printChar @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
b @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
$func1355 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0x2050b7
$func640 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0xa95fb
$func713 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0xb4f66
$func405 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0x4ba07
$func1190 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0x1af170
$func1231 @ _cwt-wasm-simd-9d918e502f01731a.wasm:0x1b90b9
(anonymous) @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
N @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
(anonymous) @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
(anonymous) @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
r.onload @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
load (async)
u @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
(anonymous) @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
o @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
$v @ _cwt-wasm-simd-9d918e502f01731a.wasm:0x1962be
i._cwt_model_load_model @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
loadModel @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
(anonymous) @ bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4
bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4 INFO: adding 0 filters
bed5c37f-7f7f-4615-8ae1-409eeb5cc493:4 INFO: model successfully loaded
VM76:2 2022-02-18T02:21:19.071Z [INFO] BackgroundBlurProcessor - received load model message: {
  "msg": "loadModel",
  "payload": 2
}
VM76:2 2022-02-18T02:21:19.071Z [INFO] BackgroundBlurProcessor - successfully loaded background blur worker segmentation model
VM76:2 2022-02-18T02:21:19.071Z [INFO] BackgroundBlurProcessor - successfully initialized the background blur processor
VM76:2 2022-02-18T02:21:19.072Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/chooseVideoInputDevice with transform device
VM76:2 2022-02-18T02:21:19.072Z [INFO] SDK - Choosing video transform device [object Object]
VM76:2 2022-02-18T02:21:19.072Z [INFO] SDK - Switched from previous VideoTransformDevice
VM76:2 2022-02-18T02:21:19.072Z [INFO] SDK - video transform device needs new intrinsic device
VM76:2 2022-02-18T02:21:19.072Z [INFO] SDK - DefaultVideoTransformDevice: detach stopping input media stream
VM76:2 2022-02-18T02:21:19.073Z [INFO] SDK - requesting new video device with constraint {"video":{"deviceId":{"exact":"a143b3800185a195bde09066998bb1524184cbf716640bad6b029039744e508c"},"width":{"ideal":960},"height":{"ideal":540},"frameRate":{"ideal":15}}}
VM76:2 2022-02-18T02:21:19.110Z [INFO] SDK - got video device for constraints {"video":{"deviceId":{"exact":"a143b3800185a195bde09066998bb1524184cbf716640bad6b029039744e508c"},"width":{"ideal":960},"height":{"ideal":540},"frameRate":{"ideal":15}}}
VM76:2 2022-02-18T02:21:19.110Z [INFO] SDK - apply processors to transform
VM76:2 2022-02-18T02:21:19.110Z [INFO] SDK - processing pipeline input stream settings {"aspectRatio":1.7777777777777777,"deviceId":"a143b3800185a195bde09066998bb1524184cbf716640bad6b029039744e508c","frameRate":15,"groupId":"0e0ae47ddb218df004d4c42b4ffdd7157704fc2cd8e56318aa0d0ed085efc21e","height":540,"resizeMode":"crop-and-scale","width":960}
VM76:2 2022-02-18T02:21:19.168Z [INFO] BackgroundBlurProcessor - frame counter setting frame rate to 15
VM76:2 2022-02-18T02:21:19.168Z [INFO] BackgroundBlurProcessor - background blur source width: 960
VM76:2 2022-02-18T02:21:19.168Z [INFO] BackgroundBlurProcessor - background blur source height: 540
VM76:2 2022-02-18T02:21:19.168Z [INFO] BackgroundBlurProcessor - background blur amount set to 15
VM76:2 2022-02-18T02:21:19.178Z [INFO] SDK - successfully replaced video track
VM76:2 video tile updated: {
  "tileId": 1,
  "localTile": true,
  "localTileStarted": false,
  "isContent": false,
  "active": true,
  "paused": false,
  "poorConnection": false,
  "boundAttendeeId": "f5832c8b-1b17-eb24-d9a6-8aaa99173036",
  "boundExternalUserId": "9d3dac59#dat",
  "boundVideoStream": {},
  "boundVideoElement": {},
  "nameplate": null,
  "videoStreamContentWidth": 960,
  "videoStreamContentHeight": 540,
  "videoElementCSSWidthPixels": 1050,
  "videoElementCSSHeightPixels": 591,
  "devicePixelRatio": 2,
  "videoElementPhysicalWidthPixels": 2100,
  "videoElementPhysicalHeightPixels": 1182,
  "streamId": null
}
VM76:2 2022-02-18T02:21:19.182Z [INFO] SDK - binding video tile 1 to video-27
VM76:2 2022-02-18T02:21:19.183Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/bindVideoElement {"tileId":1,"videoElementId":"video-27"}
VM76:2 setting nameplate to dat
VM76:2 setting attendeeId to f5832c8b-1b17-eb24-d9a6-8aaa99173036
VM76:2 2022-02-18T02:21:19.183Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/hasStartedLocalVideoTile null -> true
VM76:2 2022-02-18T02:21:19.184Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/hasStartedLocalVideoTile null -> true
VM76:2 2022-02-18T02:21:19.184Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getLocalVideoTile
VM76:2 2022-02-18T02:21:19.184Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:19.185Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:19.185Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/hasStartedLocalVideoTile null -> true
VM76:2 2022-02-18T02:21:19.185Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getLocalVideoTile
VM76:2 2022-02-18T02:21:19.186Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:19.186Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:19.187Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/hasStartedLocalVideoTile null -> true
VM76:2 2022-02-18T02:21:19.189Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/hasStartedLocalVideoTile null -> true
VM76:2 2022-02-18T02:21:19.189Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getLocalVideoTile
VM76:2 2022-02-18T02:21:19.190Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:19.190Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getAllVideoTiles
VM76:2 2022-02-18T02:21:19.214Z [INFO] SDK - video transform device processing started
VM76:2 2022-02-18T02:21:19.887Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getRTCPeerConnectionStats null
VM76:2 [DEMO] background filter CPU utilization high: 40%
VM76:2 2022-02-18T02:21:20.888Z [INFO] SDK - API/DefaultAudioVideoFacade/eeecc3b0-26a1-420d-839c-70958b872713/f5832c8b-1b17-eb24-d9a6-8aaa99173036/getRTCPeerConnectionStats null
VM76:2 [DEMO] background filter CPU utilization high: 41%
@xuesichao xuesichao added the video-ml Machine learning features on video stream like background blur label Feb 18, 2022
@xuesichao
Copy link
Contributor

Hi @dathacky. There are two types of device: Device or VideoTransformDevice. The original camera is of Device type. The VideoTransformDevice is a device that applies processing to another device. So background blur function is actually a VideoTransformDevice. It applies background blur processing to the original devices.

When you switch the background blur filter between different CPU utilization, the actual logic here is JS SDK is switching between different devices.

The reason you observe the black video tile is that the video tile is not removed during the switching, but the camera stream is stopped for a while.

@hekmatba
Copy link
Contributor

Hi, sorry for the inconvenience and thank you for reporting this. We will look into this and respond to this ticket. Thank you.

@dathacky
Copy link
Author

@xuesichao @hekmatba
Thank you, hope it gets fixed soon

@dathacky
Copy link
Author

dathacky commented Mar 4, 2022

@hekmatba
Hi, hope you are still interested in this issue

@dathacky
Copy link
Author

@hekmatba @ltrung
I hope this ticket is not forgotten

@michhyun1
Copy link
Contributor

@dathacky I think this is not a bug, but rather a side effect of selecting a different device. @xuesichao mentioned the behavior above, which is expected when switching between devices.

@mrnav90
Copy link

mrnav90 commented Mar 25, 2022

@michhyun1
So this is not a bug, can we create a fake track ( stream ) when device selecting?
Before, we're already used Jitsi Library and also happen that so my team create a fake track replace when device selecting..

@michhyun1
Copy link
Contributor

@mrnav90 what is the purpose of a fake stream? When you switch from one device to another device, we need to turn off one device and then select and turn on the next device. Currently, that includes switching between a regular video device and a VideoTransformDevice that is applying a video filter to a regular video device.

@ltrung ltrung added video-ml Machine learning features on video stream like background blur and removed video-ml Machine learning features on video stream like background blur labels Apr 13, 2022
@michhyun1 michhyun1 added the feature-request New feature or request label Jun 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature-request New feature or request Triaged video-ml Machine learning features on video stream like background blur
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@mrnav90 @xuesichao @dathacky @michhyun1 @linsang21 @ltrung @hekmatba