-
Notifications
You must be signed in to change notification settings - Fork 94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add background blur feature for supported devices #2812
Open
toger5
wants to merge
21
commits into
livekit
Choose a base branch
from
toger5/track-processor-blur
base: livekit
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+364
−51
Open
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
c798c79
Add background Blur
toger5 140fe48
make compatible with unsopported browsers
toger5 70ff435
add types
toger5 d23beef
remove unused dependencies.
toger5 95bc7db
update blur checkbox (disabled state)
toger5 0be312a
i18n
toger5 aade8f9
eslint + prettier
toger5 ae71c6e
add @types/dom-mediacapture-transform for the remaining type errors
toger5 a6745ef
properly add @livekit/track-processors
toger5 c2b6518
dont update if processor already set
toger5 7d8e9a7
fixes and gpu based blurring
toger5 9eae919
make default false
toger5 d179db1
more detailed error message
toger5 95c3ca8
review
toger5 c6d4844
review
toger5 574c895
disable blur in local track again
toger5 b77c4af
refactor
toger5 59bc73c
Use local assets for blurring
hughns 1860646
Lint
hughns aa5cf2f
Wording change
hughns b6cc9c5
Lint
hughns File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
/* | ||
Copyright 2024 New Vector Ltd. | ||
|
||
SPDX-License-Identifier: AGPL-3.0-only | ||
Please see LICENSE in the repository root for full details. | ||
*/ | ||
|
||
import { | ||
BackgroundTransformer, | ||
VideoTransformer, | ||
VideoTransformerInitOptions, | ||
} from "@livekit/track-processors"; | ||
import { ImageSegmenter } from "@mediapipe/tasks-vision"; | ||
|
||
interface WasmFileset { | ||
/** The path to the Wasm loader script. */ | ||
wasmLoaderPath: string; | ||
/** The path to the Wasm binary. */ | ||
wasmBinaryPath: string; | ||
} | ||
|
||
// n.b. this only includes the SIMD versions of the WASM files which have good support: | ||
// https://caniuse.com/?search=simd | ||
const wasmFileset: WasmFileset = { | ||
wasmLoaderPath: new URL( | ||
"../../node_modules/@mediapipe/tasks-vision/wasm/vision_wasm_internal.js", | ||
import.meta.url, | ||
).href, | ||
wasmBinaryPath: new URL( | ||
"../../node_modules/@mediapipe/tasks-vision/wasm/vision_wasm_internal.wasm", | ||
import.meta.url, | ||
).href, | ||
}; | ||
|
||
const modelAssetPath = new URL( | ||
"../mediapipe/imageSegmenter/selfie_segmenter.tflite", | ||
import.meta.url, | ||
).href; | ||
|
||
export class BlurBackgroundTransformer extends BackgroundTransformer { | ||
public async init({ | ||
outputCanvas, | ||
inputElement: inputVideo, | ||
}: VideoTransformerInitOptions): Promise<void> { | ||
// call super.super.init() | ||
await VideoTransformer.prototype.init.call(this, { | ||
outputCanvas, | ||
inputElement: inputVideo, | ||
}); | ||
|
||
this.imageSegmenter = await ImageSegmenter.createFromOptions(wasmFileset, { | ||
baseOptions: { | ||
modelAssetPath, | ||
delegate: "GPU", | ||
...this.options.segmenterOptions, | ||
}, | ||
runningMode: "VIDEO", | ||
outputCategoryMask: true, | ||
outputConfidenceMasks: false, | ||
}); | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,113 @@ | ||
/* | ||
Copyright 2024 New Vector Ltd. | ||
|
||
SPDX-License-Identifier: AGPL-3.0-only | ||
Please see LICENSE in the repository root for full details. | ||
*/ | ||
|
||
import { BackgroundOptions, ProcessorWrapper } from "@livekit/track-processors"; | ||
import { | ||
createContext, | ||
FC, | ||
useCallback, | ||
useContext, | ||
useEffect, | ||
useRef, | ||
useState, | ||
} from "react"; | ||
import { logger } from "matrix-js-sdk/src/logger"; | ||
import { LocalVideoTrack } from "livekit-client"; | ||
|
||
import { | ||
backgroundBlur as backgroundBlurSettings, | ||
useSetting, | ||
} from "../settings/settings"; | ||
import { BlurBackgroundTransformer } from "./BlurBackgroundTransformer"; | ||
|
||
type ProcessorState = { | ||
supported: boolean | undefined; | ||
processor: undefined | ProcessorWrapper<BackgroundOptions>; | ||
/** | ||
* Call this method to try to initialize a processor. | ||
* This only needs to happen if supported is undefined. | ||
* If the backgroundBlur setting is set to true this does not need to be called | ||
* and the processorState.supported will update automatically to the correct value. | ||
*/ | ||
checkSupported: () => void; | ||
}; | ||
const ProcessorContext = createContext<ProcessorState | undefined>(undefined); | ||
|
||
export const useTrackProcessor = (): ProcessorState | undefined => | ||
useContext(ProcessorContext); | ||
|
||
export const useTrackProcessorSync = ( | ||
videoTrack: LocalVideoTrack | null, | ||
): void => { | ||
const { processor } = useTrackProcessor() || {}; | ||
useEffect(() => { | ||
if (processor && !videoTrack?.getProcessor()) { | ||
void videoTrack?.setProcessor(processor); | ||
} | ||
if (!processor && videoTrack?.getProcessor()) { | ||
void videoTrack?.stopProcessor(); | ||
} | ||
}, [processor, videoTrack]); | ||
}; | ||
|
||
interface Props { | ||
children: JSX.Element; | ||
} | ||
export const ProcessorProvider: FC<Props> = ({ children }) => { | ||
// The setting the user wants to have | ||
const [blurActivated] = useSetting(backgroundBlurSettings); | ||
|
||
// If `ProcessorState.supported` is undefined the user can activate that we want | ||
// to have it at least checked (this is useful to show the settings menu properly) | ||
// We dont want to try initializing the blur if the user is not even looking at the setting | ||
const [shouldCheckSupport, setShouldCheckSupport] = useState(blurActivated); | ||
|
||
// Cache the processor so we only need to initialize it once. | ||
const blur = useRef<ProcessorWrapper<BackgroundOptions> | undefined>( | ||
undefined, | ||
); | ||
|
||
const checkSupported = useCallback(() => { | ||
setShouldCheckSupport(true); | ||
}, []); | ||
// This is the actual state exposed through the context | ||
const [processorState, setProcessorState] = useState<ProcessorState>(() => ({ | ||
supported: false, | ||
processor: undefined, | ||
checkSupported, | ||
})); | ||
|
||
useEffect(() => { | ||
if (!shouldCheckSupport) return; | ||
try { | ||
if (!blur.current) { | ||
blur.current = new ProcessorWrapper( | ||
new BlurBackgroundTransformer({ blurRadius: 15 }), | ||
"background-blur", | ||
); | ||
} | ||
setProcessorState({ | ||
checkSupported, | ||
supported: true, | ||
processor: blurActivated ? blur.current : undefined, | ||
}); | ||
} catch (e) { | ||
setProcessorState({ | ||
checkSupported, | ||
supported: false, | ||
processor: undefined, | ||
}); | ||
logger.error("disable background blur", e); | ||
} | ||
}, [blurActivated, checkSupported, shouldCheckSupport]); | ||
|
||
return ( | ||
<ProcessorContext.Provider value={processorState}> | ||
{children} | ||
</ProcessorContext.Provider> | ||
); | ||
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# Google AI Edge MediaPipe Selfie Segmentation | ||
|
||
- See: https://ai.google.dev/edge/mediapipe/solutions/vision/image_segmenter | ||
- Latest: https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter/float16/latest/selfie_segmenter.tflite | ||
- License: Apache 2.0 as per https://storage.googleapis.com/mediapipe-assets/Model%20Card%20MediaPipe%20Selfie%20Segmentation.pdf |
Binary file not shown.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be better to use
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did try that, but I ended up with a vite error and so did this approach instead. 🤷