feat: Skia for V4! 🥳

[mrousavy]

What

Skia integration for V4! 🥳

TL;DR: Allows you to "draw stuff onto the Camera Frame" in realtime.
This is different to just adding a view ontop of the <Camera>, because this actually draws inside the Frame buffer, meaning this can be extended to also be applied to captured photos and videos in the future. For now it's preview only.

For example, you can draw color filters, bounding boxes or circles for faces/qr-codes, face or license plate blurring, color corrections, animations on the frame, etc etc

Backstory

After almost a year since the first idea and prototype of integrating Skia into VisionCamera, we can finally see it happen.
The main issues back then (July/August 2023) were that the react-native-skia dependency in VisionCamera's codebase was too harsh. It had a strong native dependency, we needed to add a bunch of native views including Metal/OpenGL Surface Views for preview, Choreographers for syncing, setting up Metal and OpenGL contexts, adding a native dependency and header include paths on libskia and all of the other skia companion libs, and we had a bunch of complicated nested branches in the Camera codebase where we switched logic between a normal Frame Processor and a Skia Frame Processor.
All of that is a thing from the past now, since I implemented a way to only expose a handle to the Frame ("platform buffer") to JS, and thanks to @wcandillon's amazing help (thank you so much for your time!) we got a MakeImageFromPlatformBuffer(..) function into react-native-skia.
Now we can create an SkImage from a Frame, straight from JS.

API

In addition to useFrameProcessor ("readonly" Frame Processors) a new API has been introduced: useSkiaFrameProcessor.

In contrast to the useFrameProcessor hook, with the useSkiaFrameProcessor hook you now get DrawableFrames instead of normal Frames.

The DrawableFrame has a render(..) function to render itself (otherwise nothing gets rendered), and all other functions from Frame and SkCanvas.

For example, to render the image with a blur filter:

const frameProcessor = useSkiaFrameProcessor((frame) => {
  'worklet'

  // 1. create blur filter
  const blurRadius = 10
  const blurFilter = Skia.ImageFilter.MakeBlur(blurRadius, blurRadius, TileMode.Repeat, null)

  // 2. wrap blur filter in Paint
  const paint = Skia.Paint()
  paint.setImageFilter(blurFilter)

  // 3. render image with the blur filter paint
  frame.render(paint)
}, [])

Or to draw a red rectangle on the middle of the Frame:

const frameProcessor = useSkiaFrameProcessor((frame) => {
  'worklet'

  // 1. render frame normally
  frame.render() 

  // 2. create a red paint
  const redPaint = Skia.Paint()
  const red = Skia.Color('red')
  redPaint.setColor(red)

  // 3. draw a rectangle in the center of the screen
  const rect = Skia.XYWHRect(frame.width / 2, frame.height / 2, 150, 150)
  frame.drawRect(rect, redPaint)
}, [])

On Android, make sure to set the Camera's pixelFormat={...} to "rgb". On iOS, you can use "yuv" or "rgb", where "yuv" is typically more efficient.

Note: Skia Frame Processors require react-native-reanimated (>= 3.x.x) and @shopify/react-native-skia (>= 1.2.0)

Required PRs

1. ✅ The MakeImageFromPlatformBuffer(...) PR in react-native-skia
2. ⏳ The SkData -> Metal Texture change PR in react-native-skia (otherwise buffers are in wrong pixelformat and look blue)
3. ⏳ The YUV CVPixelBuffers PR in react-native-skia (otherwise we can only use pixelFormat="rgb", no pixelFormat="yuv"/default)

Example

Example shader running directly on the Camera Frame at 60 FPS:

RPReplay_Final1712748323.mov

Changes

• Adds a new API: useSkiaFrameProcessor (or createSkiaFrameProcessor)
• Marks the Java FrameProcessorPlugin's callback function throwable (throws Throwable), you can now throw exceptions in Java Frame Processors
• Adds getPlatformBuffer() to Frame
• Adds a new prop to the <Camera>: preview={boolean}. This allows you to manually disable the Preview if not needed. (e.g. when using Skia it will use a Skia Canvas instead of the native preview output)
• Fixes an issue where the camera outputs/cameraId are being accessed before they are initialized (device/no-device or session/no-outputs error) by early aborting duplicate configure { ... } calls.
• Minor improvements around the native Frame object synchronizations
• Fixes an issue where the Frame Processor pipeline would just stall after a second on Android (forgot to close() the parent ImageProxy, whoops)
• Fixes an issue where Frame.orientation was wrong on iOS (it is always portrait since we rotate buffers accordingly in the pipeline)
• Removes the "native" pixel format, native does not exist in a Frame Processor - it's always RGB or YUV
• Removes the supportedPixelFormats prop on a Format, both iOS and Android always support YUV and RGB, you are free to choose.
• Throws more precise error messages when a YUV pixel format (HDR or compression) cannot be found on iOS, and automatically fallback to the best format to avoid blackscreens
• Adds a jsi::Scope to Frame Processor calls to maybe improve GC calls
• Adds lazy proxy dependencies on react-native-worklets-core, react-native-skia and react-native-reanimated

Tested on

• iPhone 15 Pro (yuv + rgb)
• iPhone 8 (yuv + rgb)
• Huawei P10 (rgb)
• Samsung S22 (rgb)

Related issues

• Resolves Custom Color Filters and Controls like Exposure, Contrast etc. 💭  #2734
• Resolves Live Filter Preview✨  #2347
• Resolves ❓ Applying color filters to the camera  #1755
• Resolves ❓ Apply filter to photos #724
• Resolves What is the recommended way to draw landmarks over the video❓  #2274
• Resolves Is it possible to do LIVE TEXT with react-native-vision-camera❓  #1215

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
react-native-vision-camera	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 17, 2024 1:32pm

[wcandillon]

this is cool 😁

[mrousavy]

released in latest V4 beta :)

everyone can try useSkiaFrameProcessor now in their apps

[efstathiosntonas]

@mrousavy I installed beta 15 and it requires Skia to be installed. Can you please consider on making it optional if possible? I guess not everyone is planning on using Skia processors and bloating the app with 6MB is not optimal.

edit: by looking at the changes it seems impossible to avoid it

[mrousavy]

Skia is optional:

react-native-vision-camera/package/package.json

Lines 120 to 122 in b751f2d

	"@shopify/react-native-skia": {
	"optional": true
	}

Are you seeing an error or anything?

[mrousavy]

I have react-native-skia in devDependencies:

react-native-vision-camera/package/package.json

Line 92 in b751f2d

"@shopify/react-native-skia": "^1.2.1",

Maybe this is why it gets included in your app as well then? I just need the types, that's why I have it in devDependencies...

I don't think this ends up in your release app though, as the devDependencies are compiled out. Not sure on this.

[efstathiosntonas]

after installation I get red screen that skia is missing

[efstathiosntonas]

more info:

Click me

[mrousavy]

@efstathiosntonas fixed in #2750 👍

[geekabhinavp]

Hey @mrousavy , I was also getting the skia missing error, i installed it manually in the package. Now that error is gone but I am getting stuck on app loader itself on launch, in the logs i am just getting:

iOS Bundling complete 5644ms
[appname] VisionCameraProxy: Creating Worklet Context...
[appname] VisionCameraProxy: Worklet Context Created!

[mrousavy]

Are there any native logs?

[geekabhinavp]

These are the complete logs:

[UIKitCore] Class CKBrowserSwitcherViewController overrides the -traitCollection getter,
which is not supported. If you're trying to override traits, you must use the
appropriate API.
iOS ./index.js ░░░░░░░░░░░░░░░░  0.1% ( 2/57)[CoreFoundation] AddInstanceForFactory: No factory registered for id <CFUUID
0x6000006c06a0> F8BB1C28-BAE8-11D6-9C31-00039315CD46
iOS Bundling complete 5644ms
[appname] VisionCameraProxy: Creating Worklet Context...
[appname] VisionCameraProxy: Worklet Context Created!