feat: Create Image.MakeImageFromPlatformBuffer(..)

[mrousavy]

Creates a new API: Skia.Image.MakeImageFromPlatformBuffer(..).

This API can create an SkImage instance from a "platform buffer".

A "platform buffer" is a platform-native GPU/CPU buffer instance:

• On iOS: This is a CMSampleBuffer.
• On Android: This is a AHardwareBuffer*

This is a preparation for VisionCamera Skia Frame Processors, where a VisionCamera Frame can be converted to an SkImage and later be used for drawing operations.

const surface = useSharedValue(null)
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'

  // 1. prepare Surface
  if (surface.value == null) {
    surface.value = Skia.Surface.MakeOffscreen(frame.width, frame.height)
  }
  const canvas = surface.value.getCanvas()

  // 2. convert Frame to SkImage
  const platformBuffer = frame.getPlatformBuffer()
  const image = Skia.Image.MakeFromPlatformBuffer(platformBuffer.pointer)

  // 3. draw it (and any other drawing operations)
  canvas.drawImage(image, 0, 0)

  // 4. flush and clean
  surface.value.flush()
  platformBuffer.delete()
}, [])

Demo of a VisionCamera Frame Processor drawing a blur shader directly onto the Camera Frame in realtime at 60 FPS:

RPReplay_Final1712748323.mov

[wcandillon]

to revert

[mrousavy]

reverted!

[wcandillon]

looks great 👍

[wcandillon]

let's run a little lint --fix here, sorry we don't have the same eslint rules

[wcandillon]

Should it be named MakeImageFromGPUBuffer or MakeImageFromBuffer?

[mrousavy]

• MakeImageFromBuffer no because a buffer is normally something like a uint8_t*. This is not a normal buffer, it is a special platform native buffer that is shared across GPU and CPU.
• MakeImageFromGPUBuffer, hmm, a bit better, but still not entirely accurate since CMSampleBuffer and AHardwareBuffer are GPU / CPU memory, shared access. That's their whole point.

[mrousavy]

Linted 👍

[wcandillon]

if we are not able to avoid the cpu memory copy on iOS for now and we want to ship this feature. This is where the CPU memory copy should happen (SkImages::MakeRaster)

[mrousavy]

Yea, for now I am doing this in JS and it works fine - on my iPhone 15 Pro it still runs at 120 FPS :)

So from my side we can ship. Maybe it is a bug in Skia or something we are doing wrong that makes the GPU Texture SkImage invalid, hence we need to do a non-texture-image copy explicitly. Fine for now, better to fix later

[github-actions]

🎉 This PR is included in version 1.2.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀