feat: replace photo library detection with human face detection

[ioannisj]

💡 Motivation and Context

Looks like we were very optimistic in our ability to detect if a UIImage originated from user's photo library. The initial thought was to detect _assetName for a UIImage that was not an image asset or a symbol image. Turns out that some system generated images, as well as remotely loaded images may also contain a UUID value in "_assetName". So it turns out that this plan will not work.

I tried experimenting with extracting GPS, TIFF and EXIF metadata from the image in an attempt to detect an image that was captured with geolocation, camera make/model or a dateTimeDigitized meta - rationale was that this would hint for a photo library image. Turns out that this metadata is removed by the OS when selecting a photo from photo library, and otherwise not accessible unless through photo library permissions. So this plan will not work as well.

In the end, I decided to repurpose the old maskPhotoLibraryImages config with maskImagesWithHumanFaces as an attempt to at least protect user's photo PII. Used CIDetector which is designed for real-time detection during video capture. In our case, the detection is only executed once and cached for each CGImage. Considering that we run at 1fps this doesn't seem to have a great impact performance (also opted for CIDetectorAccuracyLow which is even faster here). Added support for SwiftUI as well here.

As a fly-by, I noticed that

• with maskAllImage we were also masking UIPickerView's background as well (system adds a blurred background in menu popup which we visit when iterating over the view hierarchy). Added a isSystemImageView check for that
• with maskAllImage we were also masking some animated images (e.g loader) in SwiftUI. Added a isAnimatedImageView check for that

Original slack thread: https://posthog.slack.com/archives/C03PB072FMJ/p1732875958511529

TODO:

• Update docs
• Update RN session replay plugin
• Update RN

💚 How did you test it?

📝 Checklist

• I reviewed the submitted code.
• I added tests to verify the changes.
• I updated the docs if needed.
• No breaking change or entry added to the changelog.

[ioannisj]

This is where we mem-cache the face detection so we avoid detecting multiple times per resource

[ioannisj]

Not sure if we should be explicit here for "<_UICutoutShadowView>" but a general test for internal types here seemed like a good idea at the time

[ioannisj]

Didn't refactor view.isNoCapture() yet to keep the scope low

[ioannisj]

Came across system loaders in SwiftUI. But could also be gif support through UIImage.animatedImageNamed() constructor

[marandaneto]

We cannot do this otherwise we will break people's builds on react native and iOS.
If needed we would need a major bump in the sdk version.

[ioannisj]

We can leave and mark as deprecated for next major release? (and also turn default to false - won't have an impact but just to be clear)

[marandaneto]

As a fly-by, I noticed that

with maskAllImage we were also masking UIPickerView's background as well (system adds a blurred background in menu popup which we visit when iterating over the view hierarchy). Added a isSystemImageView check for that
with maskAllImage we were also masking some animated images (e.g loader) in SwiftUI. Added a isAnimatedImageView check for that

Just an idea: Maybe this can be a separated PR so we can land this asap as this one might drag a bit longer?

[marandaneto]

got it, seems it's the new config under config.sessionReplayConfig.maskPhotoLibraryImages Seems to catch picker view backgrounds as well. On it

IIRC, we added those configs because of the photo user picker, which is solved with the maskAllSandboxedViews option.
maskPhotoLibraryImages is only used if the user has an image and the picture came from that user's photo, right?

The problem is that the maskPhotoLibraryImages config had a side-effect masking what should not have been masked, instead of adding a face detector that is only about the user's face and is not as good as a general "maskPhotoLibraryImages", eg a photo of a document can still be PII but the face detector would fail, seems like an approach for a very specific use case and not worth for now, people could either enable masking all images or expicillity mask that user's face image using the ph-no-capture approach or view modifiers.

IMO we should just deprecate the maskPhotoLibraryImages, change it to false, add a description explaining that this flag has no effect (also in RN btw), and add also in the comments that you can do the ph-no-capture, etc.

WDYT?

[ioannisj]

IIRC, we added those configs because of the photo user picker, which is solved with the maskAllSandboxedViews option.
maskPhotoLibraryImages is only used if the user has an image and the picture came from that user's photo, right?

Yeah, that's correct

The problem is that the maskPhotoLibraryImages config had a side-effect masking what should not have been masked, instead of adding a face detector that is only about the user's face and is not as good as a general "maskPhotoLibraryImages", eg a photo of a document can still be PII but the face detector would fail, seems like an approach for a very specific use case and not worth for now, people could either enable masking all images or expicillity mask that user's face image using the ph-no-capture approach or view modifiers.

Agreed. The initial thought was to come up with an alternative to the originally intended functionality - thinking about this again I fully agree. It's indeed an niche use case - any images other than the picker that are in-app are controlled fully by the developers so they can manually mask themselves and run any other logic, like face detection, they want. Let’s scrap this.

IMO we should just deprecate the maskPhotoLibraryImages, change it to false, add a description explaining that this flag has no effect (also in RN btw), and add also in the comments that you can do the ph-no-capture, etc.

Agreed

[ioannisj]

see: #268