3d terrain crashes Safari 15.3

[underbluewaters]

I've created a basic 3d style that demonstrates this behavior.

On MacOS 12.2, Safari 15.3, the browser will consistently crash when interacting with this map on both my Intel iMac and M1 MacBook Pro. No issues with the same machines using Chrome. No Javascript exceptions can be provided since the entire browser session crashes.

[ryanhamley]

Thanks for the report @underbluewaters. I can't reproduce this at the moment since I'm on MacOS 11.6 and Safari 15.1. I don't see anything obvious in the 15.3 changelog that might cause a crash so we'll have to dig into this further.

How long does it take for the browser to crash? Is it immediate or does it only happen after several minutes?

[underbluewaters]

I've had reports from different Safari users who haven't had issues but unfortunately I'm not sure what their specs were. I've got at least 2 other users who have seen the same problem.

Sometimes the crash happens immediately, sometimes only after panning the map. More a matter of activity than time. Just now I tried on my M1 and failed to trigger the issue until I put the map into full screen and did a lot of spinning around. After triggering the initial crash however, I could very easily trigger it again even after decreasing the size of the window.

I've attached the Safari crash report.

mapbox-crash.txt

[underbluewaters]

One more note… I don’t regularly test in Safari so I’m not sure that 15.3 is specifically the problem. Could go back further.

[karimnaaji]

Thanks for the report! I believe that they may already have automatic crash reporting, but it might be worth reporting that crash to Apple.

Even if we were doing something particularly bad in the library, Safari shouldn't entirely crash by having invalid memory access in any circumstance due to client code it might run, as it could be seen as a security issue. It may be difficult to find a workaround as it is a native crash directly related to an issue from Safari, it should be addressed there.

[ryanhamley]

git-bisect pointed me to #10494 which changed how cache control data is retained. Specifically, these lines appear to be the culprit:

mapbox-gl-js/src/util/ajax.js

Lines 357 to 361 in 47479ce

	if (window.createImageBitmap) {
	arrayBufferToImageBitmap(data, (err, imgBitmap) => callback(err, imgBitmap, cacheControl, expires));
	} else {
	arrayBufferToImage(data, (err, img) => callback(err, img, cacheControl, expires));
	}

If you revert these to what they were before #10494, the crash appears to resolve itself.

if (offscreenCanvasSupported()) {
    arrayBufferToImageBitmap(data, callback);
} else {
    arrayBufferToImage(data, callback, cacheControl, expires);
}

This change was released in v2.3.0 so it would appear that any maps which use v2.3.0+ on Safari 15.3 could run into this issue. The Safari 15.3 changelog shows a couple of security issues that may have caused this.

[tristan-morris]

This crash appears to occur when running mapbox-gl-js within a webview (WKWebView) on iOS, slightly niche configuration. What is strange is that it's on 15.2 (19C56) and the existing app has been stable for months with this configuration... As far as I know, Safari doesn't update independent of iOS.

Reproduced with map.setTerrain({'source': 'mapbox-dem',}) on/off.

[tristan-morris]

Attached is an ios crash log (of maybe 50):
com.apple.WebKit.WebContent-2022-02-13-122911.ips.txt which also hints at the same region WebCore::BitmapImage::colorSpace() as the OSX crash log.

I applied the change @ryanhamley identified which resolved the problem. However, I reverted/removed the change and haven't been able to crash the device again. Which I'd suggest the change might not be the underlying cause. This could be a @tristan-morris issue somehow as well.

I'm wondering if there's a possibility of bad data being somehow served, cached and then reloaded again and again causing the crash on the device. We keep the user's map looking at the same position they panned to. The map would part-load, crash, repeat. But my error came and went within an hour or so. Uninstalling also didn't help! But then eventually it came good. Now I haven't been able to crash it again.

[underbluewaters]

There could certainly be some bad state being persisted. It's doesn't take too long to trigger the crash on my machine, but once I do it happens almost immediately upon reloading the page.

FYI this is not just happening in a webview or ios on my system. I've seen it on desktop macos using both intel and m1 processors, and my colleague has seen it on ios.

[ryanhamley]

We've determined that this is caused by an underlying bug in Safari 15.2 and 15.3 caused by the decoding of certain types of WebP images with window.createImageBitmap. This bug has existed since Safari 14.3, but 15.2 introduced a regression so that the bug was no longer gracefully handled and led to a crash. We've filed a bug with Webkit for this issue.

We're working on a fix on the server-side to serve PNG images instead of WebP for affected raster-terrain images (most images are not affected by this issue). In the meantime, we've identified a client-side fix using the map's transformRequest option to request PNG images instead of WebP.

const map = new mapboxgl.Map({
    container: 'map',
    zoom: 14.14,
    center: [-118.39997, 33.75873],
    style: myStyle,
    transformRequest: (url, resourceType) => {
        if (resourceType === 'Tile' && url.indexOf('terrain') > -1) {
            return {
                url: url.replace('webp', 'png')
            };
        }
    }
});

Since PNG files are larger than WebP, you may want to limit this as much as possible by detecting Safari useragents and only requesting PNG images for the affected browsers. We'll follow up in this ticket as soon as the server-side fix is available.

[tristan-morris]

Thanks @ryanhamley!! That's amazing you've managed to get to the bottom of what's going on. I had spent the day pulling out the code path and was experimenting with bad webp images :)

We've deployed Cloudflare Worker to re-write the url from .webp to .png, based on your workaround.

[ryanhamley]

@underbluewaters @tristan-morris We have rolled out a server-side change that prevents the problematic WebP images from being served. In our testing, we haven't been able to replicate the crashes. No client-side changes are required to avoid these crashes going forward (the workaround mentioned earlier is no longer necessary). Tiles can be cached on a browser for up to 12 hours, so if a bad tile has been previously cached, this may be replicable until the cache is cleared or the tile expires and is re-requested. Thanks so much for reporting this and helping us understand what was happening.

[underbluewaters]

Awesome! That seems to have done it. I'm not seeing the problem anymore on my Intel system. I'll check the M1 mac when I'm back in the office. Thank you for addressing this so quickly.

[tristan-morris]

Fantastic - thank you @underbluewaters and @ryanhamley!

[underbluewaters]

Just confirming this works as expected now on my M1 MacBook Pro now.