MapLibre Native and Metal

[wipfli]

Apple might remove OpenGL support at some point in time. We need Metal shaders one way or the other. This discussion can be a place to keep notes centralized on that topic and help us find out which way we want to go...

[wipfli]

If someone could explain what the difference is between the Metal and MetalANGLE to a newcomer like me it would be really helpful...

[atierian]

A quick high level overview of the historical context up front.

MapLibre GL Native leverages the rendering API from OpenGL ES to handle calls to the GPU for rendering graphics. OpenGL ES interacts directly with the device's GPU to accomplish this performantly. Apple introduced Metal with iOS 8.0 as an alternative to OpenGL ES that is optimized for Apple devices. OpenGL ES and Metal are very similar in principle, they allow developers to interact directly with the GPU through their own shading language, both of which are subsets of C++. Apple deprecated the OpenGL ES framework with iOS 12.0, but have still not announced an end of life date (when the APIs will no longer work). Regardless of when they announce it, based on previous behavior, Apple will most likely give a long timeline before OpenGL ES no longer works on Apple platforms. It's important to remember that OpenGL ES is heavily used by games, which are a huge revenue stream for them in the App Store.

To recap, OpenGL ES is an open source cross platform rendering framework. Metal is an Apple specific rendering framework that is highly optimized for Apple devices. Based on the OpenGL ES deprecation, it's a general assumption that one day, OpenGL ES APIs will no longer work on Apple devices.

What role does MetalANGLE play here?

MetalANGLE is an open source library that translates OpenGL ES API calls to Metal calls under the hood. The purpose of this is to allow apps / frameworks to execute Metal APIs with very little change to their code base.

[wipfli]

Amazing, thanks @atierian for this great summary

[atierian]

I tried to keep the response above as factual as possible; this one might be more opinionated. Speaking of which, any opinions expressed here are my own and I'm not representing my employer.

Metal vs MetalANGLE

Bottom line up front
We should focus on providing true Metal support rather than investing time into improvements with MetalANGLE.

Longer answer
MetalANGLE is a seriously impressive library that handles extraordinarily complex tasks. However its inflexibility makes it the wrong candidate to build a foundation upon for the future of MapLibre GL Native for Apple platforms. I documented one of these (imo major) issues in #235.

The tradeoffs between both choices essentially boil down to short-term vs long-term investment. In the short-term MetalANGLE wins; it required relatively little effort to get working. In the long-term, MetalANGLE will be significantly harder to maintain. Bug fixes and new features can take significantly longer to complete and the likelihood of hitting hard limitations (without changing the MetalANGLE implementation) is much higher. In contrast, migrating to Metal will be a very large up front effort. The reward of completing that will be easier maintenance, smaller binary size, and better performance.

Suggestion

1. Leave the metal-support branch mostly alone (the metal-support branch is the MetalANGLE implementation). Despite not being perfect, it works today. And in the unlikely event that Apple announces end of life for OpenGL ES in the next few months, developers have a viable option. Basically, I'm suggesting that we keep the metal-support branch alive by fixing security issues and crucial bugs, but nothing else.

2. Start planning true metal support. I've been playing around with this myself in a fork, but it's still very much in a broken early POC state. I would love to work on this, but we (MapLibre) definitely shouldn't depend on me to complete it. It's a huge time investment and time is something I unfortunately don't have much of these days. What I can commit to, is making and publicizing a rough draft high level plan for migrating to Metal that we can iterate on. We could track the status so contributors could more easily see where meaningful contributions could be made.

[hactar]

I'd just like to add that this isn't just about "And in the unlikely event that Apple announces end of life for OpenGL ES in the next few months" - this is all relevant today: M1 and M2 processors. Apple devices running with these processors do not run the OpenGL versions of Mapbox/Maplibre. This includes the iPhone Simulator, making it impossible to develop for iOS on a modern Apple computer. And if you wish to create an M1 Mac app - the same issue applies.

MapLibre MetalAngle is in use in production - Bikemap for example uses it already to allow devs to develop on M1 macs and for the Mac version of the app, though we've upgraded the MetalAngle version when compared to the currently available builds.

[wipfli]

Some comments from yesterday's Technical Steering Committee meeting:

Shader transpilation

• @mwilsnd: proof of concept SPIR-V Cross https://github.com/mwilsnd/glsl2msl
• Pablo: MoltenVK could be an option
  • Is run-time, might have binary size impact
• @mwilsnd: tried Dawn, gave 5 MB extra size
• @mwilsnd: Forge could be an inspiration
• @mousebird: Other approach would be to use a native Metal implementation. True native implementations (native dev tools) versus a common core.
  • People could insert their own layers with a true native Metal implementation
  • @klokan: Could that lead to performance problems, binary size bloat?
    • @mousebird: The current architecture is not well adapted to modern GPUs, also power is better with real native.
    • @mwilsnd: Agree. On the other hand, keeping the shaders closer to GL JS might help us avoid divergence and just having one ground truth.
• @lseelenbinder: Can we get Metal and Vulkan with one implementation?
  • @mwilsnd: for Meta would be best to have that as a compile-time option. Should be possible
• Jesse: How can we facilitate contributions to such large changes?
  • @wipfli: Discussion on GitHub, keep track of what was decided. Write design proposals and stick to them.
  • @birkskyum: First refactor, then implement Metal? @mousebird: yes.
• Pablo: Short term need is Metal. How can we find the best strategy for the long term?
  • @wipfli: Discuss, and document decisions. Last word is always at the Governing Board.
  • @mousebird: We want to only work on things that are supported by the community. This is a requirement from our potential clients at Stamen.
• @mousebird: What Stamen is up to with MapLibre Native
  • If you’d like to engage more on the native Metal issues, please contact me at sjg@mousebirdconsulting.com
  • Notes on Stamen and Metal: https://gist.github.com/mousebird/dcd0672e1a3eb91899f237e8377ce183

[ghost]

Some thoughts:

@mwilsnd: proof of concept SPIR-V Cross https://github.com/mwilsnd/glsl2msl

Note that we can't do this offline at this point, precisely because of the #pragma mapbox stuff, which either turns variables into attributes or uniforms. We'd have to generate all combinations of these offline which is impractical. So at least the shader pre-processor would need an MSL port. We'd also have to introduce a hack so the glsl2msl code would preserve the pragmas (which conflicts with offline-optimization etc. - because the code is not known in advance)

@mwilsnd: tried Dawn, gave 5 MB extra size
Pablo: MoltenVK could be an option

These are a bit confusing.

Current code is OpenGL ES API.

• Dawn implements WebGPU API on a number of backends.
• MoltenVK implements Vulkan API on Metal backend.

What we'd need (right now) is ANGLE, which implements OpenGL ES API on a number of backends (including Metal).
Note: On Windows, ANGLE is typically part of browsers as OpenGL ES serves as base for WebGL. MapLibre GL JS is probably most often rendered by ANGLE.

Changes to the codebase could potentialy port to Vulkan or WebGPU API and then make projects like Dawn or MoltenVK practical.
I assume this is what was brought up by Pablo, but the protocol doesn't make this clear: Pablos comment implies we should be using Vulkan API in MapLibre instead of Metal.
The follow up comment by mwilsnd then implies we should be using WebGPU API in MapLibre instead of Metal or Vulkan.

ANGLE has quite a bit of work to do to get from OpenGL ES (a high-level) API to lower level APIs like Vulkan, WebGPU or Metal.
However, I expect MoltenVK to be relatively lightweight due to more similarities between lower level APIs such as Vulkan, WebGPU and Metal.
This will reflect in performance and binary size.

I don't expect a "5MB extra size" for Dawn, if MapLibre used WebGPU API (which it doesn't, yet).
I don't expect much growth with MoltenVK if MapLibre used Vulkan API (which it doesn't, yet).
I do expect a slight growth with ANGLE if MapLibre used OpenGL ES API (which it does, and there's a PoC in the metal branch).

However, I expect all of these API implementations (Dawn, MoltenVK, ANGLE) will be relatively small (far below 5MB).
They likely don't come with huge look-up-tables and they shouldn't have large dependencies.
What might grow is the symbol table for debug builds?
I'd expect growth much less than 1MB, probably much less after compression.

@mwilsnd: Forge could be an inspiration

What is "Forge" and what could it be an inspiration for?

@mousebird: Other approach would be to use a native Metal implementation.

I'd be against a Metal implementation, because there are probably enough graphics abstraction frameworks which are good for us.
Here are some options:

• Using Vulkan as default API runs natively on many platforms, and for Apple we'd add a small bit of wrapping in the form of MoltenVK = small cost on iOS and zero-cost elsewhere.
• Using Metal as default API runs natively on Apple, and for elsewhere we'd add a bit of wrapping in some Metal -> Vulkan code; however we don't know which direction Apple will take and it's not an open-standard and less people will be familiar with it = zero-cost on iOS and small cost elsewhere, with more development headaches.
• D3D12 etc. will be ignored; similar situation to Metal, except MS likely won't deprecate OpenGL and D3D12 is also mostly unused in mobile space.
• WebGPU API (or another API which isn't directly implemented as a GPU driver) is a trade-off where everyone looses, but at least it's an open standard and many people will be familiar; it's also a lower level API and maps relatively good to Metal or Vulkan = small cost everywhere, including iOS.
• OpenGL [ES] API used to be an API which talked directly to hardware, but these days it's mostly an abstraction which is hard to map to modern GPUs = easy development, but high cost on all platforms in terms of performance, and as OpenGL disappears from drivers we'll also have to bundle some code like ANGLE (although there are signs that OS will bundle this instead to provide OpenGL, even if it's missing from drivers).

There is also a problem that not all users have one of the lower-level APIs implemented in their drivers yet: https://developer.android.com/about/dashboards#Vulkan.
So, if we implemented WebGPU, Vulkan or Metal as the default API, then we'd need something that implements those APIs on an OpenGL backend. However, because OpenGL is more limited, this is likely impossible to do, or we can only use a subset of the lower-level APIs / would need custom abstraction = more development overhead.

One potentially also has to consider custom-layers too. This would be made a lot easier with a common API across all platforms, so custom-layers can be portable.

At the time of writing, OpenGL API makes the most sense (to me) here. In the future I think WebGPU or Vulkan make the most sense.
For Metal, there'd be ANGLE (OpenGL), Dawn (WebGPU) and MoltenVK (Vulkan).
Out of those, I'd only expect ANGLE to hinder performance and binary size.

True native implementations (native dev tools) versus a common core.

I don't think this is necessarily a good argument. Even with MoltenVK and MapLibre using Vulkan as API, one could probably still debug the code (just that you'd debug in MoltenVK). Because the abstraction is relatively thin, I don't expect many issues.
However, obviously debugging ANGLE (= OpenGL API / Metal API) with a Metal debugger is going to suck due to many layers of abstraction and different concepts.

Debugging also isn't that necessary if there's a clean abstraction of the renderer.
That's also the only way forward if we want to implement multiple backends.
Just adding code everywhere is not a practical solution.

@mousebird: The current architecture is not well adapted to modern GPUs, also power is better with real native.

True.

But I'd expect MoltenVK to be lightweight enough to not cause trouble as Vulkan → Metal should be relatively thin; both reflect current GPUs, and I'd argue that Vulkan does it better.
However, we can't go all-in on Vulkan (or Metal or WebGPU) because of lack of support for many devices: https://developer.android.com/about/dashboards#Vulkan. So we still need some OpenGL backend.

So, even then, I'd advocate for only 2 backends: 1. OpenGL and 2. WebGPU or Dawn
If necessary (if MoltenVK or Dawn were too large) one could still add a third backend for Metal, but I think this could turn into a maintenance nightmare. It would also complicate custom-layers. So this might be an optional feature or in a fork for high-bandwidth users like Meta (where every byte of the binary size counts, even if it means breaking some features).

@mwilsnd: Agree. On the other hand, keeping the shaders closer to GL JS might help us avoid divergence and just having one ground truth.

This is the most important argument. We should strive to have 1 codebase for MapLibre Native and JS.

This is why WebGPU and WebGL (~ OpenGL ES) are good candidates as they'd work in both codebases.

Alternatively a custom graphics abstraction could simplify most layer implementations (which is where most development happens) - if those abstractions were similar in Native and JS.
Transpiling shader solves part of the problem, but other algorithmic changes must still be ported from C++ ⇆ TS.
Ideally, we'd factor out more code and use something like asm.js / wasm - that decision is probably as (if not more) important than the graphics API discussion.

@lseelenbinder: Can we get Metal and Vulkan with one implementation?
@mwilsnd: for Meta would be best to have that as a compile-time option. Should be possible

I think we should only implement Vulkan or WebGPU and then ask users to use MoltenVK / Dawn. I expect a small (acceptable) size impact.

We'll still need to keep the OpenGL code around, so it probably makes sense to add more abstraction to the existing graphics abstraction: https://github.com/maplibre/maplibre-gl-native/tree/main/src/mbgl/gfx (and to change it if necessary, for Vulkan, WebGPU and Metal support - however, it looks already somewhat suitable).

Unfortunately for JS, there is no such (clean) graphics abstraction, instead, OpenGL commands are all over the place and porting that code to WebGPU is going to be a challenge. This is an issue, because web has more performance issues to begin with.

---

My Conclusion:

• I prefer OpenGL and Vulkan as backends in Native, with WebGPU being a candidate to replace Vulkan in future.
  • Which backend is compiled in should be a compile time option.
  • For users who need the smalles binary, Metal could be considered as additional backend, but if they want to optimize binary size, they might need a fork (and even then, the binary size is probably much less than a couple of tiles that will also be downloaded).
• Shaders should probably primarily use GLSL (I didn't mention this above, but it's the only good choice I see).
• JS renderer should be refactored to be more similar to Native, to allow for WebGPU in future.
• TS / C++ for most logic should be factored out (and use the common graphics abstraction / renderer that JS and Native will share). Especially with WebGPU, Rust would be a good candidate.

I did, however, not check the state of https://github.com/maplibre/maplibre-rs; it solves many of the problems - one major problem it seems to have is the lack of a legacy renderer (such as OpenGL) for lower end platforms.
Overall, I believe that a rewrite from scratch (copying chunks from either native or JS) doesn't sound too bad either.
It might be less work than refactoring a lot of JS and Native and it will reduce maintenance overhead in the future.
If the API is kept (or a wrapper was provided), then users wouldn't see large changes either.

[wipfli]

Thanks for your perspective, @JannikGM! This is great input I think.

If I understood @mousebird correctly, the idea would be to have multiple steps. Step 1 would be to make the existing OpenGL part in MapLibre GL Native more modular such that it can be replaced with for example a Metal renderer, or also a SPIR-V based renderer.

[mwilsnd]

@JannikGM

What is "Forge" and what could it be an inspiration for?

The-Forge is a cross-platform rendering abstraction, complete with a custom shading language. General design considerations could be taken from projects such as this. Other examples of multi-backend graphics include Diligent and Filament.

I don't expect a "5MB extra size" for Dawn, if MapLibre used WebGPU API (which it doesn't, yet).

The issue comes from having to ship an underlying WebGPU implementation on iOS devices, which would be dawn. It is not probable that Apple would include a low-level WebGPU framework in their devices, at least not for the foreseeable future. That means shipping a WebGPU implementation which will incur a large binary cost. Especially if we need to ship the WGSL shader compiler (dawn tint).

I don't expect much growth with MoltenVK if MapLibre used Vulkan API (which it doesn't, yet).

Could you define much growth here? I'd expect at least 50-100KB in an -Oz build for something like MoltenVK which is a (for Meta) high cost to pay for an abstraction layer alone. Again, if we have to ship a shader compiler as well then that size only goes up.

because of the #pragma mapbox

Yes, the data driven attributes will need special handling. I believe an additional pre & post-processing stage can migrate these pragmas into the MSL. If pragmas are treated as uniforms for the SPIR-V compilation step, then pragma metadata can be first recorded and then generated MSL struct uniforms modified to convert these variables back in to pragmas, including the initializer statements in the shader entry points. The other option is rearranging dependencies and breaking the shaders out in to their own unit, with gl-js and gl-native both consuming the shaders. From there, either the above automated process can take place or manual porting and upkeep of MSL variants can be done.

For Meta the primary concern is binary size. While I'd love to have a centralized API like WebGPU or Vulkan, that is just going to add to MapLibre's binary size for the sake of reduced development effort. OpenGL was the prior universal graphics API and was actually supported on all major platforms for a long time. It is disappointing to see the API ecosystem fragment now but this is mostly because of decisions made by Apple in not supporting Vulkan or their deprecation of OpenGL. I think until Apple changes things, any library looking to offer genuine support for the Apple ecosystem has to support Metal.

[mousebird]

There are a lot of interesting things to discuss here, but let me restate what Stamen/myself are up to and emphasize the point Oliver has brought up.

We're in the middle of a proposal to a company to fund a multi-stage effort that would eventually lead to a Metal port if that's what the MapLibre community wants. If they don't want it, then we won't do it. But we want to produce useful work along the way.

To that end, we're proposing that the first part of the contract is a study and the second part is a modularization of the existing rendering framework. The metaphor I've used is that we're going to dig out the foundation, load the house up on a big truck and then move it to its new location. Like most metaphors, it only works up to a point.

I've actually added my own rendering functionality to a local version of MapLibre Native as a test (for another client) and I can tell you it's really painful. We do weather and we were using a tile set with data values (wind intensity) and our own shader to draw them. I also added an off-screen rendering stage, necessary for things like particles, and that was equally complex and hack-y.

One big goal for the modularization work is to make that kind of project much easier. It should be possible to render style sheet elements and tile sets with your own shaders. It should be possible to extend and modify the rendering pipeline. It should be possible to document that with a nice, friendly tutorial. We do these things with own toolkit (WhirlyGlobe-Maply) so we can borrow many of those ideas.

By approaching the problem in these stages we can provide something of value early on (rendering modularization) and begin the complex discussion of alternatives (Metal vs. Other).

So for now I'm going to hold off on the details of the Metal vs Other discussion. I'd like to sort out the contract with our sponsor first.

[wipfli]

In Slack, we had these discussion threads on shader transpilation with for example SPIR-V Cross:

https://osmus.slack.com/archives/C01G3D28DAB/p1657781011156829

https://osmus.slack.com/archives/C01G3D28DAB/p1663158261210299

[wipfli]

And even earlier slack discussions: https://osmus.slack.com/archives/C02B2CBSNBU/p1641411470008700

[wipfli]

We had an extra meeting on the modernization proposal by Steve and Stamen. The notes can be found here: maplibre/maplibre#118