Supported method for retrieving sprite images

[ZeLonewolf]

Motivation

Currently, there is no documented method for retrieving a sprite image from a mapLibre map. This has caused style authors such as myself to hunt around in the codebase looking for a way to access sprite data, such as I did in #2159 where I accessed internal (but nonetheless exposed) variables. Retrieving sprite images is needed for runtime icon manipulation, such as we do in OSM Americana. There is presently a method map.style.getImage(id) in the code, however this is not publicly documented.

Design Alternatives

This functionality is currently accessible via map.style.getImage(id), however, this option is undocumented.

Design

The recommended implementation is to implement Map.getImage() to delegate back to map.style.getImage(id) as is used in other parts of map.ts.

Implementation

Additional work would be needed to implement this across the maplibre-gl variants. However, the naming convention of getImage() follows the same naming conventions elsewhere in this file.

[HarelM]

Can you elaborate on the relevant use cases?
In general you can simply use addImage to add any image you want how ever you want it without even using the sprites, so I'm not 100% sure I understand the motivation...

[ZeLonewolf]

Sure. In OSM Americana we generate custom highway shields using raster sprites as a base "blank" graphic, which is then modified in several ways in order to create the final graphic that's shown on the map.

For example, the US National Highway System uses a graphical shield that looks something like this:

It is infeasible to generate a graphic for every possible US highway route, of which there are hundreds. Instead, we start with a blank, which might look something like this:

To generate the finalized graphic above, the styleimagemissing is hooked when it gets a request for a US 40 graphic. That code then retrieves the blank above, which is stored in the sprite sheet, draws the "40" on top of it, and then invokes addImage to put the image back in. We chose to load the blanks into the sprite sheet so that they're available at runtime rather than inserting URL downloads into the render loop, which I hope we can all agree is a bad idea.

Our sprite sheet presently looks like this. Look at all those pretty shield blanks! You'll notice that there are not "basic" shapes, e.g. circles, rectangles, pentagons, etc. These are all drawn in canvas code and inserted on the fly in addImage() as they're needed. However, the more complex shapes like George Washington's head aren't feasible to draw on the fly so they're loaded into the sprite sheet.

There are additional variants that we need to support as well for the full proliferation of shields. There are several examples of base shields which take on different colors in different cases. For example, historic US Highways use the same US highway shield, but are drawn in brown instead of black. Rather than add an additional shield blank, we can simply retrieve the existing shield blank, re-color it brown, and then insert it back into the sprite list with addImage():

But wait, there's more. We also have bannered routes, in which 1-3 banners need to be displayed as part of the shield graphic:

The most absurd of these examples is the US 30 Alternate-Truck-Business route, which has a triple-stack of these banners:

Alternate approach that doesn't work

The naive approach to shield rendering (the only possibility before #716 was merged) was to say "okay, this road is a certain network, use the shield blank as an icon and then draw text on top in the style". The style would specify a wide blank for a 3-digit route number and a narrow blank for a 2-digit route. The reason this doesn't work is that you can only have one icon, and in the real world, multiple routes can share the same stretch of road, which is called a "concurrency". Additionally, North American cartography expects concurrencies to be displayed as a "snake" of shields that are drawn along the path of the road, as shown here:

In order to achieve this, multiple attributes are present in the tile data which lists the routes that are concurrent for a stretch of road. There is no way to say to maplibre "render this list of graphics and draw them in a repeating snake pattern with each graphic upright". However, there are formatted expressions, which accept images, which allows this to happen. Because this methodology accepts images, they must be constructed using runtime styling as I described above.

In addition, runtime styling (retrieving blanks, modifying them, inserting the final image back in the map) allows the style author to have pixel-perfect full control over the appearance of the rendered graphic.

See OSM American's shieldtest demo for a demonstration of runtime styling in action.

In summary, the historical approach to shield drawing (single image representing "most important highway network in a concurrency") is insufficient for the cartographic need to draw a snake of shields for concurrent routes.

[HarelM]

Thanks for this long explanation. It doesn't answer my question.
In general, I can't help but get the feeling there is some obsession about getting the whole shields "just right", which brings a lot of complexity and awkward decision to the codebase, these are later hard to maintain and understand.
For example #1800 and the entire section of the sprite which talks about how to increase an icon in all kind of complicated ways that a normal human being will need a few long hours to decipher...

Also the last sentence that you wrote feels like putting even more effort to it while it doesn't solve the problem seems like a waste of everyone's time - if we implement things and they are not adopted by the people that pushed them, it's a very alarming sign... I might have interpret what you wrote wrongly though...

[1ec5]

Let’s take a step back for a moment. The ability to get an item is one of the four basic elements of a CRUD interface. The others are already implemented as addImage (create), removeImage (delete), and updateImage (update). An image accessor is needed for parity with the public accessors that were added to iOS/macOS in mapbox/mapbox-gl-native#7096 and Android in mapbox/mapbox-gl-native#9763. In GL JS, getImage was always intended to be a public API: mapbox/mapbox-gl-js#5775. The lack of documentation was merely an oversight because the method wound up on Style instead of Map. The corresponding accessors on other platforms are documented and likely well-used.

I agree with @ZeLonewolf that getImage is useful in conjunction with a styleimagemissing event listener. Americana’s particular use case looks intimidating, but it’s basic map functionality that ideally would be better encapsulated in a map rendering library once Americana’s implementation matures. In the meantime, the MapLibre project has already accepted this use case in #716.

Setting aside shield rendering and other styleimagemissing use cases, there are other practical reasons why a Web application may need to introspect this portion of the style. For example, the application may display a dynamic legend based on the currently visible map features, leveraging queryRenderedFeatures: mapbox/mapbox-gl-js#5775 (comment) osm-americana/openstreetmap-americana#632. A feature querying result can tell you the image name that icon-image resolved to for that feature at the current zoom level, but it can’t give you a URL or raw image data to render the icon separately from the map. Sure, an application could copy part of the layer definition into hard-coded logic for other UI functionality, but the same could be said for any other accessor in the runtime styling API. That alternative wouldn’t allow an application to be style-agnostic and it would hinder a style’s portability across platforms.

[1ec5]

In summary, the historical approach to shield drawing (single image representing "most important highway network in a concurrency") is insufficient for the cartographic need to draw a snake of shields for concurrent routes.

Also the last sentence that you wrote feels like putting even more effort to it while it doesn't solve the problem seems like a waste of everyone's time - if we implement things and they are not adopted by the people that pushed them, it's a very alarming sign... I might have interpret what you wrote wrongly though...

I think there’s been a misunderstanding. @ZeLonewolf isn’t saying we wouldn’t be using this API – we already are! 😅 Rather, he’s highlighting the fact that many OSM-based or GL JS–based maps have historically settled for a basic design for marking routes that falls short of longstanding cartographic conventions.

Don’t take our word for it: a 2015 conference of cartographers across North America overwhelmingly favored grouping shields along a line (option F) over the approach taken by popular Mapnik-based styles such as openstreetmap-carto (option A). Laypeople also notice the difference between concurrency-capable maps (e.g., Apple Maps, Organic Maps) and those that render a generic image or only one shield at a time without any layout capabilities (openstreetmap-carto, Mapbox Streets, and many other GL JS styles out there). This is because print maps have been laying out shields in this manner since at least the 1920s.

What may appear like an obsession is actually a desire for parity with print maps and high-quality digital maps, especially those geared towards audiences in the many countries where concurrent route shields are part of everyday life for drivers, cyclists, and hikers. Yet we know that not everyone has this experience in every country, so we take every opportunity to explain it to those who are unfamiliar. Please excuse the information overload.

[HarelM]

The only valid technical argument I got in this thread is related to CRUD, which is ok, but a bit weak, but I still don't understand what this getImage will be used for and what is the user expected to do with the return value.
It's also worth mentioning that getImage also does some processing, so it's not a regular getter in that aspect.
I need a valid example, not a history lesson, sorry to be blunt... 😕

[1ec5]

A clearer articulation of this project’s overarching goals would help contributors make arguments that are less weak in your estimation. For example, one of the original goals of gl-js/gl-native was platform parity. If that goal remains – as #2064 (comment) seemed to imply – then the method in question needs to be publicly documented, or it needs to be exposed by another publicly documented method in GL JS, for parity with the other platforms. It may not be such an exciting enhancement, but the small things matter too.

You’re absolutely right that the return value would need to be well documented along with the method. Currently, getImage’s return value normally looks like this:

// image
Object { data: {…}, pixelRatio: 2, sdf: undefined, stretchX: undefined, stretchY: undefined, content: undefined }
// data
Object { width: 70, height: 70, data: Uint8Array(19600) }

However, if the image happens to have been added dynamically, it looks like this:

Object { data: {…}, pixelRatio: 2, stretchX: undefined, stretchY: undefined, content: undefined, sdf: false, version: 0, userImage: {…} }
// userImage
Object { width: 40, height: 42, data: Uint8ClampedArray(6720) }

If I’m not mistaken, this corresponds to the following type:

maplibre-gl-js/src/style/style_image.ts

Lines 13 to 29 in c8615fa

	export type StyleImageData = {
	data: RGBAImage;
	version?: number;
	hasRenderCallback?: boolean;
	userImage?: StyleImageInterface;
	spriteData?: SpriteOnDemandStyleImage;
	};
	export type StyleImageMetadata = {
	pixelRatio: number;
	sdf: boolean;
	stretchX?: Array<[number, number]>;
	stretchY?: Array<[number, number]>;
	content?: [number, number, number, number];
	};
	export type StyleImage = StyleImageData & StyleImageMetadata;

part of which is already publicly documented as part of both the API reference and an example:

maplibre-gl-js/src/style/style_image.ts

Lines 31 to 41 in c8615fa

	/**
	* Interface for dynamically generated style images. This is a specification for
	* implementers to model: it is not an exported method or class.
	*
	* Images implementing this interface can be redrawn for every frame. They can be used to animate
	* icons and patterns or make them respond to user input. Style images can implement a
	* {@link StyleImageInterface#render} method. The method is called every frame and
	* can be used to update the image.
	*
	* @interface StyleImageInterface
	* @see [Add an animated icon to the map.](https://maplibre.org/maplibre-gl-js-docs/example/add-image-animated/)

I’ve found this format to be reasonably usable, but the v3.0 version bump does present an opportunity to make ergonomic changes or lock down any parts you think would be unsustainable.

For comparison, on each of the native platforms, the return value is actually the platform’s standard type for an image data container (Bitmap on Android, UIImage on iOS, NSImage on macOS) rather than a custom type. This is workable because these types support concepts such as nine-part images natively (and even animation, in the case of iOS/macOS). The closest analogue on the Web would be HTMLImageElement. But again, as a user, I don’t feel a particular need for something more sophisticated than what’s already being returned and already being used by OSM Americana.

As to how Americana is using this method, I invite you to open Americana and click the Legend button at the bottom. Then zoom in to where you can see POI icons or route shields and click the Legend button again. The icons come directly from feature querying results plus getImage. This dynamic legend functionality is destined to become a plugin for any GL JS–powered application, at which point it would not be able to hard-code major swaths of style-specific business logic.

[HarelM]

I have yet to receive a short motivation for this.
Try formulating the next response as follows:
"As a user I would like to get access to the loaded images so that I can...".

[ZeLonewolf]

Thanks for clarifying, I wasn't aware that there was an expectation for user stories in issue tickets. I've opened #2166 to document this explicitly, which should help mitigate future misunderstandings.

[ZeLonewolf]

I would write a user story for this as follows:

As a style author, I would like to access loaded images so that I can make modified variants of them at runtime without synchronous HTTP fetches.

[HarelM]

Great! This helps a lot!
Why do you need maplire's infrastructure to do so when you have addImage? Why relay on all this complicated internal processing when basically the sprite is not really relevant?

[ZeLonewolf]

Why do you need maplire's infrastructure to do so when you have addImage? Why relay on all this complicated internal processing when basically the sprite is not really relevant?

I do use addImage, but it's not enough by itself.

The problem is that the required contents of the sprite is not known until runtime. Why should I re-fetch the base artwork when it's already loaded and rasterized in maplibre's sprite sheet? It makes more sense to retrieve this work that's already been done, make the needed modifications, and then insert the result back in with addImage.

I disagree that "retrieving a graphic from cache, drawing a number and label on it, and inserting a copy back in the cache" is "complicated internal processing".

[HarelM]

I see.
I'm asking since the concept behind the sprite, from my point of view is of static images, you can, in theory, move all the dynamic stuff out and manage it yourself, saving the pain of breaking, missing APIs etc.
The fact that you use "missing icons" to accomplish this just feels more and more like a hack, one that can be solved more elegantly, IMHO.
Having said that, I understand the requirement for API completeness (although update is also missing IIRC) and exposing this isn't complicated or risky. So I have no real objection around it.

[ZeLonewolf]

although update is also missing IIRC

Update is already implemented:
https://maplibre.org/maplibre-gl-js-docs/api/map/#map#updateimage

[ZeLonewolf]

The fact that you use "missing icons" to accomplish this just feels more and more like a hack

It may feel like a hack, but it's explicitly documented with the purpose of "dynamically generate a missing icon at runtime and add it to the map", which is the purpose that we use it for also. It's also separate from this discussion, which is about whether it's acceptable to use the sprite sheet as a repository for graphics storage.

To be fair - an alternate approach would have us preload and store all the graphic templates outside of maplibre, and otherwise follow the same pattern we're following now, to insert them as needed. However, even if we did that, as you've suggested in a few spots, we would still need to use the styleimagemissing hook to prompt us to create the dynamic graphic, regardless of where all the graphics are stored.

The question of whether maplibre users should be allowed access to maplibre's graphics storage repository for general purposes is more of a philosophical discussion, but I do appreciate your position on it.

[1ec5]

In case this user story was overlooked earlier in #2162 (comment):

For example, the application may display a dynamic legend based on the currently visible map features, leveraging queryRenderedFeatures: mapbox/mapbox-gl-js#5775 (comment) osm-americana/openstreetmap-americana#632. A feature querying result can tell you the image name that icon-image resolved to for that feature at the current zoom level, but it can’t give you a URL or raw image data to render the icon separately from the map. Sure, an application could copy part of the layer definition into hard-coded logic for other UI functionality, but the same could be said for any other accessor in the runtime styling API. That alternative wouldn’t allow an application to be style-agnostic and it would hinder a style’s portability across platforms.

Rephrased:

As a user, I would like to replicate the appearance of a map feature outside of the map in a data-driven manner. For aesthetic reasons, I’m uninterested in simply screenshotting the map, because I want to isolate the feature and discard its geometry. I can already do this using publicly documented APIs for any feature querying result in any layer. I would like access to the loaded images to turn image-typed property values into images.