Support more writing systems using HarfBuzz

[wipfli]

MapLibre GL Native currently does not support writing systems like for example Indic scripts or Khmer.

I think we should support more writing systems such that people from all parts of the world can use MapLibre...

Let's use this as a tracking issue to collect ideas and material how we can extend writing system support. Maybe Harfbuzz will be the best tool.

[wipfli]

If only I knew which part of the code base was responsible for text rendering...

[alanchenboy]

GrabMap Khmer Render solution.pdf
This is the slide how grabmap Implement harfbuzz + freetype.

[brawer]

Here’s my personal advice on this. (Disclaimer: While I’ve spent quite many years working on maps, internationalization, fonts, and text rendering, I’m a complete newbie to MapLibre. Please apologize for anything that doesn’t make sense in the MapLibre context).

Generally, GrabMap’s approach for Khmer goes in the right direction, but it’s incomplete, and you don’t need a separate hack for each language. Instead, once you’ve implemented text rendering correctly, a single code path can display all text in any language. This will also improve the display of Latin/English text in terms of kerning, ligatures, typographic features, and a big chunk of what’s needed for Emoji. Unfortunately, getting this right is fairly non-trivial, and also not exactly well documented. However, there’s several open-source implementations that might serve as inspiration. Personally, I’d mainly recommend looking at the source code of Chromium and Firefox; they both have state-of-the-art text rendering stacks. Another source of inspiration might be the Pango library used by GNOME/GTK and Qt, although with some reservations.

Here’s how modern text rendering works from a high level:

1. Start by running the Unicode Bidirectional Algorithm. This results in a sequence of “bidi runs”. A good implementation is GNU FriBiDi, licensed under LGPL-2.1.
2. Split the bidi runs into “script runs”, which are contiguous character sequences in the same writing system (and writing direction, as per step 1). This step is called “script itemization”, and there’s some subtleties. Some are documented in Annex 24, but this is not the full truth. There’s an implementation in the Win32 API, and one in Qt although one should double-check how good that is. It would be best to look at the Chromium and Firefox sources.
3. Send each script run (with correct bidi and language tags) through HarfBuzz, using the default font for the chosen style. This step is called “shaping”, and results in “glyph vectors”, a sequence of (glyph ID, x, y).
4. If the font hasn’t been able to display some pieces of text, the glyph vector will contain glyphs with glyph ID zero. In this case, find the failing substring, and recursively call HarfBuzz to shape the failing substring with a fallback font. Finding fallback fonts is non-trivial, performance-critical, and platform-specific. Again, have a look at the Chromium and Firefox sources.
5. Break the labels into lines. Line-breaking (and hyphenation, which might be quite useful for maps) is non-trivial, language-specific, and performance-critical. Unicode defines an algorithm in Annex 14 but again, this isn’t the full story, so check out what web browsers do. It might be worth supporting soft hyphens, so a server-side tile renderer could use large, language-specific dictionaries (like those used by TeX) to find potential hyphenation points, and pass them through vector tiles to MapLibre.
6. Render the glyphs. Given that MapLibre already has GPU rendering with Signed Distance Fields, GrabMap’s use of FreeType seemed a little surprising to me; there’s no obvious reason why SDF wouldn’t for for Khmer, Devanagari, or any other script in Unicode. However, if MapLibre wants to eventually support color fonts or color Emoji, there’ll be some complications here.

Language tags: For good text rendering, you’ll actually need to know the language of each label being rendered, and pass it down the rendering stack (into HarfBuzz) as an IETF BCP-47 language tag. This is the same language code that’s also used for language tagging in HTML, XML, and other data formats; modern browsers use it to tweak text rendering. Knowing the language mainly matters for East Asia, where certain glyphs should look slightly differently depending on the language (and region/country, which is part of IETF language tags). For example, this picture illustrates how the same Unicode codepoint U+8FD4 should look for various languages/regions. Knowing the language can also make a visible difference when rendering certain minority languages in South-East Asia such as Shan or Mon, and even for European languages like Polish, but these cases are admittedly rather high-end typography. Unfortunately, the MapBox Vector Tile format does not encode the language of labels. In the long term, I’d recommend extending the MVT format so that a tile renderer can encode the language. (In case extending MVT is too complicated: Unicode had once defined an escape mechanism that encodes language tags as special codepoints within a character stream. While Unicode has deprecated and strongly discouraged this escape mechanism, it might be still appropriate for MapLibre if the MVT format can’t be extended). For the medium term, I’d recommend implementing a client-side heuristics in MapLibre, at least for East Asia. That heuristic might also be useful when a tile doesn’t come with language tags, or when rendering other formats such as GeoJSON. In the short term, my recommendation would be to do nothing: Users will still be able to read the text, but they may complain about the “wrong font” being used. Knowing (or guessing) the language of each rendered label will also be needed if MapLibre ever wants to compute hyphenation points on the client side.

Color and variable fonts: Supporting color (and variable) fonts is a little complicated. But certainly doable, and I think both color and variations could have very nice applications for cartography. But it’s clearly less important than making text readable in the first place.

Web fonts: According to their slides, GrabMap seems to load Noto Sans Khmer over the web. However, on most modern devices, this wouldn’t actually be necessary; both Apple and Google bundle most of Noto with their operating systems. Although Apple hides the presence of Noto from its user interfaces, apps can still access the glyphs. Likewise, Microsoft Windows bundles a lot of international fonts. Said that, it certainly would make sense for MapLibre to support web fonts, both for custom styling and as a fallback when device fonts don’t cover the Unicode range needed for display. My recommendation would be to re-implement MapLibre’s text stack so it supports web fonts in the same way as Chromium and Firefox. However, this would likely be a sizeable chunk of work.

Font formats: Maybe I’m missing something here, but to me personally, the Mapbox font API seems a little weird. Again, my recommendation would be to make MapLibre behave like a modern web browser, support the same (standard) web font formats, and perform the conversion from Bézier curves to GPU-renderable Signed Distance Fields on the client device.

Styles: To define the style of map labels, my suggestion would be to implement Text and Fonts of CSS3, just like a modern web browser. This would be quite a bit of work, though.

[1ec5]

Some of the steps above are implemented, but steps 3 and 4 are the weak points.

Break the labels into lines. Line-breaking (and hyphenation, which might be quite useful for maps) is non-trivial, language-specific, and performance-critical. Unicode defines an algorithm in Annex 14 but again, this isn’t the full story, so check out what web browsers do. It might be worth supporting soft hyphens, so a server-side tile renderer could use large, language-specific dictionaries (like those used by TeX) to find potential hyphenation points, and pass them through vector tiles to MapLibre.

Soft hyphens and zero-width non-joiners were implemented in mapbox/mapbox-gl-js#1617 mapbox/mapbox-gl-native#2598. Dictionary-based line breaking was never implemented, however: mapbox/mapbox-gl-native#7362. Instead, one approach has been for the vector tiles to include ZWNJs at line breaking opportunities. This saves clients from having to bundle all the relevant hyphenation dictionaries, which could get enormous. An alternative approach might be to lay out the text in the DOM with CSS hyphenation enabled and somehow introspect the lines that result.

Although the homegrown text layout engine in GL JS and gl-native is very basic in most respects, it does feature relatively uncommon support for vertical justification (in other words, balancing the lengths of lines in a label): mapbox/mapbox-gl-js#3420. This feature is more important for maps than for documents in general, so this library implements balanced lines by default even though it’s only in the CSS4 specification and not yet supported by any browser. (A polyfill is available.)

Knowing the language can also make a visible difference when rendering certain minority languages in South-East Asia such as Shan or Mon, and even for European languages like Polish, but these cases are admittedly rather high-end typography.

European languages would benefit from GL JS knowing the text’s language, even in “low-end” typography. Namely, each language has different rules for uppercasing and lowercasing text: mapbox/mapbox-gl-js#3999. That issue predates expressions, so the proposed technical design was clunkier than what would make sense today. Today, the upcase and downcase expressions would simply take a collator as an additional argument. Ideally, collators would be able to take expressions as values, not just hard-coded literals: mapbox/mapbox-gl-js#6155 (comment).

[alanchenboy]

6. Render the glyphs. Given that MapLibre already has GPU rendering with Signed Distance Fields, GrabMap’s use of FreeType seemed a little surprising to me; there’s no obvious reason why SDF wouldn’t for for Khmer, Devanagari, or any other script in Unicode. However, if MapLibre wants to eventually support color fonts or color Emoji, there’ll be some complications here.

GrabMap does use SDF for complex text (Khmer && Burmese), GrabMap uses Freetype to render the 8-bit bitmap and then convert it to the SDF image, just like the LocalGlyphRasterizer.

[brawer]

Worth reading: Text layout is a loose hierarchy of segmentation. There’s been several attempts to bundle text layout into a single library, such as Raqm, Minikin, ICU Paragraph Layout, or Cobbletext. Personally, I’d recommend looking at them for inspiration, although I’m not sure if they really fit MapLibre. Another source to consider might be lib/ui/text in Flutter: this is a fork of Chrome’s text handling, and less entangled than Chrome, but it’s deeply integrated with Flutter so not directly usable for MapLibre. On the other hand, if you just want to fix rendering quickly with little work, and if you don’t care about line breaking, hyphenation, and rendering text on GPUs, Raqm might be a good solution. Minikin implements line breaking, but being part of Android, it would need to be ported to other platforms.

[maxammann]

Thanks for the great summary, I'll check it out later in more detail!

A thing I was wondering regarding the Glyph rendering aspect: Does maplibre really need resolution independent glyph rendering?

In Maplibre the glyphs "exist" within the 3D world. That means glyphs need to be resized depending on the zoom.

Wouldn't it be enought for the major usecases to render glyphs with a static resolution as an overlay?

[brawer]

Does maplibre really need resolution independent glyph rendering? In Maplibre the glyphs "exist" within the 3D world. That means glyphs need to be resized depending on the zoom. Wouldn't it be enought for the major usecases to render glyphs with a static resolution as an overlay?

Personally I’d find it a nice feature if text were able to grow and shrink with the rest of the map. Zooming would feel smoother that way, especially on deep zoom levels. But admittedly that’s pretty far off; the current user experience doesn’t seem to need this. Also, you can always rasterize glyphs on the CPU (by calling FreeType) to any desired resolution; it just won’t feel as smooth as when doing it on GPU.

[maxammann]

Personally I’d find it a nice feature if text were able to grow and shrink with the rest of the map. Zooming would feel smoother that way, especially on deep zoom levels. But admittedly that’s pretty far off; the current user experience doesn’t seem to need this. Also, you can always rasterize glyphs on the CPU (by calling FreeType) to any desired resolution; it just won’t feel as smooth as when doing it on GPU.

That is also my feeling. Yeah it would be cool to have resolution independant glphy rendering. But at the same time I'm wondering if we really need it. I honestly don't know right now why MapBox when that way originally. There must be some reason why it's neccassary.

[ramSeraph]

I just want to add one more library for consideration w.r.t client side text breaking - https://github.com/unicode-org/icu4x

This one is written especially for cases like maplibre.

Edit:
I thought this was an issue in maplibre-gl-js. This might not be required for maplibre-gl-native.

[wipfli]

Unfortunately, the MapBox Vector Tile format does not encode the language of labels. In the long term, I’d recommend extending the MVT format so that a tile renderer can encode the language.

@brawer feel free to add this point to the discussion at nyurik/future-mvt#1

[wipfli]

We are not the first ones to think about supporting more writing systems. Here are some Mapbox issues:

• Add support for rendering characters from unicode supplementary planes mapbox/mapbox-gl-js#4001
• Full "complex text" support: indic scripts, ligatures, kerning, etc. mapbox/mapbox-gl-native#7774
• Use HarfBuzz for text shaping mapbox/mapbox-gl-native#7528

[wipfli]

Almost 10 years ago, in the third issue created in the Mapbox GL JS repo, people talked about HarfBuzz mapbox/mapbox-gl-js#3.

[maxammann]

Almost 10 years ago, in the third issue created in the Mapbox GL JS repo, people talked about HarfBuzz mapbox/mapbox-gl-js#3.

Uuuh doesn't that issue suggest that they already used freetype/ICU lib?

[1ec5]

Some history: mapbox/DEPRECATED-mapbox-gl#4 (comment).

[alanchenboy]

Mapbox uses icu to shape arabic.

[1ec5]

That’s unrelated to the discussion that was asked about above. Originally, Mapbox used FreeType on the server side for text in general but yanked it out because it wasn’t flexible enough for styling on the client side. Much later on, the mapbox-gl-rtl-text plugin was introduced to integrate a very small subset of ICU that shapes Arabic and Hebrew text using the legacy Unicode compatibility forms for those scripts. However, this plugin’s approach can’t be generalized to other scripts.

[1ec5]

Also lots of discussion in mapbox/DEPRECATED-mapbox-gl#4.

[ramSeraph]

My set of reading material/potential repos.. I hope it helps and is not an overload of data :)

• font-rendering-overview:

  • https://drafts.csswg.org/css-text-3/
  • https://drafts.csswg.org/css-inline/
  • https://w3.org/TR/SVG/text.html
  • https://mrandri19.github.io/2019/07/24/modern-text-rendering-linux-overview.html
  • https://increment.com/programming-languages/unplain-text-primer-on-non-latin/
  • http://behdad.org/text/
  • https://scripts.sil.org/cms/scripts/page.php?item_id=IWS-Chapter09
  • https://bdon.org/maps-for-the-web/
  • https://raphlinus.github.io/text/2020/10/26/text-layout.html
  • https://faultlore.com/blah/text-hates-you/
  • https://github.com/n8willis/opentype-shaping-documents/ - allsorts attempt at documenting opentype shaping

• harfbuzz and mapbox problems:

  • https://mail.gnome.org/archives/gtk-i18n-list/2009-August/msg00025.html - harfbuzz api commentary
  • Use HarfBuzz for text shaping mapbox/mapbox-gl-native#7528 - mapbox issue researching harfbuzz and font subsets
  • https://github.com/mapbox/mapbox-gl-native/wiki/Text-Rendering - mapbox summary of text rendering

• gpu font rendering:

  • https://wdobbie.com/post/gpu-text-rendering-with-vector-textures/#fn:ca55d41100a7dc46c62e28d5643b32c3:9
  • https://terathon.com/i3d2018_lengyel.pdf
  • https://medium.com/@evanwallace/easy-scalable-text-rendering-on-the-gpu-c3f4d782c5ac

• font-rendering-tools:

  • https://github.com/harfbuzz/harfbuzzjs - harfbuzz wasm port
  • https://github.com/RazrFalcon/rustybuzz - rust harfbuzz port
  • https://github.com/loveencounterflow/rustybuzz-wasm - rustybuzz wasm wrapper
  • https://github.com/opentypejs/opentype.js - js opentype font parser and shaper
  • https://github.com/Pomax/lib-font - js font parsing
  • https://github.com/cupcakearmy/glyphance - font subsetting by unicode range
  • https://github.com/arielm/chronotext-cross - source for tangrams font handling code
  • https://github.com/hjanetzek/alfons - also related to tangram font handling
  • https://github.com/dfrg/lipi - font independent shaping lib
  • https://github.com/dfrg/bidi
  • https://github.com/servo/unicode-bidi
  • https://github.com/unicode-org/icu4x - unicode libraries with customisable data pipelines, includes segmentation using LSTM for Thai
  • https://github.com/mdebbar/icu4x_bidi - unicode-bidi implementation using icu4x data pipeline
  • https://github.com/savage13/agg - agg rust port
  • https://github.com/behdad/glyphy - sdf generation
  • https://github.com/servo/pathfinder - font rasterisation lib
  • https://github.com/pop-os/cosmic-text
  • https://github.com/mapbox/mapbox-icu
  • https://github.com/fetisov/ttf2mesh

• Meta Links:

  • https://wiki.nikiv.dev/design/fonts
  • Enhanced text layout: links, thoughts and discussion koreader/crengine#307

EDIT: Cleaned up and recategorized

[1ec5]

Some good news on the JS side is that apparently it is possible to get HarfBuzz down to a reasonable size: mapbox/mapbox-gl-native#7528 (comment). For a sense of perspective, the numbers in that comment are only a fraction of the size of the OpenStreetMap Americana style JSON.

[maxammann]

@ramSeraph Thanks for that collection. Do you mind if I include that in https://maplibre.org/maplibre-rs/book/development-documents/font-rendering.html?

[ramSeraph]

@maxammann you can definitely include them.

I wasn't sure if I should put effort into maplibre-rs or here. I wasn't sure how far from production maplibre-rs was and I don't know rust( that can be remedied though :) )

I have to say, it was heartwarming to see the top priority of this issue at maplibre-rs - maplibre/maplibre-rs#36 (comment)

Is there a place where I can add more of the rust text util research? ( I can see that you already have looked at a few things I have )

[maxammann]

Basically that issue you mentioned it the place to start discussions :)

maplibre-rs is still rather in a proof of concept phase. But soon it will exit that phase and get a first alpha release. After that it potentially is already usable for simple usecases.

Feel free to join the matrix chat to discuss text rendering :) https://matrix.to/#/!FJjNawYTQMCYZmCfbw:matrix.org

[ramSeraph]

Unfortunately, the MapBox Vector Tile format does not encode the language of labels. In the long term, I’d recommend extending the MVT format so that a tile renderer can encode the language.

@brawer feel free to add this point to the discussion at nyurik/future-mvt#1

I wonder if this should be part of the maplibre style spec or MVT specification.

Also, please consider dropping the glyph api from the maplibre style spec if possible. The alternative mentioned in this issue tracker seems like a good idea.

maptiler/tileserver-gl#641 (comment)

[wipfli]

Thanks @alanchenboy for the nice presentation! Here are some links from the chat:

• iOS/macOS already uses Core Text for CJK: https://github.com/maplibre/maplibre-gl-native/blob/ea234edf67bb3aec75f077e15c1c30c99756b926/platform/darwin/src/local_glyph_rasterizer.mm
• https://svgwg.org/svg2-draft/text.html#InterfaceSVGTextContentElement
• https://github.com/tangrams/tangram/blob/master/src/styles/text/text_segments.js#L24
• https://alanchenboy.github.io/

[alanchenboy]

The font face name is fixed in local_glyph_rasterizer, in the old version is hardcoded like iOS is "PingFang", Android is "sans-serif". That means you can't specify font face in style.
So CJK can render locally but only on font face type, all CJK texts in the map are the same type, It does not make sense.

Using the Harfbuzz standalone is a cool idea, maybe I will practice it in the future.

[ramSeraph]

So, If I am understanding this correctly, if local_glyph_rasterizer is enabled, the remote fonts doesn't get used even when they become available after getting downloaded with delay?

Also these new experiments in opentype(?) might also be of interest for the CJK problem - https://github.com/harfbuzz/boring-expansion-spec#3-enable-components-to-take-advantage-of-variations.

And some experiments with subsetting fonts based on clusters derived from usage dictionaries rather than plain glyph ranges might also help alleviate the CJK size problem.

[ramSeraph]

I think the maplibre style spec should have an option for how long to wait for a high priority font before falling back to a backup font and what the behaviour should be once the high priority font becomes available.

[1ec5]

This guide describes the current font selection behavior on iOS/macOS. The style can influence the font selected on the client side, but the style is constrained by the fact that the Fontstack API interprets the entire font fallback list as essentially the name of a single composite font.

[alanchenboy]

So, If I am understanding this correctly, if local_glyph_rasterizer is enabled, the remote fonts doesn't get used even when they become available after getting downloaded with delay?

correct

[wipfli]

@alanchenboy thanks for the nice presentation.

I wonder: Can we remove the language encoding from the GrabMaps solution and decide which font file to used based on the Unicode range?

[alanchenboy]

There is no spacing for the clustering glyph in the Unicode range. And if we want to use harfbuzz the font file is the must-have data. But if we don't wish to add a new API in style, we could also directly use the "glyphs" setting and define some code range PBF file is the font file or a PB file that includes the font binary data, but this implementation is just as same as GrabMaps just the API name different and the PBF file format different.

Unless we want to do the shaping on our own, we need to study the cluster protocol, how to encapsulate the clustered glyph in PBF, and how to define the Unicode value for the clustered glyph and the codepoint offset information, etc. So that case maybe the Unicode range is acceptable, but also needs to change to support shaping in PBF format, and make it not the totally Unicode range anymore.

[wipfli]

Thanks @ChrisLoer for pointing out a completely different approach than SDFs to scale-indelendent text rendering: vector font rendering. Gets used at Figma https://medium.com/@evanwallace/easy-scalable-text-rendering-on-the-gpu-c3f4d782c5ac

[maxammann]

@FabianWildgrube implemented a prototype of it here some time ago :) maplibre/maplibre-rs#4

[ramSeraph]

Wrong link I think - maplibre/maplibre-rs#40

Can it do halos?

[alanchenboy]

This looks really cool, can't wait to see it in Maplibre.

[wipfli]

https://github.com/wipfli/maplibre-canvas-labels

Played around with rendering text labels in a canvas. Its a bit slow, but once the images are loaded, it does not look too bad...

Demo:

[wipfli]

I hope I spam this discussion not too much with stuff about my little proof-of-concept, but look, the demo now uses TinySDF for the labels:

[wipfli]

You are right, @alanchenboy. Here is some nice Khmer:

[ramSeraph]

I hope I spam this discussion not too much with stuff about my little proof-of-concept, but look, the demo now uses TinySDF for the labels:

Minor issue, I am guessing some problem with text length calculation

My OS: MacOS Ventura Version 13.0.1 (22A400)
My browser: Google Chrome Version 110.0.5481.77 (Official Build) (arm64)

EDIT:
Don't know how to check system fonts :( . ( Begs the question of how I am even qualified to talk about any of this : ) )

[wipfli]

I saw clipping in some cases too. I use a large 30 pixel boarder in tinysdf which helps often but for long labels is not enough.

[wipfli]

Solved the clipping issue. Look at all these beautiful writing systems in South-East Asia:

All text is now rendered in a html canvas element. The name tag value is directly used.

[wipfli]

@ramSeraph and @alanchenboy what should be our next step? Bring Grab's solution to a proof-of-concept in MapLibre GL Native?

[ramSeraph]

I would personally suggest putting in more thought into the style spec before adding any more hacks into the maplibre-gl-js or native codebases.

I understand that would mean a loss of momentum here. But, given that the last 10 year old mistake is still haunting us.. this would be the wise path.

I also think @brawer and @1ec5 are more knowledgable people for designing such a spec( I am passing on the hot potato here :) )

[1ec5]

GL JS and gl-native make two longstanding poor assumptions that block progress on many of the proposals raised so far. I suspect these would be good tasks to chew off first, regardless of the overall solution:

• A one-to-one relationship between Unicode codepoints and font glyph indices, due to the Mapbox Fonts API’s design – also, implicitly, the assumption that a glyph is font-agnostic. This is kind of insidious within the codebase, but it is possible to refactor: Allow using system fonts or fonts bundled in the app mapbox/mapbox-gl-native#7862 (comment). If I understand correctly, Grab’s implementation attempts to preserve backwards compatibility by allowing either a codepoint or a glyph index in the same slot, depending on the script. But a cleaner design would standardize on glyph indices, interpreting a Fonts API PBF as a font where all the glyph indices happen to be codepoints by coincidence. Breaking the codepoint–glyph index assumption would facilitate support for non-BMP characters, including some CJK characters that OpenStreetMap data requires: Add support for rendering characters from unicode supplementary planes mapbox/mapbox-gl-js#4001. Also many emoji.

• Each glyph can be rasterized individually, out of context, and reused out of context. The local glyph rasterizer preserves this assumption. This is why the experiments with Core Text in Allow using system fonts or fonts bundled in the app mapbox/mapbox-gl-native#7862 (comment) Full "complex text" support: indic scripts, ligatures, kerning, etc. mapbox/mapbox-gl-native#7774 (comment) looked pretty but still failed to shape Indic text correctly. Grab sends an entire text fragment to HarfBuzz to ensure that each glyph is considered as part of a whole word. In principle, the same approach would’ve worked in the Core Text experiments, but it would’ve required changes outside of the local glyph rasterizer. @wipfli’s canvas-based experiment also works around this assumption by forcing GL JS to rasterize the label as a whole and treat the image as a black box, with implications for rendering performance and memory usage.

[alanchenboy]

• Grab’s implementation attempts to preserve backwards compatibility by allowing either a codepoint or a glyph index in the same slot

This is a good point, but it still could be improved in the layout stage. like: "simple text complex text simpletext" --> "pbf indics(unicode value)" + "glyph indices" + "pbf indices" --> "pbf indics(unicode value) glyph indices pbf indices", we can combine the shaped texts in the origin slot together, GrabMap just split the slot by language, maybe it should be reunited again after shaping.

[1ec5]

A full solution for complex text rendering would need to address line-placed labels, curving text along a curved line: mapbox/DEPRECATED-mapbox-gl#4 (comment). In Slack, @bdon pointed out that curving Devanagari text would cause the horizontal line to get broken up in much the same fashion as maplibre/maplibre-gl-js#2170.

To demonstrate this issue, here are SVGs of Hindi and Arabic text placed on the inner or outer side of a spiral:

Chrome (which uses HarfBuzz for text shaping) and Firefox (which uses Graphite) both show bunching on one side of the curve and gaps on the other. Safari makes an utter mess of it; apparently it sends isolated characters to Core Text, just like in mapbox/mapbox-gl-native#7862 (comment).

I wonder how other custom text layout engines have addressed shaping calligraphic text along a path. In Arabic, inserting kashida characters could close the gaps as a workaround, but I don’t think there’s an equivalent for Devanagari. In any case, I assume that native speakers would prefer these seams over the status quo.

[ramSeraph]

Started a requirements discussion for i18n support at https://github.com/maplibre/maplibre-gl-style-spec/discussions/37 as a follow up.

[1ec5]

maplibre/maplibre-style-spec#145 (reply in thread) has a proof of concept of first-class support for complex text by properly segmenting the text by grapheme cluster, similar to how Blink and Gecko render the above SVGs (as opposed to how WebKit does it).

[ramSeraph]

Text by grapheme cluster might not be enough for complex text. I think you will need typographic clusters( mentioned in the css spec ). I hope I am wrong and the grapheme clustering is enough :)

I suspect a lot of typographic clustering might be independent of the font and just dependent on some unicode data and some logic encoded in the opentype complex text state machine. This assertion needs to be verified though. I think https://github.com/dfrg/lipi might be an implementation of the typographic clustering.

EDIT: Looks like the extended grapheme cluster as defined by UAX29 might be enough. And the state machine which I thought was in the opentype spec might actually be in the UAX29 spec. Still not entirely sure though.

[wipfli]

@ramSeraph can you have a look at the demo an let me know if you find some issues with Tegulu?

https://wipfli.github.io/index-by-grapheme/#map=5.21/14.987/78.891&language=te

[1ec5]

As the CSS specification notes, a grapheme cluster is an approximation, a compromise of sorts, but clients are expected to tailor the segmentation depending on the operation. I get the impression that it’s designed to be primarily useful for collation and interactive text selection but not necessarily accurate for text rendering.

Isolating each grapheme cluster into a separate string prevents the browser’s text engine from respecting initial/medial/final forms. For example, Intl.Segmenter segments “يعاملو” into six segments, the first of which is “ي” rather than “يـ”. So while it might be good enough for Indic scripts, it will still butcher calligraphic scripts – or for that matter, cursive Latin fonts. At least Arabic-language text can be spared by running the text through the mapbox-gl-rtl-text plugin first, but that leaves out other orthographies like Persian and Urdu. A zero-width joiner can override this behavior, but the result depends on the script.

For that matter, grapheme cluster segmentation is locale-dependent, but the text layout engine has no idea which language the label is in. If you don’t specify the locale when constructing the Intl.Segmenter, it’ll default to your current locale, which should be fine in most cases, but maybe not when rendering a panlingual map of the world. Ideally, TileJSON would indicate the language of each field, as it sometimes does today, in unstructured form): mapbox/tilejson-spec#14. Then the style could pass in that locale when specifying a collator or other language-specific operation. There could also be a general language property, akin to xml:lang in XML or lang in HTML, that would affect the segmentation.