Some character combinations aren't rendered

[MaxSem]

Minimum repro: "א-" - that's Hebrew letter aleph and minus sign. This is reproduceable at least with Hebrew, Arabic and Georgian characters, if appropriate fonts are present in fallback chain.

In the sample attached, aleph is not properly rendered in combination with minus sign while it has no problems with being rendered alone or combined with Latin (and that's the same Unicode range as minus) or Chinese characters.

Full XML to reproduce this bug, along with fonts used and C++ and Node.js examples is at https://github.com/MaxSem/mapnik-bug

[artemp]

Debugging reveals that call to ubidi_getDirection(bidi); in itemizer.cpp returns UBIDI_MIXED for aא+ input
This results in three text_items (correctly)

text_item:0,1 // LTR
text_item:1,2 // RTL
text_item:2,3 // LTR

and UBIDI_RTL -> only one item for א+

text_item:0,2 // RTL

@MaxSem - I think the problem is that 'ICU itemizer' sets RTL direction to the whole run even there is a neutral char '+' or '-' etc. I'm not sure if this is expected and we need look at harfbuzz shaping or? Your feedback would be super useful as I'm not an expert on RTL languages.

/cc @springmeyer

[artemp]

http://www.w3.org/International/articles/inline-bidi-markup/uba-basics

[MaxSem]

I don't know anything about it either, but I have people around who do:) My colleague says that the second example is correct too.

[artemp]

ICU has

ubidi_setReorderingMode(..);
ubidi_setReorderingOptions(..);

Passing various option combinations ^ result in "א-" rendered or not. But I don't know if we should be using non-default options or .. I feel we need an expert advice here :)

/cc @springmeyer @behdad @MaxSem

[behdad]

I have no clue what this bug is about, but playing with bidi settings sounds like the wrong approach. It's definitely a bug in your (mapnik) code.

[behdad]

Debugging reveals that call to ubidi_getDirection(bidi); in itemizer.cpp returns UBIDI_MIXED for aא+ input
This results in three text_items (correctly)

text_item:0,1 // LTR
text_item:1,2 // RTL
text_item:2,3 // LTR

and UBIDI_RTL -> only one item for א+

text_item:0,2 // RTL

@MaxSem - I think the problem is that 'ICU itemizer' sets RTL direction to the whole run even there is a neutral char '+' or '-' etc. I'm not sure if this is expected and we need look at harfbuzz shaping or? Your feedback would be super useful as I'm not an expert on RTL languages.

These are all correct. No matter what the bidi algorithm returns, if a char is not being displayed, that sounds like a problem in mapnik code. That's regardless of whether the bidi is correct or not.

[kroll-j]

Hmmm. I looked into this a bit and it seems to happen with all non-alphanumeric chars, not just minus. See here. Haven't figured out why this happens yet.

[nyurik]

It seems to fail whenever RTL character (aleph) is preceded by the direction-neutral character.

[nyurik]

After some debugging, I think I understand the problem, but not the solution. Script itemizer treats USCRIPT_COMMON characters (such as '-') as being equal to USCRIPT_HEBREW - scrptrun.cpp#L112 (see unicode constants). Thus an aleph 'א' followed by a '-' end up in the same substring, with their script value being set to USCRIPT_HEBREW. During the text shaping in harfbuzz_shaper.hpp#L120, the code point for '-' is missing from the hebrew font. But it tries to render the whole substring in one font, and failing to find a font that has every character of the substring, it draws '-' as a box.
I tried to change the sameScript() function to treat common as different script, thus causing it to break into parts, but that fails because common is the initial value of any substring, thus it causes an infinite loop. Suggestions and fixes are welcome. ))

[khaledhosny]

You don’t need to change the script itemisation, you need to fix the font fallback mechanism; If mapnik has one it is clearly failing here.

[nyurik]

If i understood it correctly, mapnik only supports the entire sub-string in the same font, because its not drawing it one letter at a time, but one sub-string at a time. That's why I think we would need to split the string further. BUT, I might be wrong here.

[khaledhosny]

Someone who knows about mapnik font fallback handling (or lack of) needs to chime in here, but one simple and effective way to handle font fallback is what is briefly described in this mailing list post: http://lists.freedesktop.org/archives/harfbuzz/2015-October/005168.html

[kroll-j]

I think I finally found and fixed the problem. The Harfbuzz shaper goes through the font face set for each text item, looking for a single font face that can render all glyphs. In the case of a text item containing both hebrew and common scripts, there is no such face. It then falls through, rendering the text item with whatever font face happens to end up last in the list.
I've changed it slightly to collect faces for each glyph in a std::map, which are then used for rendering. The fork is here. This is the relevant commit.
This fixes the problem for me, the output looks correct now. I hope this is the right approach. Can somebody confirm?

[kroll-j]

(thinking about it, a simple array would probably have been the better choice for this, instead of a map...)

[nyurik]

Should this issue be closed, now that #3151 has been merged?

[artemp]

@nyurik - yes, closing.