Fix Extended_Pictographic in UnicodeProperty subclasses

[macchiati]

This is both in the JSPs and in the regular unicodetools.

There is a special hack for performance that works for most properties. Only one character in the unassigned range is tested for the property in some of the internals if the property is marked as hasUniformUnassigned() = true.

If the value is false, then every unassigned character is tested. It is the responsibility of each UnicodeProperty subclass to mark any exceptions. For example,

protected ICUProperty(String propName, int propEnum) {
      setName(propName);
      this.propEnum = propEnum;
      setType(internalGetPropertyType(propEnum));
      if (propEnum == UProperty.DEFAULT_IGNORABLE_CODE_POINT
              || propEnum == UProperty.BIDI_CLASS
              || propEnum == UProperty.BLOCK
              || propEnum == UProperty.EAST_ASIAN_WIDTH
              || propEnum == UProperty.LINE_BREAK
              || propEnum == UProperty.NONCHARACTER_CODE_POINT
              || propEnum == UProperty.PATTERN_SYNTAX
              || propEnum == UProperty.PATTERN_WHITE_SPACE
              || propEnum == UProperty.CHANGES_WHEN_CASEFOLDED
              || propEnum == UProperty.EMOJI
              || propEnum == UProperty.EMOJI_MODIFIER
              || propEnum == UProperty.EMOJI_MODIFIER_BASE
              || propEnum == UProperty.EMOJI_PRESENTATION
              ) {
        setUniformUnassigned(false);
      }
}

New properties like Extended_Pictographic need to be added to lists like this. The way to fix it is to search for setUniformUnassigned(false) and check that the lists are right.

NOTE: This performance hack was from a long time ago, and may not be needed anymore, but we should analyze the impact before removing.

[srl295]

what would be best is to have a test that verifies that the heuristic is correct… test all unassigned codepoints.

[macchiati]

I agree. Could be pretty simple.

1. Create a subclass of UnicodeProperty that wraps any other, but resets setUniformUnassigned(true).
2. Walk through all the properties that have hasUniformUnassigned() = true, and create a wrapped class.
3. For each such case, walk through all Unicode code points to verify that both instances have identical values.

[nedley]

Is this issue distinct from #53?

[macchiati]

53 is a duplicate (and has less information) Mark

…

On Mon, Jul 19, 2021 at 8:30 PM Ned Holbrook ***@***.***> wrote: Is this issue distinct from #53 <#53>? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#97 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMF3ZC3KCHAIEVD42YLTYTUUVANCNFSM5AQC7ZVQ> .

[srl295]

@macchiati done the quick fix, but need to do unit test also, so keeping this open

[macchiati]

sounds good, thanks! Mark

…

On Wed, Jul 21, 2021 at 10:25 AM Steven R. Loomis ***@***.***> wrote: @macchiati <https://github.com/macchiati> done the quick fix, but need to do unit test also, so keeping this open — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#97 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMEOJHJRQWQ5LJXC2H3TY37IHANCNFSM5AQC7ZVQ> .

[macchiati]

And thanks for jumping on it. Mark

…

On Wed, Jul 21, 2021 at 10:28 AM Mark Davis ☕️ ***@***.***> wrote: sounds good, thanks! Mark On Wed, Jul 21, 2021 at 10:25 AM Steven R. Loomis < ***@***.***> wrote: > @macchiati <https://github.com/macchiati> done the quick fix, but need > to do unit test also, so keeping this open > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#97 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACJLEMEOJHJRQWQ5LJXC2H3TY37IHANCNFSM5AQC7ZVQ> > . >