First draft of some registry functions #420

mihnita · 2023-07-12T20:53:18Z

No description provided.

mihnita · 2023-07-12T20:56:15Z

The .md needs updating, but I though I should do that only after we finish polishing the .xml.

macchiati · 2023-07-12T21:10:30Z

Some comments from a very quick scan:

  <option name="minimumFractionDigits" values="0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20"/>

The number could be a BigDecimal, and you don't want to list all of those possible values! Maybe just \d+?

        Match a numerical value against CLDR plural categories or against a number literal.

should be something like:

Match a formatted numerical value against CLDR plural categories or against a number literal.

That is, I'm looking for some way we can make it harder (or at least alert people to )make the following mistake:

match {$count}
when one {You have a book}
when * {You have {$count :number maximumFractionDigits=0} books}

That is, when $count = 1.1, this fails, with *You have 1 books

mihnita · 2023-07-12T22:08:41Z

Thank you!

The number could be a BigDecimal, and you don't want to list all of those possible values! Maybe just \d+?

The info comes from ECMAScript, which limits the possible values:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat/NumberFormat#minimumfractiondigits

The intention was to start with something that is common between ICU and JS.

spec/registry.xml

aphillips · 2023-07-12T22:41:07Z

spec/registry.xml

+      <!-- The time zone to use. The only value implementations must recognize
+         is "UTC"; the default is the runtime's default time zone.
+         Implementations may also recognize the time zone names of the IANA
+         time zone database, such as "Asia/Shanghai", "Asia/Kolkata",
+         "America/New_York".
+      -->
+      <option name="timeZone" pattern="timeZoneId"/>


Don't forget offset time zones.

It is 100% what ECMA says:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/DateTimeFormat/DateTimeFormat#timezone

I tried to not include anything that is not currently supported by ECMAScript, as we know there is strong opposition to that.

I'm for it, happy to discuss, but I didn't change this xml.

Added a topic in #422

This is what the function accepts as validated input. It doesn't mean that every possible value is supported

spec/registry.xml

aphillips · 2023-07-12T22:45:39Z

spec/registry.xml

+        the default for currency formatting is the number of minor unit digits provided by
+        the ISO 4217 currency code list (2 if the list doesn't provide that information).


... unless there is an override that says otherwise, e.g. some curriences with the cash format (e.g. technically COP has two digits but practically it has none)

Agree. But this is 100% ECMAScript quote.

aphillips · 2023-07-12T22:46:04Z

spec/registry.xml

+        A value with a smaller number of integer digits than this number will be
+        left-padded with zeros (to the specified length) when formatted.
+      -->
+      <option name="minimumIntegerDigits" values="1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21" default="1"/>


do we really need a list?

Created issue #422 for this (and more)

aphillips · 2023-07-12T22:47:39Z

spec/registry.xml

+      <option name="maximumSignificantDigits" values="1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21" default="21"/>
+
+      <match pattern="anyNumber"/>
+      <match values="zero one two few many"/>


does * need to appear here? Some locales produce other as output from plural, which is effectively * in our syntax. That is, do we want to be able to say "this selector can request the default message"?

I think our spec makes the presence of * mandatory.
So I didn't include other for plural and the other selectors.

Of course, open for discussion.

I think it is an interesting minor issue for the registry spec.

I think our spec makes the presence of * mandatory. So I didn't include other for plural and the other selectors.

Why omit other if the spec only requires *?

My thinking (why I excluded it): because they mean the same thing?
If I see other in the registry, I expect to also see other handled in the when

Option: we add it here as a possible value, we also add default=other, and update the registry spec to say that the default value for selectors maps to *

I think I like that option.

This assumption is unfortunately false, likely on accident in CLDR, although I miss the context. For Polish, other is specifically used for fractions, while many is what you'd expect to be a good other in most other languages.

macchiati · 2023-07-13T00:44:49Z

Thank you!

The number could be a BigDecimal, and you don't want to list all of those possible values! Maybe just \d+?

The info comes from ECMAScript, which limits the possible values: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat/NumberFormat#minimumfractiondigits

The intention was to start with something that is common between ICU and JS.

There is, however, no need to have that limitation here, and artificially limit more powerful implementations.

Instead, what I think we should do is document that if the value exceeds what the implementation is capable of, the maximum supported by the implementation is used.

eemeli

This looks like a much bigger subset of the JS Intl.NumberFormat & Intl.DateTimeFormat formatters' options than I expected, which is great! Besides the line comments, I would find it easier to consider exactly which parts are being left out: Could a list of such options with their reasonings be provided?

spec/registry.dtd

spec/registry.xml

mihnita · 2023-07-13T15:45:04Z

There is, however, no need to have that limitation here, and artificially limit more powerful implementations.

Instead, what I think we should do is document that if the value exceeds what the implementation is capable of, the maximum supported by the implementation is used.

Some of the group members strongly opposed including anything that is not supported by ECMAScript.
So I tried to start as non-controversial as possible.

I anticipate (and hope) that we will discuss PR in the next meeting.

mihnita · 2023-07-13T16:24:12Z

Match a formatted numerical value against CLDR plural categories or against a number literal.

Updated registry.md and the registry.xml

That is, I'm looking for some way we can make it harder (or at least alert people to )make the following mistake:

match {$count}
when one {You have a book}
when * {You have {$count :number maximumFractionDigits=0} books}

That is, when $count = 1.1, this fails, with *You have 1 books

ACK, and agree.

I don't think we can capture this in the registry.xml, at least not with the current registry dtd.
Should be easy to implement as a lint.

Added it as a discussion point in #422

mihnita · 2023-07-13T16:35:41Z

Reverted most changes to the .dtd, except for the ones that were clearly unintentional, and prevented a valid .xml.

For example:

<!ELEMENT registry (function*|pattern*)>

This means that the registry can contain (0 or more function) OR (0 or more pattern)
But function AND pattern is illegal, it is either / or.

This makes the examples in the .md invalid, and the registry can't cover the use cases we need.

spec/registry.xml

macchiati · 2023-07-16T23:21:11Z

So this is ECMAScriptMessageFormatting only? More seriously, if the restriction is removed, it simply allows clients to pass in more digits of accuracy; doesn't mean that the implementations have to support that in formatting.

…

On Thu, Jul 13, 2023 at 8:45 AM Mihai Nita ***@***.***> wrote: There is, however, no need to have that limitation here, and artificially limit more powerful implementations. Instead, what I think we should do is document that if the value exceeds what the implementation is capable of, the maximum supported by the implementation is used. Some of the group members strongly opposed including anything that is not supported by ECMAScript. So I tried to start as non-controversial as possible. I anticipate (and hope) that we will discuss PR in the next meeting. — Reply to this email directly, view it on GitHub <#420 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMCIGHGOMNQULQ7KN23XQAJYXANCNFSM6AAAAAA2IAO7QA> . You are receiving this because you commented.Message ID: ***@***.***>

Co-authored-by: Addison Phillips <addisonI18N@gmail.com>

spec/registry.xml

ryzokuken

This is great! Thanks for kicking this off.

ryzokuken · 2023-07-20T12:22:54Z

spec/registry.md

@@ -90,7 +90,7 @@ For the sake of brevity, only `locales="en"` is considered.
    <function name="number">
        <description>
            Format a number.
-            Match a numerical value against CLDR plural categories or against a number literal.
+            Match a **formatted** numerical value against CLDR plural categories or against a number literal.


Does the imply that the input is a localized number that'd be parsed and then matched? Also since this function includes both selection and formatting, should this also say something about the formatting part?

No. I've merged a suggestion from Addison which does not mention formatting.

There is an ongoing discussion in issue #425
Especially this comment #425 (comment)

ryzokuken · 2023-07-20T12:32:13Z

spec/registry.xml

+  <pattern id="anyNumber" regex="-?[0-9]+(\.[0-9]+)"/>
+  <pattern id="positiveInteger" regex="[0-9]+"/>
+  <pattern id="currencyCode" regex="[A-Z]{3}"/>
+  <pattern id="timeZoneId" regex="[a-zA-Z/]+"/>


We could use a more restricted grammar that only allows valid IANA timezone IDs, but I don't feel very strongly about it. Exhibit A: https://tc39.es/proposal-temporal/#prod-TimeZoneIANAName

The current registry spec does not allow us to do this, only regex and list of values.

I suggested changing the spec to allow for URLs to external specs.
Added it as a discussion point in #422

The Temporal polyfill includes a Regexp that works for this purpose, but unfortunately it isn't a literal but I imagine that if we agreed on having a restrictive regex here that only allows valid IANA tzids, then we could come up with a simplified regex that works in this case.

I think that in the longer run (not first commit) we should also add support for Unicode Time Zone Identifiers

The main reason would be stability, which the Olson database IDs does not offer (see linked)

spec/registry.xml

ryzokuken · 2023-07-20T12:45:46Z

spec/registry.xml

+      <!-- The minimum number of significant digits to use. -->
+      <option name="minimumSignificantDigits" values="1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21" default="1"/>
+      <!-- The maximum number of significant digits to use. -->
+      <option name="maximumSignificantDigits" values="1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21" default="21"/>


For all of these *Digits <option>s, I wonder if we could add a convenient shorthand in the DTD... Perhaps a simple <range start="x" end="y"/> definition could work?

Changed to positiveInteger.
But I think the range option might also be handy in other cases.

Co-authored-by: Eemeli Aro <eemeli@gmail.com>

spec/registry.xml

Co-authored-by: Ujjwal Sharma <ryzokuken@disroot.org>

mihnita · 2023-07-20T15:00:25Z

There is, however, no need to have that limitation here, and artificially limit more powerful implementations.

More seriously, if the restriction is removed, it simply allows clients to pass in more digits of accuracy; doesn't mean that the implementations have to support that in formatting.

Between several people commenting on this, and the news that ECMAScript considers removing the restriction, I changed this to positiveInteger. I hope it will not be controversial :-)

Co-authored-by: Eemeli Aro <eemeli@gmail.com>

eemeli · 2023-07-20T15:33:41Z

Between several people commenting on this, and the news that ECMAScript considers removing the restriction, I changed this to positiveInteger. I hope it will not be controversial :-)

The JS limit isn't being removed; it's being increased from 20 to 100. Attempting to format a number with a minimumFractionDigits value greater than that will still throw a RangeError. As long as we consider that to be acceptable behaviour for a comforming implementation, I'm fine with this option accepting any positive integer value.

@mihnita

…, matchSignature A spin-off from unicode-org#420, in which @mihnita noticed that `registry.dtd` uses alternatives in a wrong way. For example, `<!ELEMENT formatSignature (input?|option*)>` means that `formatSignature` is allowed to have as children either: at most one `input` OR any number of `options`, but not both. Instead, @mihnita suggested using sequences: `<!ELEMENT formatSignature (input?,option*)>`, which this PR does. Note that sequences require the children to appear in a specific order, which isn't something that's useful to us. However, I'm not aware of any way of lifting this requirement that also allows to enforce at most on `input` or exactly one `description`.

stasm · 2023-07-23T21:31:34Z

spec/registry.dtd

@@ -1,6 +1,6 @@
-<!ELEMENT registry (function*|pattern*)>
+<!ELEMENT registry (function,pattern)*>


Thanks for noticing this. I filed #434 to discuss and fix this independently of this PR, so that you can focus it on registry.xml.

eemeli

This looks like a decent starting point for us.

I'd prefer the DTD changes to be handled separately, but I'm fine with them being included here and then iterated on in other PRs.

spec/registry.xml

eemeli · 2023-07-24T12:04:38Z

spec/registry.xml

+      <!-- The formatting style to use. -->
+      <option name="style" values="decimal currency percent unit" default="decimal"/>


To consider: If we were to leave out currency and unit formatting from the default, then we wouldn't need to say how compound units need to work.

Co-authored-by: Eemeli Aro <eemeli@gmail.com>

@mihnita

…, matchSignature (#434) A spin-off from #420, in which @mihnita noticed that `registry.dtd` uses alternatives in a wrong way. For example, `<!ELEMENT formatSignature (input?|option*)>` means that `formatSignature` is allowed to have as children either: at most one `input` OR any number of `options`, but not both. Instead, @mihnita suggested using sequences: `<!ELEMENT formatSignature (input?,option*)>`, which this PR does. Note that sequences require the children to appear in a specific order, which isn't something that's useful to us. However, I'm not aware of any way of lifting this requirement that also allows to enforce at most on `input` or exactly one `description`. Co-authored-by: Addison Phillips <addison@unicode.org>

First draft of some registry functions

8e7d994

mihnita requested review from stasm, aphillips, eemeli and echeran July 12, 2023 20:53

aphillips reviewed Jul 12, 2023

View reviewed changes

spec/registry.xml Outdated Show resolved Hide resolved

aphillips reviewed Jul 12, 2023

View reviewed changes

spec/registry.xml Outdated Show resolved Hide resolved

aphillips reviewed Jul 12, 2023

View reviewed changes

spec/registry.xml Show resolved Hide resolved

aphillips reviewed Jul 12, 2023

View reviewed changes

spec/registry.xml Outdated Show resolved Hide resolved

aphillips reviewed Jul 12, 2023

View reviewed changes

eemeli requested changes Jul 13, 2023

View reviewed changes

mihnita mentioned this pull request Jul 13, 2023

Friction points when adding the first functions to the registry #422

Closed

mihnita added 2 commits July 13, 2023 09:27

Implemented a first round of feedback

ba26ac6

OK to have a function without input

196d004

A function can have 0 or 1 input

9a45c14

aphillips reviewed Jul 13, 2023

View reviewed changes

spec/registry.xml Outdated Show resolved Hide resolved

eemeli reviewed Jul 14, 2023

View reviewed changes

spec/registry.xml Outdated Show resolved Hide resolved

eemeli mentioned this pull request Jul 15, 2023

Selection method expectations #425

Closed

eemeli mentioned this pull request Jul 17, 2023

Establish literal parsing guidance for custom function options #427

Closed

Update spec/registry.xml

8452064

Co-authored-by: Addison Phillips <addisonI18N@gmail.com>

eemeli reviewed Jul 18, 2023

View reviewed changes

spec/registry.xml Outdated Show resolved Hide resolved

ryzokuken approved these changes Jul 20, 2023

View reviewed changes

mihnita and others added 2 commits July 20, 2023 07:52

Changed long lists of numeric values with positiveInteger

a33d0b1

Update positiveInteger regexp

ff6723d

Co-authored-by: Eemeli Aro <eemeli@gmail.com>

ryzokuken reviewed Jul 20, 2023

View reviewed changes

spec/registry.xml Outdated Show resolved Hide resolved

mihnita and others added 2 commits July 20, 2023 07:56

Update regex for iso8601 (although still not enough)

05f50f3

Co-authored-by: Ujjwal Sharma <ryzokuken@disroot.org>

Fixed double positiveInteger

7463489

Co-authored-by: Ujjwal Sharma <ryzokuken@disroot.org>

Better regex for anyNumber (although still not enough)

7c1ebd7

Co-authored-by: Eemeli Aro <eemeli@gmail.com>

stasm mentioned this pull request Jul 23, 2023

Fix registry.dtd to not use alternatives in function, formatSignature, matchSignature #434

Merged

stasm reviewed Jul 23, 2023

View reviewed changes

stasm mentioned this pull request Jul 23, 2023

(registry.dtd) Rename pattern to validationRule #435

Merged

Remove any references to selectors, for now

d98b642

eemeli approved these changes Jul 24, 2023

View reviewed changes

Update regex for anyNumber

5d5b9f3

Co-authored-by: Eemeli Aro <eemeli@gmail.com>

aphillips merged commit 32c2467 into unicode-org:main Jul 24, 2023

ryzokuken mentioned this pull request Aug 22, 2023

Add some capabilities for the builtin functions #457

Merged

eemeli mentioned this pull request Dec 3, 2023

Include input & option descriptions as data in registry #533

Merged

eemeli mentioned this pull request Jan 15, 2024

Review all registry.xml descriptions #596

Closed

mihnita deleted the mihnita_registry branch February 14, 2024 21:27

		the default for currency formatting is the number of minor unit digits provided by
		the ISO 4217 currency code list (2 if the list doesn't provide that information).

		@@ -1,6 +1,6 @@
		<!ELEMENT registry (function\|pattern)>
		<!ELEMENT registry (function,pattern)*>

		<!-- The formatting style to use. -->
		<option name="style" values="decimal currency percent unit" default="decimal"/>

First draft of some registry functions #420

First draft of some registry functions #420

Conversation

mihnita commented Jul 12, 2023

mihnita commented Jul 12, 2023

macchiati commented Jul 12, 2023

mihnita commented Jul 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mihnita Jul 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mihnita Jul 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mihnita Jul 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

macchiati commented Jul 13, 2023

eemeli left a comment

Choose a reason for hiding this comment

mihnita commented Jul 13, 2023

mihnita commented Jul 13, 2023

mihnita commented Jul 13, 2023

macchiati commented Jul 16, 2023 via email

ryzokuken left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mihnita commented Jul 20, 2023 • edited Loading

eemeli commented Jul 20, 2023

Choose a reason for hiding this comment

eemeli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mihnita Jul 13, 2023 •

edited

Loading

mihnita Jul 13, 2023 •

edited

Loading

mihnita Jul 24, 2023 •

edited

Loading

mihnita commented Jul 20, 2023 •

edited

Loading