Propose reserving additional sigils for future use #374

aphillips · 2023-04-10T13:33:40Z

This proposal proposes to reserve sigils for future use with reserved sigils allowed in the same place that function names are currently, e.g. standalone ({%foo}), with options ({%foo key=val}), or subsidiary to variable/literals ({|quote me| %foo} or {$var %foo}).

A case could be made for reserving a comment syntax that allows a placeholder with prose in it (and I would suggest using the % for it). @eemeli's proposal is effectively this. I didn't propose this because of the impact on parsers in the future. Let's discuss.

Following our decisions in the 2023-04-10 call, updating to make the reserved sigils introduce expressions containing undifferentiated text. Note that I reused the text construct, which requires { and } to be escaped (for the benefit of parsers). I also corrected the list of reserved sigils. I left out a few potential sigils, specifically U+0060, (, ), _, and =.

@eemeli

This proposal proposes to reserve sigils for future use with reserved sigils allowed in the same place that function names are currently, e.g. standalone (`{%foo}`), with options (`{%foo key=val}`), or subsidiary to variable/literals (`{|quote me| %foo}` or `{$var %foo}`). A case could be made for reserving a comment syntax that allows a placeholder with prose in it (and I would suggest using the `%` for it). @eemeli's proposal is effectively this. I didn't propose this because of the impact on parsers in the future. Let's discuss.

eemeli

@aphillips Did I understand right from the call that you'd be updating this PR?

aphillips · 2023-04-15T08:00:38Z

@eemeli That's correct.

aphillips · 2023-04-15T08:34:04Z

Note: this PR is still missing the work on the spec prose.

spec/message.abnf

I think this works. I'm going to commit it to the PR and then do some additional cleanup. Co-authored-by: Eemeli Aro <eemeli@mozilla.com>

- align some items - move `function` with `variable` - move `reserve-escape` with the other escapes

Note that this change also removed the undefined term "operand" in favor of just using literal and variable (which are defined terms). It also imposes 2119 keywords.

eemeli

Looks good! I think we'll want to add a section on reserved to the syntax text, but I like the general shape of this.

spec/syntax.md

spec/message.abnf

eemeli

A few requested changes documented inline. Furthermore, the error handling & fallback string representations in formatting.md should be updated as a part of this change.

eemeli · 2023-04-17T16:21:49Z

spec/syntax.md

@@ -108,7 +109,7 @@ let hello = new MessageFormat('{Hello, world!}')
 hello.format()
 ```

-### Expressions
+### Placeholder Expressions


As discussed at our last meeting, an expression is also used for variable definitions and for selectors; not just placeholders. I think changing that terminology should be beyond the scope of this PR.

I made the links work in the spec doc by expanding the term here. Note that the term "placeholder" doesn't appear in the ABNF and is not currently defined anywhere (else). The syntax of an expression includes the {/} markers and so I concluded that it was reasonable to say that all expressions are placeholders. The text recognizes that placeholder expressions can appear in variable definitions and selectors.

The easier way to fix this would be to use singular instead of plural and I'll do that for now.

eemeli · 2023-04-17T16:24:42Z

spec/syntax.md

@@ -345,20 +346,17 @@ Whitespace within a _pattern_ is meaningful and MUST be preserved.

 ### Expressions

-**_Expressions_** can either start with an operand or a function call.
+_Expressions_ ***must*** start with a _literal_, a _variable_, or an _annotation_, or consist of a _reserved_ string. An _expression_ ***must not*** be empty.


Thus far the spec markdown files have used Semantic Line Breaks, which make it easier to track changes within paragraphs. Would it be possible to continue doing so? These line breaks have no effect on the rendered spec.

Also, this sentence is a little bit wrong now that reserved may follow an argument.

I will touch up the formatting.

I missed correcting this when making the change. I mean, technically, it's correct, but the reserved sequence is just part of an annotation :-).

spec/syntax.md

eemeli · 2023-04-17T16:44:03Z

spec/syntax.md

@@ -495,6 +497,8 @@ Inside _patterns_,
 whitespace is part of the translatable content and is recorded and stored verbatim.
 Whitespace is not significant outside translatable text, except where required by the syntax.

+***NOTE:*** Whitespace **_is_** significant in the `reserved` production and implementations need to be careful not to trim trailing whitespace from reserved sequences.


As an alternative solution, we could also tie our future hands a little, and explicitly make the trailing whitespace of reserved not significant. Realistically, I don't think that would cause any problems, especially as we're already considering trailing whitespace non-significant in other expressions.

We can do that, but we would need to change the ABNF. The problem is that all of our other productions are bounded in some way and the exterior whitespace is applied by the optional production [s]. Because reserved sequences are not delimited, there is no way to know where the whitespace belongs to the reserved or to the [s]. Future standardization would most likely use productions like name or nmtoken and the resulting trailing whitespace would "emerge" from the murky fog of reserved to be captured by [s]. Note that this is one of the reasons I originally proposed reserving the sigils to the function-like syntax.

I tend to agree with wanting to make the exterior whitespace not significant, in part because it interferes with user expectations. These two sequences are logically identical:

{ $foo :bar } {$foo :bar}

But these two are not:

{ $foo ~bar } {$foo ~bar}

I think we should fix this. Probably:

reserved = reserved-start reserved-body reserved-end reserved-end = ; reserved-body less SP / HTAB/ CR / LF

Why do we need to distinguish the trailing spaces of reserved in the ABNF? Would it not be enough to include some statement like this in the text on reserved about future use?

Future uses of reserved expressions MUST NOT assign any significance to trailing whitespace.

We should make the parser return the right tokens. See what you think of the revised abnf.

macchiati · 2023-04-17T17:04:41Z

BTW, big fan of Semantic Line Breaks

…format-wg#374)

aphillips requested review from stasm, eemeli, echeran and mihnita April 10, 2023 13:33

eemeli reviewed Apr 13, 2023

View reviewed changes

Merge branch 'main' into aphillips-issue-360

b64094f

aphillips requested a review from eemeli April 15, 2023 08:25

eemeli requested changes Apr 15, 2023

View reviewed changes

spec/message.abnf Outdated Show resolved Hide resolved

aphillips and others added 3 commits April 15, 2023 12:25

Update spec/message.abnf

0e7454a

I think this works. I'm going to commit it to the PR and then do some additional cleanup. Co-authored-by: Eemeli Aro <eemeli@mozilla.com>

Cleanup

60dc2ee

- align some items - move `function` with `variable` - move `reserve-escape` with the other escapes

Fix typo

aef7a27

aphillips requested a review from eemeli April 15, 2023 10:32

Add reserved to the spec

4a73158

Note that this change also removed the undefined term "operand" in favor of just using literal and variable (which are defined terms). It also imposes 2119 keywords.

eemeli approved these changes Apr 15, 2023

View reviewed changes

spec/syntax.md Outdated Show resolved Hide resolved

eemeli reviewed Apr 15, 2023

View reviewed changes

spec/message.abnf Outdated Show resolved Hide resolved

aphillips added 2 commits April 16, 2023 12:13

allow passing values to reserved

a8b8b48

Add warning about whitespace significance in reserved

0df07f4

aphillips requested a review from eemeli April 16, 2023 10:28

Fix expression links, add more about reserved

a3dcef4

eemeli requested changes Apr 17, 2023

View reviewed changes

eemeli mentioned this pull request Apr 17, 2023

Should one or two reserved sigils be available for implementation use? #378

Closed

eemeli reviewed Apr 17, 2023

View reviewed changes

aphillips added 3 commits April 18, 2023 10:37

Address comments

da35503

Make spaces insignificant at the end of reserved sequences

0d61ad4

Remove the warning about trailing whitespace on reserved

510f872

aphillips requested a review from eemeli April 18, 2023 09:07

Fix reserved to ensure no ending whitespace

27bf944

aphillips merged commit b3b9feb into main Apr 24, 2023

aphillips mentioned this pull request Apr 24, 2023

Reserve sigals or syntax for future expansion #360

Closed

eemeli added a commit to messageformat/messageformat that referenced this pull request Jun 6, 2023

feat(mf2): Support parsing reserved expressions (unicode-org/message-…

5a47c70

…format-wg#374)

eemeli mentioned this pull request Jun 6, 2023

Update MF2 implementation to match upstream messageformat/messageformat#398

Merged

4 tasks

eemeli added a commit to messageformat/messageformat that referenced this pull request Jun 6, 2023

feat(mf2): Support parsing reserved expressions (unicode-org/message-…

68f4066

…format-wg#374)

This was referenced Jul 3, 2023

Compatibility strategy (backward, forward), data model extensions. #222

Closed

Fix reserved-body to use quoted rather than literal #415

Merged

aphillips deleted the aphillips-issue-360 branch July 24, 2023 15:28

This was referenced Jul 31, 2023

Clarify that Reserved may also represent private-use in data model #444

Merged

Reconsider using text for private-use and reserved #446

Closed

This was referenced Nov 28, 2023

Implement code more introducer #529

Merged

Update the rest of the spec to match the ABNF after adding .keywords #548

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Propose reserving additional sigils for future use #374

Propose reserving additional sigils for future use #374

aphillips commented Apr 10, 2023 •

edited

Loading

eemeli left a comment

aphillips commented Apr 15, 2023

aphillips commented Apr 15, 2023

eemeli left a comment

eemeli left a comment

eemeli Apr 17, 2023

aphillips Apr 18, 2023

eemeli Apr 17, 2023

aphillips Apr 18, 2023

eemeli Apr 17, 2023

aphillips Apr 18, 2023

eemeli Apr 18, 2023

aphillips Apr 18, 2023

macchiati commented Apr 17, 2023

Propose reserving additional sigils for future use #374

Propose reserving additional sigils for future use #374

Conversation

aphillips commented Apr 10, 2023 • edited Loading

eemeli left a comment

Choose a reason for hiding this comment

aphillips commented Apr 15, 2023

aphillips commented Apr 15, 2023

eemeli left a comment

Choose a reason for hiding this comment

eemeli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

macchiati commented Apr 17, 2023

aphillips commented Apr 10, 2023 •

edited

Loading