Adopt RFC 7405 (Case-Sensitive String Support in ABNF) for grammar

[duerst]

The grammar at https://github.com/unicode-org/message-format-wg/blob/main/spec/syntax.md currently uses case-insensitive ABNF, but MF2 is defined as case-sensitive.

I propose that (once the syntax is a bit more settled), we switch to RFC 7405 (Case-Sensitive String Support in ABNF). That should make the syntax a bit more readable.

[aphillips]

This seems like a good idea.

[stasm]

We briefly discussed this when we introduced the ABNF, and back then we felt that requiring RFC 7405 just for 3 keywords wasn't worth it, just in case some tools for working with ABNF didn't support the extension.

• Define the grammar as an ABNF (RFC 5234) #347 (comment)
• Define the grammar as an ABNF (RFC 5234) #347 (comment)

[eemeli]

On the other hand, if we consider the ABNF's primary audience to be human readers, then making it case-sensitive would improve its legibility.

Especially as we already assume with the *-expression rules that an implementation will first transform the ABNF to a shape that doesn't need rewinding.

[stasm]

Does the current ABNF suffer from not being readable? Here it is in its current form:

; reserved keywords are always lowercase
input = %x69.6E.70.75.74  ; "input"
local = %x6C.6F.63.61.6C  ; "local"
match = %x6D.61.74.63.68  ; "match"
when  = %x77.68.65.6E     ; "when"

; used as:
input-declaration = input [s] variable-expression
local-declaration = local s variable [s] "=" [s] expression
selectors         = match 1*([s] expression)
variant           = when 1*(s key) [s] quoted-pattern

[macchiati]

IMO it is fundamentally unreadable when humans depend on comments to be able to understand the ABNF, comments which could be misaligned with what the 'code' says. The format in https://www.w3.org/TR/REC-xml/#sec-notation is far more understandable for modern readers (and not only in how it handles literal characters).

…

On Fri, Nov 3, 2023 at 9:04 AM Stanisław Małolepszy < ***@***.***> wrote: Does the current ABNF suffer from not being readable? Here it is in its current form: ; reserved keywords are always lowercaseinput = %x69.6E.70.75.74 ; "input"local = %x6C.6F.63.61.6C ; "local"match = %x6D.61.74.63.68 ; "match"when = %x77.68.65.6E ; "when" ; used as:input-declaration = input [s] variable-expressionlocal-declaration = local s variable [s] "=" [s] expressionselectors = match 1*([s] expression)variant = when 1*(s key) [s] quoted-pattern — Reply to this email directly, view it on GitHub <#501 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMBOOAWOIOZRXG365CLYCUIYVAVCNFSM6AAAAAA6NLIDZGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJSG4YDSMJZGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[aphillips]

We're not changing to EBNF now. Using 7405 to allow plain quotes for the keywords would be nice, provided that's the only impact.

IMO it is fundamentally unreadable when humans depend on comments to be
able to understand the ABNF, comments which could be misaligned with what
the 'code' says.

Yeah, but the comments are not misaligned nor would we allow them to be. Since we want case-sensitive (required lowercase) keywords, using 7405 would simplify our ABNF to be clearer:

input-declaration = "input" [s] variable-expression
local-declaration = "local" s variable [s] "=" [s] expression
selectors         = "match" 1*([s] expression)
variant           = "when" 1*(s key) [s] quoted-pattern

Alternatively we can just call out the case sensitivity aspect or provide a classical ABNF "appendix" (it's just the four lines you quote, @stasm) for tool users who don't have 7405.

[macchiati]

Sorry, didn't mean to imply we need to change to EBNF now; that should have been marked as an aside. I firmly agree on 7405 being much better than the current ABNF.

…

On Sat, Nov 4, 2023 at 11:09 AM Addison Phillips ***@***.***> wrote: We're not changing to EBNF now. Using 7405 to allow plain quotes for the keywords would be nice, provided that's the only impact. IMO it is fundamentally unreadable when humans depend on comments to be able to understand the ABNF, comments which could be misaligned with what the 'code' says. Yeah, but the comments are not misaligned nor would we allow them to be. Since we want case-sensitive (required lowercase) keywords, using 7405 would simplify our ABNF to be clearer: input-declaration = "input" [s] variable-expressionlocal-declaration = "local" s variable [s] "=" [s] expressionselectors = "match" 1*([s] expression)variant = "when" 1*(s key) [s] quoted-pattern Alternatively we can just call out the case sensitivity aspect or provide a classical ABNF "appendix" (it's just the four lines you quote, @stasm <https://github.com/stasm>) for tool users who don't have 7405. — Reply to this email directly, view it on GitHub <#501 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMEYT4RHJLUMU4U2WKTYC2AHFAVCNFSM6AAAAAA6NLIDZGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJTGUYTAOJRGI> . You are receiving this because you commented.Message ID: ***@***.***>

[stasm]

I don't have strong feelings about this issue. If others find 7405 much more readable, let's do it.

I'll just note that, as I was originally working on #347, I researched a number of tools and libraries that could help me validate, parse or otherwise work with the ABNF. Of those that I tested, 3 supported 7405 and 2 didn't. It was the main reason why I didn't feel pulling 7405 in was worth it for a handful keywords.

Tools with 7405 support

• https://author-tools.ietf.org/abnf - An online parser.
• https://pypi.org/project/abnf/ - A parser written in Python.
• http://www.2p.cz/en/abnf_gen - A tool generating example text matching the ABNF grammar, good for fuzzing.

Tools without 7405 support

• https://www.bottlecaps.de/convert/ - Conversion tool from ABNF to W3C EBNF, which I used to create railroad diagrams.
• http://abnf.msweet.org/index.php - ABNF to regex, useful for generating valid productions and then using them to test custom parsers.

[aphillips]

@stasm I don't disagree. I think that publishing a short appendix (four lines of ABNF!) to convert 7405 to 5234 would satisfy consumers who are using one of the other tools. If we decide to make the change, in fact, we should include it in the PR. Let's discuss in the call today (2023-11-06)

[aphillips]

accepted in 2023-11-06

[aphillips]

#514