feat: Permit non-ASCII within <t> without the use of <u>

[cabo]

Permit non-ASCII within <t> without the use of <u>

See RSAB decision and RSWG discussion.

On Sep 19, 2022 the RPC raised the following policy question with the RSAB:

“The authors of RFC 9290 [1] would like to include a Hebrew string within <t> [2] without using <u> [3], which is currently not permitted by xml2rfc. This has been the case since the transition to v3 XML. The "XML and Style Guide Change Management Team” (CMT) previously discussed the possibility of allowing non-ASCII characters everywhere; I believe both CMT and RPAT are generally supportive but believe caution is needed because internationalization is complex. Limited use of non-ASCII characters has been de facto policy since the transition to v3 XML (3 years ago). In our current discussion, some believe that RFC 7997 [4] permits non-ASCII within <t> without the use of <u> so all that’s needed is a ticket to request that xml2rfc be updated, while some consider this a larger change that may require more review and perhaps RSWG action.

[1] https://www.rfc-editor.org/authors/rfc9290.html#appendix-A.3 <https://www.rfc-editor.org/authors/rfc9290.html#appendix-A.3>
Note that the string is currently included as <artwork>.
[2] https://authors.ietf.org/rfcxml-vocabulary#t <https://authors.ietf.org/rfcxml-vocabulary#t>
[3] https://authors.ietf.org/rfcxml-vocabulary#u <https://authors.ietf.org/rfcxml-vocabulary#u>
[4] https://www.rfc-editor.org/rfc/rfc7997.txt <https://www.rfc-editor.org/rfc/rfc7997.txt>”

Later on the RPC provided the following update:

“Note that the authors decided to publish RFC 9290 with a workaround because it was unclear how long their document would be delayed. While this issue no longer applies to a document pending action, we request a decision as we expect similar cases to come up in the near future. There are recent examples where this change would enhance the document, e.g., for μs, ≤, and ±”

After discussion, the RSAB concluded that section 3.1 of RFC 7997 describes the current policy that covers the case in question. That section permits non-ASCII within <t> without the use of <u> “[w]here the use of non-ASCII characters is purely part of an example and not otherwise required for correct protocol operation” [RFC7997]. The current practice by the RPC is manifested in tooling limitations that are more strict than the policy in RFC7997 and only allow the use of <u> for non-ASCII chars within <t>. Therefore, to align xml2rfc with RFC 7997, a proposed fix has been suggested in GitHub (#895 <https://github.com/ietf-tools/xml2rfc/pull/895>). In addition, the RPC has updated the “Web Portion of the Style Guide” to provide guidance about where non-ASCII characters may be used without <u> (see https://www.rfc-editor.org/styleguide/part2/#nonascii <https://www.rfc-editor.org/styleguide/part2/#nonascii>).

Further, the RSAB would like to note to the RSWG that RFC7991 and RFC7997 provide only limited guidance about the exact use of unicode characters. RFC7997 leaves it to the RPC to decide where <u> is required as part of the spec that is essential for correct operation. We believe this process is appropriate with the new RFC editor model; inline with section 4.4 of RFC9280 we recommended that the RPC consults with the responsible area director (in case of IETF stream documents) or the stream representatives, in those cases where the correct use is not clear. Also, if a further question on interpretation of policy appears, then the RPC is free to bring that back to RSAB. In addition, the RPC raised concerns about this change in the operational practice in order to align with RFC7997, e.g. due to complexities with internationalization, and therefore identified a need for potentially more implementation guidance. We recommended the RPC to raise these concerns with the RSWG, and if the community sees a need for additional policy and guidance, new policy should to be discussed and developed in the RSWG.

Original text on PR:
(See discussion in auth48archive and others.)

[rjsparks]

So far this will make it so that non-ascii utf-8 can appear in t without any warning being emitted. The warning/error changes would downshift the error to a warning for other places xml2rc was being strict about utf-8 for.

I'm not sure that's the intended result of the PR?

Unless we have some other tool to let the rfc-editor know they've got utf-8 to look at for reasonableness in the chain, having it pass it quietly seems a step too far.

[cabo]

The change only kicks in when the RFC has a number. Up to then, xml2rfc works like before.

[rjsparks]

and sometimes big swaths of text get added after the RFC has a number.

[cabo]

Yes, and having a tool that gives the RFC editor a histogram of character usage is probably quite useful. SMOP.

[cabo]

require 'unicode/name'
hist = Hash.new(0)
ARGF.read.each_char do |c|
  hist[c] += 1 unless c.ord < 128
end
cl = "*** Latin"
hist.keys.sort.partition {|c| c =~ /\A\p{Latin}\z/}.each do |l|
  puts cl
  l.each  do |c|
    puts "#{c}: #{"%4d" % hist[c]} #{Unicode::Name.correct(c)}"
  end
  cl = "*** Non-Latin"
end

$ unicode-histo rfc9277.xml
*** Latin
ß:    1 LATIN SMALL LETTER SHARP S
ä:    2 LATIN SMALL LETTER A WITH DIAERESIS
ö:    2 LATIN SMALL LETTER O WITH DIAERESIS
*** Non-Latin
 :   52 NO-BREAK SPACE
≤:    2 LESS-THAN OR EQUAL TO

[sginoza]

I need to think about this a bit more but 2 things come to mind:
a) slightly concerned that this isn't getting wider review in that this has been normal handling since the transition to v3 (so almost 3 years)
b) if we're making the switch to allow non-ascii everywhere, flagging if and where these characters appear would be helpful. I want to talk to the team about this to see what is helpful for all, but my first thought is it would be helpful to have some of the surrounding text or line numbers so we know where the characters appear (maybe some instances are ok and some aren't). We have a script we use now, but having something built into xml2rfc might be nice to have one less separate check to run.

[kesara]

@rjsparks, @cabo: I have committed a new change to allow bare Unicode in the <t> element.
This also introduces new flag --warn-bare-unicode when set, xml2rfc warns about bare Unicode in the <t> element.
By default, this is set to False.

Have a look at the following test results (in a different branch, I haven't merged the results to this PR yet): kesara@3998d8e#diff-c2d2944d758bb3948e7924d76edf0e05292cda738d52c51636723fbbd444cfa9

Are those test results acceptable?

[cabo]

Here is another example why this needs to be fixed:

ietf-wg-jsonpath/draft-ietf-jsonpath-base#274

[martinthomson]

Please add, both here and below: dd, dt, li, blockquote, and any other block-level elements I missed. Then add, both here and below, the "inline" elements: cref (?), em, eref (?), iref (?), relref, strong, sub, sup, tt, and xref. (I didn't think much about the cross reference stuff, but I think they can contain text.)

I have immediate use for many of these. I can wrap in <t> for some of the block-level elements, but not the inline-level ones.

[martinthomson]

Note that with a list this long, you might consider how the latter list (bare_unicode_tags_with_notice) can be merged into the former (unicode_content_tags).

[rjsparks]

I agree with the goal of the request. But I don't think we can do that without re-engaging the RSAB for a revised interpretation, or get a doc from RSWG through the RSAB that is explicit.

[duerst]

Unfortunately, I have to agree with @rjsparks here. RFC 7997 allows non-ASCII in examples, but not e.g. in Mathematical formulæ or other cases in general text. Of course I agree with @martinthomson that it would be highly desirable to use non-ASCII for these cases, but I think that requires a change in policy.

[reschke]

The change only kicks in when the RFC has a number. Up to then, xml2rfc works like before.

That sounds counter-intuitive to me. If it's an error until AUTH48, how will it be used by draft authors?

[cabo]

The change only kicks in when the RFC has a number. Up to then, xml2rfc works like before.

That sounds counter-intuitive to me. If it's an error until AUTH48, how will it be used by draft authors?

My original idea was to minimize the appearance of a change: use the workaround until RFC editing stage, and use the RFC editor's authority to decide what actually should be done.
However, that approach is no longer part of the proposal here.

[duerst]

@cabo wrote:

$ unicode-histo rfc9277.xml
*** Latin
ß:    1 LATIN SMALL LETTER SHARP S
ä:    2 LATIN SMALL LETTER A WITH DIAERESIS
ö:    2 LATIN SMALL LETTER O WITH DIAERESIS
*** Non-Latin
 :   52 NO-BREAK SPACE
≤:    2 LESS-THAN OR EQUAL TO

I think separating Latin and non-Latin is a good first step, but for the RPC, some other statistics may also be helpful:

• Counts per script (e.g. 4 HEBREW), not necessarily per character.
• Instances of non-normalized or confusable characters (implementation of this might take some effort).
• Instances of spaces (except for U+0020, the ASCII SPACE), marks,...

[cabo]

On 21. Dec 2022, at 21:17, Robert Sparks ***@***.***> wrote: @rjsparks commented on this pull request. In xml2rfc/util/unicode.py: > @@ -34,6 +34,7 @@ 'street', 'title', 'u', + 't', I agree with the goal of the request. But I don't think we can do that without re-engaging the RSAB for a revised interpretation, or get a doc from RSWG through the RSAB that is explicit.

I’m repeating myself here: This fixes a bug in xml2rfc. There is no policy change. (This is unfortunate, but we have to make these steps one after another.) Whether support for implementing the actual policy is added to xml2rfc or to some linter tool (or some diagnostic aid like the unicode-histo sketch I put in above) is orthogonal to fixing this bug. Grüße, Carsten

[rjsparks]

I’m repeating myself here: This fixes a bug in xml2rfc.

I don't disagree with you, but it's clear that the nascent RSWG/RSAB engine feels it has a role. I misremembered the status of this PR during the RSAB meeting at 116 (claiming it was already merged), and there were several people who were very upset about it.

So, yes, where Martin points is the right place to go. But I think trying to go there without nods from that new engine will make landing it take longer.