diagnostics: translation infrastructure #95512

davidtwco · 2022-03-31T12:21:42Z

An implementation of the infrastructure required to have translatable diagnostic messages.

Introduces a DiagnosticMessage type which can represent both the current non-translatable messages and identifiers for Fluent.
Modifies current diagnostic API so that existing calls still work but DiagnosticMessages can be provided too.
Adds support for always loading a "fallback bundle" containing the English diagnostic messages, which are used when a DiagnosticMessage::FluentIdentifier is used in a diagnostic being emitted.
Adds support for loading a "primary bundle" which contains the user's preferred language translation, and is used preferentially when it contains a diagnostic message being emitted. Primary bundles are loaded either from the path provided to -Ztranslate-alternate-ftl (for testing), or from the sysroot at $sysroot/locale/$locale/*.ftl given a locale with -Ztranslate-lang (which is parsed as a language identifier).
Adds "diagnostic args" which enable normally-interpolated variables to be made available as variables for Fluent messages to use.
Updates #[derive(SessionDiagnostic)] so that it can only be used for translatable diagnostics and update the handful of diagnostics which used the derive to be translatable.

For example, the following diagnostic...

#[derive(SessionDiagnostic)]
#[error = "E0195"]
pub struct LifetimesOrBoundsMismatchOnTrait {
    #[message = "lifetime parameters or bounds on {item_kind} `{ident}` do not match the trait declaration"]
    #[label = "lifetimes do not match {item_kind} in trait"]
    pub span: Span,
    #[label = "lifetimes in impl do not match this {item_kind} in trait"]
    pub generics_span: Option<Span>,
    pub item_kind: &'static str,
    pub ident: Ident,
}

...becomes...

#[derive(SessionDiagnostic)]
#[error(code = "E0195", slug = "typeck-lifetimes-or-bounds-mismatch-on-trait")]
pub struct LifetimesOrBoundsMismatchOnTrait {
    #[primary_span]
    #[label]
    pub span: Span,
    #[label = "generics-label"]
    pub generics_span: Option<Span>,
    pub item_kind: &'static str,
    pub ident: Ident,
}

typeck-lifetimes-or-bounds-mismatch-on-trait =
    lifetime parameters or bounds on {$item_kind} `{$ident}` do not match the trait declaration
    .label = lifetimes do not match {$item_kind} in trait
    .generics-label = lifetimes in impl do not match this {$item_kind} in trait

r? @estebank
cc @oli-obk @Manishearth

rust-highfive · 2022-03-31T12:21:46Z

Some changes occurred in src/tools/clippy.

cc @rust-lang/clippy

Some changes occurred in src/tools/rustfmt.

cc @calebcartwright

davidtwco · 2022-04-01T02:44:59Z

I'll wait for a review and feedback before fixing any CI failures, just in case any larger changes are required.

Manishearth

Overall looks quite good from an i18n standpoint! For the fluent stuff we probably want to support directories of ftl files, not just single large ftl files.

You can see what the website does here

compiler/rustc_error_messages/src/lib.rs

Manishearth · 2022-04-01T22:50:07Z

compiler/rustc_errors/src/diagnostic.rs

+/// Simplified version of `FluentArg` that can implement `Encodable` and `Decodable`. Collection of
+/// `DiagnosticArg` are converted to `FluentArgs` (consuming the collection) at the start of
+/// diagnostic emission.
+pub type DiagnosticArg<'source> = (Cow<'source, str>, DiagnosticArgValue<'source>);


nit: probably worth making this a struct (tuple or otherwise)

I'm deliberately using a tuple here because the FromIterator impl on FluentArgs works with this representation.

compiler/rustc_error_messages/src/lib.rs

compiler/rustc_macros/src/session_diagnostic.rs

davidtwco · 2022-04-02T01:51:15Z

For the fluent stuff we probably want to support directories of ftl files, not just single large ftl files.

This is how I've implemented the user-requested language support, it loads every resource (.ftl file) in the $sysroot/share/locale/$locale/ directory. I haven't done this with the fallback bundle just because I was using include_str!, but there's no reason we couldn't just add more include_str!s for each new .ftl file we want in the default bundle.

compiler/rustc_error_messages/src/lib.rs

oli-obk · 2022-04-02T12:13:04Z

r? @oli-obk

Taking over from Esteban to keep his queue from growing. Will sync separately with him on it

src/tools/tidy/src/deps.rs

oli-obk · 2022-04-03T09:42:55Z

@bors r+ p=1 bitrotty

bors · 2022-04-03T09:42:56Z

📌 Commit bb04e494e2c237e5195b33fbdf4db1afb8ffb878 has been approved by oli-obk

bors · 2022-04-03T10:33:55Z

⌛ Testing commit bb04e494e2c237e5195b33fbdf4db1afb8ffb878 with merge 4c269a7b4d2715f1379c500ff51ffc6f4aa6dd08...

bors · 2022-04-05T06:02:05Z

📌 Commit ccd4820 has been approved by oli-obk

bjorn3 · 2022-04-05T08:27:34Z

This is how I've implemented the user-requested language support, it loads every resource (.ftl file) in the $sysroot/share/locale/$locale/ directory.

Does it fallback to the default sysroot if --sysroot is passed and this sysroot doesn't have translations?

davidtwco · 2022-04-05T08:57:25Z

This is how I've implemented the user-requested language support, it loads every resource (.ftl file) in the $sysroot/share/locale/$locale/ directory.

Does it fallback to the default sysroot if --sysroot is passed and this sysroot doesn't have translations?

It will only fail when it looks for a locale-specific directory in the sysroot and it isn't there, i.e. when the $user_sysroot/share/locale/$locale directory doesn't exist.

If that directory exists but doesn't contain any Fluent resources (or even anything that all), then it won't fail, it'll just always fallback to the default locale. If that directory contains Fluent resources which are empty, then it also won't fail, it'll just fallback to the default locale. In both of these cases we construct a FluentBundle for that locale, it just ends up being empty and that's the same as if it had every message except for one and we wanted to fallback to the English default.

I don't do anything different if it's a --sysroot from the user or the default.

bjorn3 · 2022-04-05T09:17:34Z

I don't do anything different if it's a --sysroot from the user or the default.

You might want to. Otherwise using xargo and other similar programs would cause translations to break as they don't add translations from the default sysroot to the created sysroot.

Rollup of 5 pull requests Successful merges: - rust-lang#95234 (bootstrap.py: nixos check in /etc/os-release with quotes) - rust-lang#95449 (Fix `x doc --stage 0 compiler`) - rust-lang#95512 (diagnostics: translation infrastructure) - rust-lang#95607 (Note invariance reason for FnDef types) - rust-lang#95645 (Fix intra doc link ICE when trying to get traits in scope for primitive) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup

bors · 2022-04-05T12:11:40Z

⌛ Testing commit ccd4820 with merge 634770c...

…ation, r=oli-obk" This reverts commit d473024, reversing changes made to 73eab35.

…t-only-warn, r=oli-obk sess: warn w/out fluent bundle w/ user sysroot Addresses rust-lang#95512 (comment). When a custom sysroot is requested, then don't error when translation resources are not found, only warn. r? `@bjorn3`

The documentation comment for this derive is out-of-date, it should have been updated in rust-lang#95512. Signed-off-by: David Wood <david.wood@huawei.com>

…nor-changes, r=oli-obk errors: minor translation-related changes - For one error in typeck, specifying "suggestion" as the attribute for the only suggestion is unnecessary, it's the default of the derive. - The documentation comment for the `SessionDiagnostic` derive is out-of-date, it should have been updated in rust-lang#95512. r? `@oli-obk`

est31 · 2022-07-09T17:19:58Z

Can the approach be changed to maybe auto-generate the ftl file or something? Because I really like the experience of grepping for error messages and then directly getting to the code that creates the error. Now I'd have to figure out a fluent identifier, then I'd have to replace the -s with _...

est31 · 2022-07-09T17:23:44Z

Furthermore, it also makes contributor experience worse because you now have to edit one more file to add a new error/lint/etc.

Sorry for posting to this pull request but I couldn't find any issue/rfc/etc linked, and wouldn't know how to get the attention of / reach the stakeholders otherwise.

est31 · 2022-07-09T19:08:22Z

I've found something that looks like it could be the "canonical" issue for this: rust-lang/community-localization#4

It also looks like that initial plan included keeping the english error strings inside the rust source code.

davidtwco · 2022-07-11T07:59:05Z

@est31 Let's chat about this in #t-compiler/wg-diagnostics :)

Manishearth · 2022-07-11T15:31:27Z

Yeah, that was the initial plan, but tbh it's getting more and more rare to do in-source l10n strings. Sure it makes grepping a bit annoying but it really just means you have to grep twice instead of once.

rust-highfive assigned estebank Mar 31, 2022

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Mar 31, 2022

rustbot added the T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. label Mar 31, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 31, 2022

davidtwco force-pushed the diagnostic-translation branch from 59753da to 5386089 Compare April 1, 2022 02:39

davidtwco marked this pull request as ready for review April 1, 2022 02:39

This comment has been minimized.

Sign in to view

Manishearth approved these changes Apr 1, 2022

View reviewed changes

davidtwco commented Apr 2, 2022

View reviewed changes

compiler/rustc_error_messages/src/lib.rs Outdated Show resolved Hide resolved

rust-highfive assigned oli-obk and unassigned estebank Apr 2, 2022

This comment has been minimized.

Sign in to view

klensy reviewed Apr 3, 2022

View reviewed changes

src/tools/tidy/src/deps.rs Outdated Show resolved Hide resolved

davidtwco force-pushed the diagnostic-translation branch from b530be6 to 87654ac Compare April 3, 2022 07:46

This comment was marked as resolved.

Sign in to view

davidtwco force-pushed the diagnostic-translation branch from 87654ac to bb04e49 Compare April 3, 2022 08:54

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 3, 2022

This comment was marked as outdated.

Sign in to view

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 3, 2022

davidtwco force-pushed the diagnostic-translation branch from bb04e49 to 7d942f8 Compare April 3, 2022 11:18

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 5, 2022

Dylan-DPC mentioned this pull request Apr 5, 2022

Rollup of 5 pull requests #95667

Merged

bors merged commit d473024 into rust-lang:master Apr 5, 2022

rustbot added this to the 1.62.0 milestone Apr 5, 2022

davidtwco deleted the diagnostic-translation branch April 5, 2022 12:12

pietroalbini mentioned this pull request Apr 5, 2022

Fx breakage caused by the introduction of diagnostic translations racer-rust/racer#1180

Merged

davidtwco mentioned this pull request Apr 6, 2022

sess: warn w/out fluent bundle w/ user sysroot #95716

Merged

rylev mentioned this pull request Apr 8, 2022

[PERF] Revert "Rollup merge of #95512" #95811

Closed

rylev added a commit to rylev/rust that referenced this pull request Apr 8, 2022

Revert "Rollup merge of rust-lang#95512 - davidtwco:diagnostic-transl…

82fbecd

…ation, r=oli-obk" This reverts commit d473024, reversing changes made to 73eab35.

light4 mentioned this pull request Apr 11, 2022

ICE: rustc -Ztranslate-lang=de #95934

Closed

davidtwco mentioned this pull request Apr 21, 2022

errors: minor translation-related changes #96269

Merged

bjorn3 mentioned this pull request Jul 19, 2022

Handle multiline strings Rust-GCC/gccrs#1399

Open

davidtwco added the A-translation Area: Translation infrastructure, and migrating existing diagnostics to SessionDiagnostic label Aug 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

diagnostics: translation infrastructure #95512

diagnostics: translation infrastructure #95512

davidtwco commented Mar 31, 2022

rust-highfive commented Mar 31, 2022

This comment has been minimized.

davidtwco commented Apr 1, 2022

Manishearth left a comment

Manishearth Apr 1, 2022

davidtwco Apr 3, 2022

davidtwco commented Apr 2, 2022

oli-obk commented Apr 2, 2022

This comment has been minimized.

This comment was marked as resolved.

oli-obk commented Apr 3, 2022

bors commented Apr 3, 2022

bors commented Apr 3, 2022

This comment was marked as outdated.

bors commented Apr 5, 2022

bjorn3 commented Apr 5, 2022

davidtwco commented Apr 5, 2022 •

edited

Loading

bjorn3 commented Apr 5, 2022

bors commented Apr 5, 2022

est31 commented Jul 9, 2022

est31 commented Jul 9, 2022 •

edited

Loading

est31 commented Jul 9, 2022

davidtwco commented Jul 11, 2022

Manishearth commented Jul 11, 2022

diagnostics: translation infrastructure #95512

diagnostics: translation infrastructure #95512

Conversation

davidtwco commented Mar 31, 2022

rust-highfive commented Mar 31, 2022

This comment has been minimized.

davidtwco commented Apr 1, 2022

Manishearth left a comment

Choose a reason for hiding this comment

Manishearth Apr 1, 2022

Choose a reason for hiding this comment

davidtwco Apr 3, 2022

Choose a reason for hiding this comment

davidtwco commented Apr 2, 2022

oli-obk commented Apr 2, 2022

This comment has been minimized.

This comment was marked as resolved.

oli-obk commented Apr 3, 2022

bors commented Apr 3, 2022

bors commented Apr 3, 2022

This comment was marked as outdated.

bors commented Apr 5, 2022

bjorn3 commented Apr 5, 2022

davidtwco commented Apr 5, 2022 • edited Loading

bjorn3 commented Apr 5, 2022

bors commented Apr 5, 2022

est31 commented Jul 9, 2022

est31 commented Jul 9, 2022 • edited Loading

est31 commented Jul 9, 2022

davidtwco commented Jul 11, 2022

Manishearth commented Jul 11, 2022

davidtwco commented Apr 5, 2022 •

edited

Loading

est31 commented Jul 9, 2022 •

edited

Loading