WIP: Refactor internationalization system #1148

BertalanD · 2020-08-28T11:29:29Z

This pull request implements the redesigned localization system as described in my forum post.

In summary, the goal is to be transparent towards templates, but customizable and explicit towards users. A Project Fluent-based string translation system will eventually be introduced that's similar to #1040 by @XAMPPRocky.

See the commit messages for an in-depth changelog.

To-do

Keats · 2020-08-31T19:20:51Z

@BertalanD I will have a look at that soon, I just want to get a 0.12 release first so this PR can be the main focus of 0.13

* Add line highlighting to code blocks * Fix highlighting of lines Apparently every line to be highlighted is provided in one chunk. * Add more documentation to codeblock.rs * Turn FenceIter into an Iterator * Move Range to fence.rs * Add tests

Better minify errors

* Allow site path to contain underscores Fixes site.css is not being generated if any part of the path contains underscores * Add tests for path with underscores

alerque · 2020-09-09T19:22:43Z

I'll definitely have my eyes pealed for 0.13 or when this PR hits master.

BertalanD · 2020-09-09T22:03:39Z

I've just been put in quarantine, so I'll start working on it tomorrow.

BertalanD · 2020-09-13T12:46:26Z

I'm finally done with the core functionality and have written up some user docs.

Updating the docs for the template API is a big task, so before that, I would like to ask for help in a couple of places:

LocalizedConfig and the copied standalone Tera contexts for each language seem awkward and slow for me. Does anybody have an idea how they might be replaced?
Are there any config values you would like to be overridable for each language, but aren't yet?
Doing {{ config.extra.whatever }} in templates seems awkward to me (especially since extra values come from translations, themes, and hopefully in the future, from the front matter, too). Any better ideas?

Lastly, I'd like @Keats' help about what tests to implement and finding beta testers.

Requesting a review.

Keats

Ok I started reviewing it but the PR is way too big to be properly reviewed on its own.
Can we split it in smaller PRs?

Keats · 2020-09-15T18:00:50Z

components/config/src/config/languages.rs

+    ///
+    /// If unset in a translation, it will fall back to how it's set for the default language.
+    /// Its default value for the primary language is `false`.
+    pub generate_feed: Option<bool>,


I would rather deprecate the top generate_feed in favour of feed than having generate_feed everywhere.

It would feel a bit ridiculous alongside build_search_index. How would you feel about also renaming build_search_index to searchable?

Keats · 2020-09-15T18:02:12Z

components/config/src/config/languages.rs

+    ///
+    /// If unset in a translation, it will fall back to how it's set for the default language.
+    /// Its default value for the default language is `false`.
+    pub build_search_index: Option<bool>,


Note: we should show a message if the language cannot be built into a search index, only a few languages are supported right now.

Keats · 2020-09-15T19:13:05Z

components/config/src/config/mod.rs

-    pub search: search::Search,
-
-    /// All user params set in [extra] in the config
-    pub extra: HashMap<String, Toml>,
 }


I don't know about that..
Things like compile_sass, minify_html etc are not language specific. Maybe pass something like language_config to the template instead?

Yep. I mainly want to have language-specific search because CJK search indices can get huge; and tight bandwidth might exclude things like feeds

components/config/src/config/mod.rs

Keats · 2020-09-15T19:24:23Z

components/config/src/config/mod.rs

+                lang: lang.clone(),
+                default_language_options,
+                languages,
+                ..self.clone()


I guess this means we clone the Config every time we render a template?

Unfortunately, yes. I'm thinking about having a struct with just references like SerializingTaxonomy and co. I'm way too inexperienced to not resort to cloning something with that many borrows.

Keats · 2020-09-15T19:32:57Z

components/library/src/content/page.rs

+                // TODO: add test
+                bail!("Page `{}` has an empty language. Did you mistakenly put 2 dots before the extension?", file_path.display());
+            } else if l == &config.default_language_options.language_alias {
+                bail!("Pages in the default language should not set it in their title (caused by `{}`).", file_path.display());


does that cause any issues?

Don't think so. The code currently does fail though if the default language is set in a file name.

if !config.languages_codes().contains(&parts[1].as_ref()) { bail!("File {:?} has a language code of {} which isn't present in the config.toml `languages`", self.path, parts[1]); }

Keats · 2020-09-15T19:40:32Z

components/library/src/content/page.rs

            if page.meta.date.is_none() {
-                page.meta.date = Some(caps.name("datetime").unwrap().as_str().to_string());
+                // TODO: do not fail silently on invalid formats


do we have a test for that?

Keats · 2020-09-15T19:41:46Z

components/library/src/content/section.rs

+        // Catch programming errors
+        // `lang` and `language_alias` must be non-empty
+        assert_ne!(section.lang, LanguageIdentifier::default());
+        assert!(!section.language_alias.is_empty());


looks like an almost copy/paste than the page.rs file

Keats · 2020-09-15T19:45:17Z

components/search/src/lib.rs

    let mut fields = vec![];
-    if config.search.include_title {
+    let search = config.0.default_language_options.search.clone().expect("search set");


can we ever have None?

Keats · 2020-09-15T19:45:36Z

components/search/src/lib.rs

-    };
+pub fn build_index(library: &Library, config: &LocalizedConfig) -> Result<String> {
+    let language = Language::from_code(config.0.lang.language.as_str()).ok_or_else(|| {
+        format!("Tried to build search index for language {} which is not supported", config.0.lang)


we can move that error at start time

BertalanD · 2020-09-15T20:01:47Z

I'm not sure how to split this up. The new LocaleOptions + LanguageIdentifier stuff takes up most of the change anyway, and dividing that into parts if not really an option, because almost all lines referencing Config would still need to be modified.
I'm open to leaving the Fluent/Docs part for later.

Sorry for the code spam.

BertalanD · 2020-09-15T20:35:31Z

components/rendering/tests/codeblock_hl_lines.rs

+    "#,
+        &context,
+    )
+    .unwrap();


Rustfmt really messed up this part. Sorry about that. I probably should revert changes to this file.

Keats · 2020-09-16T06:14:31Z

I put a comment on https://zola.discourse.group/t/rfc-internationalization-system-rework/546/7, let's think about the "what are we trying to achieve" first so you can do more incremental changes.

…1166) * Update compile_sass docs, this is just for sass files in site root * Improve file watch error, question whether the target exists

This commit starts the rework of the localization system, as described in my [forum post] a while ago. Its intent is to make it easier to eventually transition to a [Fluent]-based localization system and allow the official [Rust website] to move to Zola. Monolingual sites should continue working without change. If you use the current localization system, take a look at the summary at the end of the commit message. User documentation has not yet been created, so the key changes are described here as follows: - Require `config.default_language` and any additional `config.languages` to be valid Unicode language identifiers. These are compatible with HTML's [BCP 47] language codes and allow for correct language-dependent behavior (e.g. date formatting...) - Added `language_alias` field for `config` and all languages in `config.languages`, which sets how the language should be searched in file names and displayed in URLs. This lets us avoid breaking links to existing multilingual sites. - Added struct `LocaleOptions`, which contains language-specific settings. The title, description, taxonomies, feed limit, search options and `extra` keys are now also set here. - `LocaleOptions` belonging to the default language are in the field `default_language_options`. Its fields are transparently flattened, so `config.toml`'s syntax stays the same for monolingual sites. - Changed type of `config.languages` from `Vec<Language>` to `HashMap<LanguageIdentifier, LocaleOptions>. - Removed `config.translations`. Its functionality is taken over by `config.languages.{lang}.extra`. - Missing fields in the `extra` field of translations fall back to the values set for the default language. - Themes can also define translations similarly to `config.toml`. All fields except `extra` and `languages.{lang}.extra` are ignored. - Removed `lang` fields of taxonomies, as they are now to be defined under a language. - Added `LocalizedConfig` that wraps `Config` with `default_language_options` replaced with a translation's settings. TODO: investigate if a `SerializingConfig` should be used instead. `config.extra` in templates now contains the values for the specific language. Other langugages' `extra` is best accessed via the `trans` function. - Language and date parsing from filenames is done in `library::content::file_info`. The new `maybe_lang` and `date` fields are not guaranteed to be semantically correct. There is no need any more to externally mutate FileInfo when setting these parameters. - Reduced the distinction of "single-language mode". Page filenames are parsed as "{date}{_,-}{slug}.{lang}.md" regardless of the presence of translations. Section indices behave the same except dates aren't parsed. - Each language has its own Tera context. This requires cloning the config way too many times, so a more suitable and idiomatic solution should be sought. - Functions taking a `lang` argument use identifiers (not aliases!; like `lang` in context), and they fall back to the active language if not specified, not the site's default language. - Additional dependency: [unic_langid]. This is also used by Project Fluent, and thus by Mozilla, so it won't likely be abandoned in the future. For template authors: links constructed in templates and containing `lang` WILL break. URLs will always use the value that's in `language_alias`. You should replace `trans()` function calls with directly getting the value from `config.extra`, as it allows for data other than strings and will be deprecated in the future. For users: unless you have a multilingual site, everything will keep working as expected. Multilingual sites WILL stop working. The new `config.toml` layout is demonstrated in `test_site_i18n/config.toml` [forum post]: https://zola.discourse.group/t/rfc-internationalization-system-rework/546 [Rust website]: https://www.rust-lang.org/ [BCP 47]: https://tools.ietf.org/html/bcp47 [unic_langid]: https://crates.io/crates/unic-langid

- renamed `clear_filename` to `strip_filename_info` - removed unnecessary String allocations from `get_language` and `get_date`

Keats · 2021-01-09T13:56:38Z

sorry >_>

Keats · 2021-03-14T09:33:09Z

Closing in favour of #1391 as it addresses some existing issues and this PR would be too much work to rebase cleanly.

Thanks a lot @BertalanD !

alerque · 2021-04-08T04:15:47Z

This PR (and #1040) should be reconsidered. I saw in notifications that #1391 got merged and came back to check out the project and see if it would work yet for any of my sites. The answer seems to be no, the i18n work actually done there is very rudimentary. I don't mean this to be rude @Keats, but please consider that while you may not have a use case for a true multilingual site this is a pretty big deal — for many a make or break issue. Please consider trusting the people that are trying to contribute something more here. Having a Fluent based static site generator with multi language templating capabilities would be a big deal. Seeing multiple people try to contribute towards this and having their issues shut down is immensely discouraging. This is still up for grabs because no other static site generator does this right yet either, but as long as you keep shutting down PRs that attempt to address this I don't see how we're going to get anywhere.

BertalanD mentioned this pull request Aug 28, 2020

Add fluent documentation and add language fallback for trans XAMPPRocky/zola#3

Closed

Keats closed this Sep 4, 2020

Next version

8d9d8cf

Keats reopened this Sep 4, 2020

Keats mentioned this pull request Sep 6, 2020

[i18n] Please use Fluent for localization #1058

Open

sburris0 and others added 5 commits September 7, 2020 17:26

Add Void Linux installation instructions

1b6a06f

Add better errors for minifying and upgrade crate

d16dea7

Merge pull request getzola#1154 from areille/fix-minify-error

d26960d

Better minify errors

Allow site path to contain underscores (getzola#1162)

94aaeed

* Allow site path to contain underscores Fixes site.css is not being generated if any part of the path contains underscores * Add tests for path with underscores

BertalanD force-pushed the i18n-refactoring branch from 23a5cdf to ba5cecf Compare September 10, 2020 20:17

Fix a crash when --output_dir is used with serve (getzola#1164)

70c23bd

BertalanD marked this pull request as ready for review September 14, 2020 18:52

BertalanD force-pushed the i18n-refactoring branch from 39e5a0c to 632f587 Compare September 15, 2020 18:49

Keats requested changes Sep 15, 2020

View reviewed changes

BertalanD commented Sep 15, 2020

View reviewed changes

timvisee and others added 8 commits September 16, 2020 12:55

Fix getzola#1165, improve situation for compile_sass errors (getzola#…

66281ac

…1166) * Update compile_sass docs, this is just for sass files in site root * Improve file watch error, question whether the target exists

Add doc comments to crate components/config

6b23d61

Add fluent function

9600f67

Fix Clippy warnings, remove unnecessary .clone()s

acf49fe

Fix benches

d640e53

Update user-facing documentation for localization rework

5073c03

Simplify duplicate language detection

af2f06c

Tweak filename handling

7a90859

- renamed `clear_filename` to `strip_filename_info` - removed unnecessary String allocations from `get_language` and `get_date`

BertalanD force-pushed the i18n-refactoring branch from 9a60078 to 7a90859 Compare September 18, 2020 15:44

Keats force-pushed the next branch from 0f12b42 to b313c97 Compare September 22, 2020 07:46

Keats force-pushed the next branch from ff1ee3d to cc3e4f9 Compare December 14, 2020 19:45

Keats closed this Jan 9, 2021

Keats deleted the branch getzola:next January 9, 2021 13:51

Keats reopened this Jan 9, 2021

Keats closed this Mar 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Refactor internationalization system #1148

WIP: Refactor internationalization system #1148

BertalanD commented Aug 28, 2020 •

edited

Loading

Keats commented Aug 31, 2020

alerque commented Sep 9, 2020

BertalanD commented Sep 9, 2020

BertalanD commented Sep 13, 2020 •

edited

Loading

Keats left a comment

Keats Sep 15, 2020

BertalanD Sep 15, 2020

Keats Sep 15, 2020

Keats Sep 15, 2020

BertalanD Sep 15, 2020

Keats Sep 15, 2020

BertalanD Sep 15, 2020

Keats Sep 15, 2020

BertalanD Sep 15, 2020

Keats Sep 15, 2020

Keats Sep 15, 2020

Keats Sep 15, 2020

Keats Sep 15, 2020

BertalanD commented Sep 15, 2020

BertalanD Sep 15, 2020 •

edited

Loading

Keats commented Sep 16, 2020

Keats commented Jan 9, 2021

Keats commented Mar 14, 2021

alerque commented Apr 8, 2021

WIP: Refactor internationalization system #1148

WIP: Refactor internationalization system #1148

Conversation

BertalanD commented Aug 28, 2020 • edited Loading

To-do

Keats commented Aug 31, 2020

alerque commented Sep 9, 2020

BertalanD commented Sep 9, 2020

BertalanD commented Sep 13, 2020 • edited Loading

Keats left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BertalanD commented Sep 15, 2020

BertalanD Sep 15, 2020 • edited Loading

Choose a reason for hiding this comment

Keats commented Sep 16, 2020

Keats commented Jan 9, 2021

Keats commented Mar 14, 2021

alerque commented Apr 8, 2021

BertalanD commented Aug 28, 2020 •

edited

Loading

BertalanD commented Sep 13, 2020 •

edited

Loading

BertalanD Sep 15, 2020 •

edited

Loading