Skip to content

Commit

Permalink
feat(v2): core v2 i18n support + Docusaurus site Crowdin integration (#…
Browse files Browse the repository at this point in the history
…3325)

* docs i18n initial poc

* docs i18n initial poc

* docs i18n initial poc

* docs i18n initial poc

* crowdin-v2 attempt

* fix source

* use crowdin env variable

* try to install crowdin on netlify

* try to install crowdin on netlify

* try to use crowdin jar directly

* try to curl the crowdin jar

* add java version cmd

* try to run crowdin jar in netlify

* fix translatedDocsDirPath

* fix loadContext issue due to site baseUrl not being modified in generted config file

* real validateLocalesFile

* add locale option to deploy command

* better LocalizationFile type

* create util getPluginI18nPath

* better core localization context loading code

* More explicit VersionMetadata type for localized docs folders

* Ability to translate blog posts with Crowdin!

* blog: refactor markdown loader + report broken links + try to get linkify working better

* upgrade crowdin config to upload all docs folder files except source code related files

* try to support translated pages

* make markdown pages translation work

* add write-translations cli command template

* fix site not  reloaded with correct options

* refactor a bit the read/write of @generated/i18n.json file

* Add <Translate> + translate() API + use it on the docusaurus homepage

* watch locale translation dir

* early POC of adding babel parsing for translation extraction

* fs.stat => pathExists

* add install:fast script

* TSC: noUnusedLocals false as it's already checked  by eslint

* POC of extracting translations from source code

* minor typo

* fix extracted key to code

* initial docs extracted translations

* stable plugin translations POC

* add crowdin commands

* quickfix for i18n deployment

* POC  of themeConfig translation

* add ability to have localized site without path prefix

* sidebar typo

* refactor translation system to output multiple translation files

* translate properly  the docs plugin

* improve theme classic translation

* rework translation extractor to handle new Chrome I18n JSON format (include id/description)

* writeTranslations: allow to pass locales cli arg

* fix ThemeConfig TS issues

* fix localizePath errors

* temporary add write-translations to netlify deploy preview

* complete example of french translated folder

* update fr folder

* remove all translations from repo

* minor translation  refactors

* fix all docs-related tests

* fix blog feed tests

* fix last blog tests

* refactor i18n context a bit, extract codeTranslations in an extra generated file

* improve @generated/i18n type

* fix some i18n todos

* minor refactor

* fix logo typing issue after merge

* move i18n.json to siteConfig instead

* try to fix windows CI build

* fix config test

* attempt to fix windows non-posix path

* increase v1 minify css jest timeout due to flaky test

* proper support for localizePath on windows

* remove non-functional install:fast

* docs, fix docsDirPathLocalized

* fix Docs i18n / md linkify issues

* ensure theme-classic swizzling will use "nextjs" sources (transpiled less aggressively, to make them human readable)

* fix some snapshots

* improve themeConfig translation code

* refactor a bit getPluginI18nPath

* readTranslationFileContent => ensure files are valid, fail fast

* fix versions tests

* add extractSourceCodeAstTranslations comments/resource links

* ignore eslint: packages/docusaurus-theme-classic/lib-next/

* fix windows CI with cross-env

* crowdin ignore .DS_Store

* improve writeTranslations + add exhaustive tests for translations.ts

* remove typo

* Wire currentLocale to algolia search

* improve i18n locale error

* Add tests for translationsExtractor.ts

* better code translation extraction regarding statically evaluable code

* fix typo

* fix typo

* improve theme-classic transpilation

* refactor  +  add i18n tests

* typo

* test new utils

* add missing snapshots

* fix snapshot

* blog onBrokenMarkdownLink

* add sidebars tests

* theme-classic index should now use ES modules

* tests for theme-classic translations

* useless comment

* add more translation tests

* simplify/cleanup writeTranslations

* try to fix Netlify fr deployment

* blog: test translated md is used during feed generation

* blog: better i18n tests regarding editUrl + md translation application

* more i18n tests for docs plugin

* more i18n tests for docs plugin

* Add tests for pages i18n

* polish docusaurus build i18n logs
  • Loading branch information
slorber authored Nov 26, 2020
1 parent 85fe96d commit 3166fab
Show file tree
Hide file tree
Showing 107 changed files with 5,465 additions and 667 deletions.
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ packages/docusaurus-plugin-ideal-image/lib/
packages/docusaurus-plugin-ideal-image/copyUntypedFiles.js
packages/docusaurus-theme-common/lib/
packages/docusaurus-theme-classic/lib/
packages/docusaurus-theme-classic/lib-next/
packages/docusaurus-theme-bootstrap/lib/
packages/docusaurus-migrate/lib/

Expand Down
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,16 @@ packages/docusaurus-plugin-sitemap/lib/
packages/docusaurus-plugin-ideal-image/lib/
packages/docusaurus-theme-common/lib/
packages/docusaurus-theme-classic/lib/
packages/docusaurus-theme-classic/lib-next/
packages/docusaurus-theme-bootstrap/lib/
packages/docusaurus-migrate/lib/

website/netlifyDeployPreview
website/netlifyDeployPreview/*
!website/netlifyDeployPreview/index.html
!website/netlifyDeployPreview/_redirects

website-1.x-migrated

website/i18n/**/*
#!website/i18n/fr
#!website/i18n/fr/**/*
155 changes: 155 additions & 0 deletions crowdin-v2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
#
# Your Crowdin credentials
#
'project_id': '416738'
'api_token_env': 'CROWDIN_PERSONAL_TOKEN'
'base_path': '.'
'base_url': 'https://api.crowdin.com'

#
# Choose file structure in Crowdin
# e.g. true or false
#
'preserve_hierarchy': true

#
# Files configuration
#
files:
[
{
'source': '/website/i18n/en/**/*',
'translation': '/website/i18n/%two_letters_code%/**/%original_file_name%',
'ignore': ['/**/.DS_Store'],
},
{
'source': '/website/docs/**/*',
'translation': '/website/i18n/%two_letters_code%/docusaurus-plugin-content-docs/current/**/%original_file_name%',
'ignore': ['/**/.DS_Store'],
},
{
'source': '/website/community/**/*',
'translation': '/website/i18n/%two_letters_code%/docusaurus-plugin-content-docs-community/current/**/%original_file_name%',
'ignore': ['/**/.DS_Store'],
},
{
'source': '/website/versioned_docs/**/*',
'translation': '/website/i18n/%two_letters_code%/docusaurus-plugin-content-docs/**/%original_file_name%',
'ignore': ['/**/.DS_Store'],
},
{
'source': '/website-1.x/blog/**/*',
'translation': '/website/i18n/%two_letters_code%/docusaurus-plugin-content-blog/**/%original_file_name%',
'ignore': ['/**/.DS_Store'],
},
{
'source': '/website/src/pages/**/*',
'translation': '/website/i18n/%two_letters_code%/docusaurus-plugin-content-pages/**/%original_file_name%',
'ignore':
[
'/**/*.js',
'/**/*.jsx',
'/**/*.ts',
'/**/*.tsx',
'/**/*.css',
'/**/.DS_Store',
],
},
]
#
# Source files filter
# e.g. "/resources/en/*.json"
#
#"source" : "/website/docs/**/*.md",
#
# Where translations will be placed
# e.g. "/resources/docs/%two_letters_code%/%original_file_name%"
#
#"translation" : "/website/i18n/%language%/docs/current/%original_file_name%",
#
# Files or directories for ignore
# e.g. ["/**/?.txt", "/**/[0-9].txt", "/**/*\?*.txt"]
#
#"ignore" : [],
#
# The dest allows you to specify a file name in Crowdin
# e.g. "/messages.json"
#
#"dest" : "",
#
# File type
# e.g. "json"
#
#"type" : "",
#
# The parameter "update_option" is optional. If it is not set, after the files update the translations for changed strings will be removed. Use to fix typos and for minor changes in the source strings
# e.g. "update_as_unapproved" or "update_without_changes"
#
#"update_option" : "",
#
# Start block (for XML only)
#
#
# Defines whether to translate tags attributes.
# e.g. 0 or 1 (Default is 1)
#
# "translate_attributes" : 1,
#
# Defines whether to translate texts placed inside the tags.
# e.g. 0 or 1 (Default is 1)
#
# "translate_content" : 1,
#
# This is an array of strings, where each item is the XPaths to DOM element that should be imported
# e.g. ["/content/text", "/content/text[@value]"]
#
# "translatable_elements" : [],
#
# Defines whether to split long texts into smaller text segments
# e.g. 0 or 1 (Default is 1)
#
# "content_segmentation" : 1,
#
# End block (for XML only)
#
#
# Start .properties block
#
#
# Defines whether single quote should be escaped by another single quote or backslash in exported translations
# e.g. 0 or 1 or 2 or 3 (Default is 3)
# 0 - do not escape single quote;
# 1 - escape single quote by another single quote;
# 2 - escape single quote by backslash;
# 3 - escape single quote by another single quote only in strings containing variables ( {0} ).
#
# "escape_quotes" : 3,
#
# Defines whether any special characters (=, :, ! and #) should be escaped by backslash in exported translations.
# e.g. 0 or 1 (Default is 0)
# 0 - do not escape special characters
# 1 - escape special characters by a backslash
#
# "escape_special_characters": 0
#
#
# End .properties block
#
#
# Often software projects have custom names for the directories where translations are placed. crowdin-cli allows you to map your own languages to be understandable by Crowdin.
#
#"languages_mapping" : {
# "two_letters_code" : {
# "crowdin_language_code" : "local_name"
# }
#},
#
# Does the first line contain header?
# e.g. true or false
#
#"first_line_contains_header" : true,
#
# for spreadsheets
# e.g. "identifier,source_phrase,context,uk,ru,fr"
#
# "scheme" : "",
11 changes: 5 additions & 6 deletions examples/bootstrap/docs/doc1.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ To serve as an example page when styling markdown based Docusaurus sites.

## Emphasis

Emphasis, aka italics, with *asterisks* or _underscores_.
Emphasis, aka italics, with _asterisks_ or _underscores_.

Strong emphasis, aka bold, with **asterisks** or __underscores__.
Strong emphasis, aka bold, with **asterisks** or **underscores**.

Combined emphasis with **asterisks and _underscores_**.

Expand All @@ -48,11 +48,11 @@ Strikethrough uses two tildes. ~~Scratch this.~~
1. Ordered sub-list
1. And another item.

* Unordered list can use asterisks
- Unordered list can use asterisks

- Or minuses
* Or minuses

+ Or pluses
- Or pluses

---

Expand Down Expand Up @@ -92,7 +92,6 @@ Images from any folder can be used by providing path to file. Path should be rel

![img](../static/img/logo.svg)


---

## Code
Expand Down
10 changes: 5 additions & 5 deletions examples/classic/docs/doc1.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ To serve as an example page when styling markdown based Docusaurus sites.

## Emphasis

Emphasis, aka italics, with *asterisks* or _underscores_.
Emphasis, aka italics, with _asterisks_ or _underscores_.

Strong emphasis, aka bold, with **asterisks** or __underscores__.
Strong emphasis, aka bold, with **asterisks** or **underscores**.

Combined emphasis with **asterisks and _underscores_**.

Expand All @@ -48,11 +48,11 @@ Strikethrough uses two tildes. ~~Scratch this.~~
1. Ordered sub-list
1. And another item.

* Unordered list can use asterisks
- Unordered list can use asterisks

- Or minuses
* Or minuses

+ Or pluses
- Or pluses

---

Expand Down
10 changes: 5 additions & 5 deletions examples/facebook/docs/doc1.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ To serve as an example page when styling markdown based Docusaurus sites.

## Emphasis

Emphasis, aka italics, with *asterisks* or _underscores_.
Emphasis, aka italics, with _asterisks_ or _underscores_.

Strong emphasis, aka bold, with **asterisks** or __underscores__.
Strong emphasis, aka bold, with **asterisks** or **underscores**.

Combined emphasis with **asterisks and _underscores_**.

Expand All @@ -48,11 +48,11 @@ Strikethrough uses two tildes. ~~Scratch this.~~
1. Ordered sub-list
1. And another item.

* Unordered list can use asterisks
- Unordered list can use asterisks

- Or minuses
* Or minuses

+ Or pluses
- Or pluses

---

Expand Down
2 changes: 2 additions & 0 deletions jest.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ const ignorePatterns = [
'/packages/docusaurus-plugin-content-blog/lib',
'/packages/docusaurus-plugin-content-docs/lib',
'/packages/docusaurus-plugin-content-pages/lib',
'/packages/docusaurus-theme-classic/lib',
'/packages/docusaurus-theme-classic/lib-next',
'/packages/docusaurus-migrate/lib',
];

Expand Down
5 changes: 5 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@
"serve:v2:ssl:gencert": "openssl req -x509 -nodes -days 365 -newkey rsa:4096 -subj \"/C=US/ST=Docusaurus/L=Anywhere/O=Dis/CN=localhost\" -keyout ./website/.docusaurus/selfsigned.key -out ./website/.docusaurus/selfsigned.crt",
"serve:v2:ssl:message": "echo '\n\n\nServing Docusaurus with HTTPS on localhost requires to disable the Chrome security: chrome://flags/#allow-insecure-localhost\n\n\n'",
"serve:v2:ssl:serve": "serve website/build --ssl-cert ./website/.docusaurus/selfsigned.crt --ssl-key ./website/.docusaurus/selfsigned.key",
"crowdin:upload:v2": "crowdin upload sources --config ./crowdin-v2.yaml",
"crowdin:uploadTranslations:v2": "crowdin upload translations --config ./crowdin-v2.yaml",
"crowdin:download:v2": "crowdin download --config ./crowdin-v2.yaml",
"changelog": "lerna-changelog",
"postinstall": "yarn lock:update && yarn build:packages",
"prettier": "prettier --config .prettierrc --write \"**/*.{js,ts}\"",
Expand Down Expand Up @@ -60,6 +63,7 @@
"@babel/core": "^7.12.3",
"@babel/plugin-proposal-nullish-coalescing-operator": "^7.12.1",
"@babel/plugin-proposal-optional-chaining": "^7.12.1",
"@babel/plugin-transform-modules-commonjs": "^7.12.1",
"@babel/preset-typescript": "^7.12.1",
"@types/express": "^4.17.2",
"@types/fs-extra": "^9.0.4",
Expand Down Expand Up @@ -92,6 +96,7 @@
"@typescript-eslint/parser": "^4.8.0",
"babel-eslint": "^10.0.3",
"concurrently": "^5.2.0",
"cross-env": "^7.0.2",
"enzyme": "^3.10.0",
"enzyme-adapter-react-16": "^1.15.1",
"eslint": "^7.13.0",
Expand Down
2 changes: 1 addition & 1 deletion packages/docusaurus-1.x/lib/server/__tests__/utils.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ describe('server utils', () => {
expect(css).toMatchSnapshot();

await expect(utils.minifyCss(notCss)).rejects.toMatchSnapshot();
});
}, 10000);

test('autoprefix css', async () => {
const testCss = fs.readFileSync(
Expand Down
26 changes: 26 additions & 0 deletions packages/docusaurus-module-type-aliases/src/index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,20 @@ declare module '@generated/globalData' {
export default globalData;
}

declare module '@generated/i18n' {
const i18n: {
defaultLocale: string;
locales: [string, ...string[]];
currentLocale: string;
};
export default i18n;
}

declare module '@generated/codeTranslations' {
const codeTranslations: Record<string, string>;
export default codeTranslations;
}

declare module '@theme/*';

declare module '@theme-original/*';
Expand All @@ -69,6 +83,18 @@ declare module '@docusaurus/Link' {
export default Link;
}

declare module '@docusaurus/Translate' {
type TranslateProps = {children: string; id?: string; description?: string};
const Translate: (props: TranslateProps) => JSX.Element;
export default Translate;

export function translate(param: {
message: string;
id?: string;
description?: string;
}): string;
}

declare module '@docusaurus/router' {
export const Redirect: (props: {to: string}) => import('react').Component;
export function matchPath(
Expand Down
4 changes: 2 additions & 2 deletions packages/docusaurus-plugin-client-redirects/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import writeRedirectFiles, {
toRedirectFilesMetadata,
RedirectFileMetadata,
} from './writeRedirectFiles';
import {removePrefix} from '@docusaurus/utils';
import {removePrefix, addLeadingSlash} from '@docusaurus/utils';

export default function pluginClientRedirectsPages(
_context: LoadContext,
Expand All @@ -27,7 +27,7 @@ export default function pluginClientRedirectsPages(
async postBuild(props: Props) {
const pluginContext: PluginContext = {
relativeRoutesPaths: props.routesPaths.map(
(path) => `/${removePrefix(path, props.baseUrl)}`,
(path) => `${addLeadingSlash(removePrefix(path, props.baseUrl))}`,
),
baseUrl: props.baseUrl,
outDir: props.outDir,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
title: This post links to another one!
---

[Good link 1](2018-12-14-Happy-First-Birthday-Slash.md)

[Good link 2](./2018-12-14-Happy-First-Birthday-Slash.md)

[Bad link 1](postNotExist1.md)

[Bad link 1](./postNotExist2.mdx)
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
title: Happy 1st Birthday Slash!
---

pattern name
Happy birthday!
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
title: Happy 1st Birthday Slash! (translated)
---

Happy birthday! (translated)
Loading

0 comments on commit 3166fab

Please sign in to comment.