Integrate with Crowdin localization project

[bvaughn]

Migrated from facebook/react/issues/8063

Coordinate with docs translators via @ericnakagawa

This issue involves multiple steps:

• Integrate with Crowdin to download localized content
• Generate localized URLs using said content
• Automatically sync English content changes from GitHub to Crowdin (see this comment and this repo)
• Maintain backwards compatibility for all current (non-localized, English) links
• Add localization toggle to header
• Update the "Edit this page" link to point to GitHub for English pages and Crowdin for other languages.

Integrate with Crowdin

This could be done via a custom Gatsby plug-in. We've already written a couple. (It would be nice to work with @KyleAMathews to share this plug-in back with Gatsby so that other users could benefit from it as well.)

If we run into any performance problems, @ericnakagawa has volunteered to work with Netlify and Crowdin to simplify or speed up this syncing process. Ideally this sync process will be fast enough to allow Netlify's PR previews to also be used to test localized builds of the site.

Misc thoughts and questions

• In the case of non-localized content, Crowdin serves English as fallback.
• How can site consume list of active locales? Is there a Crowdin API?
• How does this impact urls (eg /docs/<file>, /docs/<lang>/<file>)?

[KyleAMathews]

Happy to help create a Gatsby source plugin for Crowdin! That'd be really powerful. And a lot of people would use it including sooner than later gatsbyjs.org

From a bit of research, it looks like it'd be pretty easy to pull data using their API:

• https://support.crowdin.com/api/download/
• Guide for creating a source plugin https://www.gatsbyjs.org/docs/create-source-plugin/ — this guide is a bit wordy right now. When someone wants to tackle this, I could take a pass at improving the guide + adding a mini-tutorial on creating source plugins.

I spent a bunch of time adding localization support to https://www.gatsbyjs.org/packages/gatsby-source-contentful/ as well so have some opinions on this. See the example site for Contentful + its source

[bvaughn]

Fantastic 😄 There's a good chance we'll end up collaborating on this soon then.

[Daniel15]

Netlify also has a way of doing redirects based on Accept-Language header and GeoIP. For example, if someone from Italy (or someone with their browser or OS set to request Italian content) visits the home page and they don't have a language cookie already set, we can automatically redirect to the Italian version. We use this on the Yarn site - You can look at how Yarn does it for inspiration

How does this impact urls (eg /docs/, /docs//)?

I'd suggest /<lang>/ (eg. /<lang>/docs/<file>), this is consistent with other Facebook projects that use localization:
https://yarnpkg.com/en/docs/getting-started
https://flow.org/en/docs/getting-started/

If we run into any performance problems, @ericnakagawa has volunteered to work with Netlify and Crowdin to simplify or speed up this syncing process. Ideally this sync process will be fast enough to allow Netlify's PR previews to also be used to test localized builds of the site.

We have perf issues with the Yarn site, but a lot of it is because we're using Jekyll. Most of the build time is actually the Jekyll build time rather than CrowdIn. We had a prototype using Hugo that was much faster. I don't think CrowdIn takes a significant amount of time there.

From a bit of research, it looks like it'd be pretty easy to pull data using their API:

Would the React site actually need a custom API integration? It could be useful, but CrowdIn has a CLI tool (https://support.crowdin.com/cli-tool/) which we might be able to use with less effort. For Yarn we just run crowdin-cli download -b master to download all the localized files as part of the build (in that case, there's a bunch of Markdown files as well as some yml files for language strings). Then you just build the site like normal and everything just works, as long as the files are in the correct locations.

[bvaughn]

How does this impact urls (eg /docs/, /docs//)?

I'd suggest // (eg. //docs/), this is consistent with other Facebook projects that use localization:

Don't have time to fully digest this response now- but wanted to mention that we should ensure whatever change we make does not break pre-existing links (like how links broke for Jest).

[KyleAMathews]

Would the React site actually need a custom API integration

It doesn't need it but it seems simpler perhaps to just query the data through graphql instead of writing it out first then pulling it in.

But yeah, definitely more straightforward to just use the cli.

Would you checkin the translated content?

[Daniel15]

but wanted to mention that we should ensure whatever change we make does not break pre-existing links

Jest is limited in what it can do, because it's using GitHub Pages. With Netlify, you can generate some redirects as part of the build. Just need to redirect "raw" URLs (/docs/foo) to the localized URL (/en/docs/foo), either with /en/ hard-coded or using Netlify's language redirects that I mentioned earlier (so it'd redirect /docs/foo to /<user's preferred language>/docs/foo).

Check it out with Yarn: https://yarnpkg.com/docs/getting-started redirects to the right place. :)

Would you checkin the translated content?

We don't for Yarn. It's pulled in as part of the build process for the site. Since CrowdIn tracks the history for the translations, we didn't really need to also have Git track it.

Some sites do commit the translated files though. Up to you - Either way is fine.

[bvaughn]

Would you checkin the translated content?

No

[bvaughn]

or using Netlify's language redirects that I mentioned earlier (so it'd redirect /docs/foo to /<user's preferred language>/docs/foo).

Cool! 👍

[Daniel15]

Here's how the Yarn site builds all the redirects from non-localized URLs to localized URLs: https://github.com/yarnpkg/website/blob/master/_redirects#L48-L70 (this would also be how you handle your redirects from 'old' URLs to new localized URLs). Unfortunately the Yarn one is super difficult to understand because it's using the Jekyll templating syntax. Somehow @thejameskyle got it working though. You could likely make a nicer JS script to do the same thing for the React site.

The resulting redirects file looks something like this:

/docs/cli/add/  /en/docs/cli/add/  302  Language=en
/docs/cli/add/  /fr/docs/cli/add/  302  Language=fr
/docs/cli/add/  /pt-BR/docs/cli/add/  302  Language=pt-br
/docs/cli/add/  /zh-Hans/docs/cli/add/  302  Language=zh-cn

Here's the entire language redirects section: https://gist.github.com/Daniel15/d84b39e8a260ee2a7d627751d015181b

Docs: https://www.netlify.com/docs/redirects/#geoip-and-language-based-redirects

Might be able to use a regex or placeholders or something rather than having to generate all the redirects like that.

[KyleAMathews]

@bvaughn wrote a nice redirect API for Gatsby + added it to gatsby-plugin-netlify so this setting up redirects would be pretty easy :-)

[smikitky]

Hi, the official doc in English is being updated frequently, but the contents in Crowdin seems to be out of sync with the newest doc. I think it's still based on the doc several months ago, and I'm seeing no doc for error boundaries ready for translation.

I'm new to Crowdin, but I want to believe there is a way to merge the updates in the original doc and enable people to translate the changed part incrementally. Is there any plan for this? I'd rather not see an overwhelming one-time update of the original doc.

[bvaughn]

Crowdin docs show support in their REST API to add, update, and delete a file- so it seems plausible that we could write a script to automatically sync english content from GitHub to Crowdin. Maybe we could even use CircleCI to run this automatically anytime there are commits to the master branch?

cc @ericnakagawa who has been leading the localization effort until now.

[Daniel15]

I think we already have something configured with Yarn to automate it - Crowdin is automatically updated when changes are committed to the English versions. I'm not quite sure how it works though.

[bvaughn]

Thanks Daniel. I added a note to the description of this issue to check what Yarn's doing for this.

[Daniel15]

It seems like Crowdin have some sort of Github integration now. I don't think Yarn is using this though: https://support.crowdin.com/github-integration/.

James Kyle set up the Yarn one, it might be worth reaching out to him.

[jamiebuilds]

Hi hi hi, yes I use https://github.com/thejameskyle/crowdin-sync for Yarn

[clemmy]

I'm interested in helping out with this as well. 🙂 @KyleAMathews @Daniel15 Do you guys have a preference on whether we do this through Github integration or just in time (during the build)?

[Daniel15]

I don't think it really makes sense to commit translations to the repo, since CrowdIn already tracks changes to them. We should just have a single source of truth. Either way is fine though, I don't have a strong opinion about it.

[bvaughn]

I think we have a couple of options that may be worth exploring. I have been leaning toward using the Crowdin API to pull data at build-time. If this turns out to be slow (either the API or the resulting Gatsby build time) then we can provide a mode to disable/bypass it for local dev use, or to only pull a subset of languages, etc.

[Daniel15]

For what it's worth, on the Yarn site we only pull down localizations in the prod build.

[bvaughn]

Agreed that we shouldn't do them for yarn dev but- depending on the speed- it might also be nice to disable for yarn build. (It's often useful to test the "production" build locally too.)

[Daniel15]

If we could use Netlify somehow to do a locale-friendly redirect for old links, that's great. I didn't know about that.

I mentioned it in an earlier comment: #82 (comment)

(I don't see it mentioned in the Netlify docs for redirects.)

It's on the page you linked to, under the "GeoIP and Language-based redirects" heading. 😛

Here's what the language redirects look like for the Yarn site, as an example: https://gist.github.com/Daniel15/d84b39e8a260ee2a7d627751d015181b

[bvaughn]

I guess I misunderstood the section of the docs on the page I linked to. The comment you linked to is a month old. I guess I forgot it. /shrug

Thanks for linking to the Yarn redirects file. The Gatsby redirects plug-in will need an update to support the additional language parameter but that should be pretty straight forward.

The Link component will still need the ability to detect and prepend the language code itself (during the build phase) because of how Gatsby generates and optimizes its content graph.

[bvaughn]

PR to add support to Gatsby + gatsby-plugin-netlify for language and country based redirects: gatsbyjs/gatsby/pull/2890

[bvaughn]

May be worth noting, explicitly, that we don't plan to support localization for pages like the "Blog" landing page. (There's no plans to translate blog post content.) If we did decide to support that in the future, we would need to fork the page it somehow so that its GraphQL query only pulled in localized content.

[techhtml]

Hi there, I have a question about localization.
When I end translating Document into Korean, How can I see that in React site?

[bvaughn]

That's not currently possible, @techhtml. Once this issue is done, it will be.

[techhtml]

@bvaughn Thank you! i'll wait that work done.

[techhtml]

@ericnakagawa Hi Eric, I have question about crowdin localization. I translate static-type-checking docs into Korean. but crowdin's document are not updated that only have old one. How can I update this docs?

[clemmy]

Hey @techhtml, I believe your translations need to be reviewed and approved by an admin of the React project on Crowdin before your changes appear.

[techhtml]

@clemmy of course. Other Korean developers will help reviews my translations.

[smikitky]

Hi, I logged in to Crowdin after a while, and found the "translated" percentage of Japanese has dropped to 65%. It was 100% (i.e., translation completed) in September last year, but no one but translators themselves had a chance to read the translated docs :)

So can someone set up a timeline and a guideline for this on which we can rely on? You have already gathered dozens of volunteers at Crowdin even though you have not officially called for volunteers. Has the localization project started officially, in the first place? Are you going to call for more collaborators, in the blog for example? When are we expected to finish the translation? I don't think the entire Crowdin project may be abandoned, but since there is no timeline nor document to guide the translators, I don't know when we can restart translating to keep up with the current doc. I don't even know what those "master" and "test-17" folders mean. (BTW, I just found the translation percentage for the "master" folder is now 1%... Does this mean almost all the effort we already made on the "old" master is going to disappear soon?)

cc @ericnakagawa

[smikitky]

Can we expect some progress before the release of React hooks? For now, articles related to hooks are not even listed in the Crowdin repository, and it appears to me that this effort has been half-abandoned recently. It's sad to see developers in my country have to depend on fairly insufficient personal blog articles when such a big change is approaching to the ecosystem.

By the way, does anyone know why materials in Crowdin are in HTML format rather than Markdown? HTML tags, especially all those <code>s, are very cumbersome to handle.

ping @bvaughn @ericnakagawa

[bvaughn]

Sorry, I don't have any updates/estimates for this. The core team is spread very thin and I don't think anyone owns this effort at the moment 😞

[smikitky]

I see. Meanwhile, is it possible to allow translators to download/export their existing contributions from Crowdin? If I understand correctly, this can be done by changing the project settings. Then each translator may host the translated version elsewhere on the internet, say, my GH pages.

[bvaughn]

That's something I think @ericnakagawa would have to manage. I don't know much about our Crowdin setup. Not even sure if I would have the necessary permissions.

[gaearon]

To give an update — if everything works out, we expect a person to jump on finishing the integration in one of the next weeks. Can't promise to get it done before Hooks are stable though but it's on our radar.

[smikitky]

For the record, here are some problems I'm currently seeing as a translator.

• Translations are not Markdown-based but HTML-based. I believe all translated materials should be version-controlled.
• There are several "root" directories on the Crowdin repo, and it is not clear which to work on.
• New commits to the original doc are not automatically synched. For example there are no articles about Hooks yet in the Crowdin repository.
• Some elements in the front matter (document IDs, etc) are wrongly marked as translatable.
• Some code examples are marked as translatable and some are not.
• This may be specific to CJK languages, but italics are not always rendered properly for those languages, so we need a different CSS styles for <em>'s.

[Daniel15]

Translations are not Markdown-based but HTML-based.

For Yarn we use Markdown in CrowdIn (at https://i18n.yarnpkg.com/), so this might just be an issue with how it's configured for React. Should be straightforward to fix it 😃

[tesseralis]

@smikitky We've come up with an alternate approach here: #1605

Does this resolve any of your issues working with Crowdin, and would you be interested in working with us on maintaining the Japanese fork? We should be able to use the work that you and others have done so far on Crowdin as a baseline.

[gaearon]

https://reactjs.org/blog/2019/02/23/is-react-translated-yet.html