Investigate how to best unbundle prepackaed langauges

[dbaeumer]

With VS Code 1.21 we started to support language packs. We should now start to think about how to best unbundle pre-packaged language and used language packs instead. This transition period should be for existing users as smooth as possible. So may be we still ship the pack and installed it if shipped. For new users it is OK to ask them to download a language pack.

[dbaeumer]

I propose to do the following to un-bundle the prepacked languages:

1. for the core languages we also create language packs and publish them to the market place except for English. This will always ship pre-bundled.
2. we ship all core languages bundled as well for the next n months.
3. when VS Code starts and the UI is in one of the core languages we download the translation from the market place and install the extension in the background. This requires work from @sandy081
4. when VS Code starts and detects an extension for one of the core languages we use the extension instead of the bundled one. We could even do some A/B testing here but since the translations should be the same there is no win. This requires work from @dbaeumer
5. after n months we remove the core languages from VS Code (except English). This require work from @dbaeumer (change build scripts)

This approach ensures a smart transition for users that already use a UI other than English. They basically don't see any differences to today.

For users that install a fresh version of Code at a point in time were we only ship English the experience would be as follows:

1. UI comes up in English.
2. a message is shown that there is a language pack for the UI language used by the OS and whether the user wants to install it. This message must be in English and the UI language used by the OS (the user might not speak English and with our current solution the UI would come up in the UI language of the OS). This requires work from @ramya-rao-a especially around the fact that we would need to present the string in English and the UI language. Ideas to get to the UI language strings would be:
   1. hard code them in the VS Core
   2. Use hidden tags in the market place (@sandy081 would this be feasible. Is there a size limit to tag values). This also requires to be online.
   3. Download the extension and have the strings in the package.json file of the extension. This also requires to be online.
3. if the user accepts the installation we will install the language pack and switch the UI language.

[dbaeumer]

@ramya-rao-a @sandy081 can you please read and comment.

[ramya-rao-a]

Sounds good. Hard coding the message in the core languages is the simplest way to go. Hidden tags for just one message feels like an overkill

[dbaeumer]

In my model we need to have the messages for all languages. Would be cool if someone runs in Bulgarian and he sees a message in Bulgarian and English. This is how Windows behaves as well.

[sandy081]

3. when VS Code starts and the UI is in one of the core languages we download the translation from the market place and install the extension in the background. This requires work from @sandy081

If this is when VS Code is bundled with core languages then Why do we need to download from Marketplace? I think this should be needed when VS Code is not bundled with core languages and user had set up VS Code before with one of the core languages.

When VS Code is not bundled with core languages, Downloading during start up will impact the start up performance (only the time we download) and what do we do if user is offline? I think we have two cases here:

Users with VS Code installed before and go through update cycle

• While updating, download the lang pack if the user is using a core lang pack. This will regress the behaviour that user cannot simply set the locale to other core languages. User has to install the language pack before setting.

• While updating, download all core language packs

Users with VS Code installed before and install/download VS Code again
Users with fresh install of VS Code

• Have the same experience as for new User?
• Did we think about a Installation wizard asking the language to pick?

Regarding message in OS Language, I would not go for tags. Hardcoding them for all languages is not practical. As @dbaeumer said, it can help for some (core+known) languages but not if there is a new. I think reading it from the package.json is a good option in which case we should come up with a property in package.json to be used. What if the extension does not provide the message in package.json? We might need a validation check while generating the vsix.

[dbaeumer]

If this is when VS Code is bundled with core languages then Why do we need to download from Marketplace?

To have a smooth migration for existing users. So we will switch them to language packs without them noticing and difference in the UI. If we first unbundle all core language then they will be promoted to download their language pack. This is to avoid that prompt.

We though about doing this in update, but we have different update stories for Mac, Windows, Deb, RPM and I was trying to come up with one solution instead of 4 :-).

New users will get a different experience (even for the current core language) since at some point we will ship English only. For them I think it is important that they see at least the 'language pack instsall message' in their UI language.

[sandy081]

To have a smooth migration for existing users. So we will switch them to language packs without them noticing and difference in the UI. If we first unbundle all core language then they will be promoted to download their language pack. This is to avoid that promp

I think it is a good idea to download the lang pack extensions from Marketplace while they are bundled with VS Code to prepare for next step. This would need confirmation from user to download. Downloading without asking might not be good. I think this can also be done while checking for the VS Code update and inform the user that update to next version of VS Code need to install so and so language pack extensions.

[dbaeumer]

Agree on the fact that we need to let the user know that we are donwloading language packs.

[dbaeumer]

@ramya-rao-a I assume now work has happened for the messages. I will move the item to Mai to keep it so that we can finish things up.

[waldyrious]

Just a quick note, in case it helps with task tracking related to this issue: if I'm understanding correctly, after this change is completed, the wiki page Visual Studio Code Community Localization Project (particularly the Language set section) will have to be updated to reflect the new setup, right?

[sandy081]

@waldyrious Yes, thanks for bringing this up

[ramya-rao-a]

@dbaeumer @sandy081

@kieferrm and I have the below proposal for the message that prompts the user to search/install the language pack corresponding to their display UI language

• There are 3 such strings. One the actual message, and the other 2 for the actions on the notification.
• Include the message and the accompanying text on the actions, as strings to be localized in the product, such that they get picked up by folks over at transifex. This is especially helpful for new languages over time
• During endgame, pull these localized strings into the product.
• When the language being shown is English, but users display language is different and we have the above strings localized in the display language, the prompt will say something like There are extensions in the Marketplace that can localize VS Code using the ${0} language
• The first action would be Search Marketplace which will pre-fill the search text in the Extensions view with the tag for corresponding locale. Example: tag:lp-pt-br. The second action would be Don't show again
• The reason I suggest Search Marketplace instead of Install extension is that in the future there can be multiple extensions providing localization for the same language. Instead of taking sides, we let the user decide.

43d48b1 is how it would look like if we pulled these localized strings into the product.json file.

The decision still pending in this proposal is the place where we pull this strings. Is product.json ok or we have a dedicated json file for this. Thoughts?

[sandy081]

• Showing Search Marketplace is a good idea but I am bit concerned that Extensions view will not have translated strings and actions will be shown in English when we show language packs in the Extensions view. Also for core languages, suggesting Install Extension makes more sense (which is also same at present). So, I think it will be helpful to have an Install Action.

Include the message and the accompanying text on the actions, as strings to be localized in the product, such that they get picked up by folks over at transifex. This is especially helpful for new languages over time

Looked at the code. I understood how it works with bundled languages, but not sure about other languages. Does the VS Code still need to build and ship out of the box translations? If not, how does the translations get applied?

[ramya-rao-a]

Yes, we can show the Install instead of Search Marketplace in the below cases

• For languages packs of the form MS-CEINTL.vscode-language-pack- i.e the ones from the https://github.com/Microsoft/vscode-loc repo. This would include the core languages plus anything that we publish from transifex
• Run the search before hand behind the scenes and show "Install" if there is only 1 result.

In my commit, the "bundled" languages do not refer to the bundled languages as we know and understand today. They are hard-coded localized strings that we should ship in product (either as a separate json file or in product.json).

[ramya-rao-a]

btw the work done for #44012 was pointless. The recommendation there is based on the platform.language which never is non English for non core languages unless there is a language pack installed.

So the prompt there appeared when the user has already installed an language pack and the platform.language wasnt an exact case match with the locale used in the package json file of the extension

[ramya-rao-a]

Can you explain to me how these strings lets say for a Bulgarian language pack get into our system / the vsix manifest

• The strings in vs/platform/node/minimalTranslations get localized to Bulgarian just like any other string in VS Code (via transifex today, or any other way in the future) and become part of the language pack
• The vsce tool will extract the translations for vs/platform/node/minimalTranslations file and put it in the manifest
• VS Code queries the marketplace, finds the language pack, queries its manifest to get the translations.

At no point above we depend on the i18n folder of our Github repo.

If they want to move from Transfix to another translation platform they can do that without us being majorly affected.

How the strings get translated doesnt matter to this PR. The only thing matters is that they get translated in the language pack.

[dbaeumer]

OK. Now I understand it. There is one thing to it though: currently we don't enforce a language pack to be complete. That means if these strings aren't translated they will be replaced with their English version when generating them. So it could easily happen that these strings might not be there in Bulgarian. Forcing them into package.json made it more obvious that these strings are mandatory in the Bulgarian language for example.

[sandy081]

I would say the approach of merging these special strings into regular translations is interesting and good. But I agree with @dbaeumer here that making these special strings part of regular translations will be confusing. Because, when you are creating a Language pack extension it is not clear about these special strings and not sure what happens if I do not provide translations to these strings. I think these strings are special in a way and extension authors should know what these strings are and when they are used.

I am also worried that vsce will now have knowledge of these translation files for reading the strings.

To be simpler and clear, How about having an optional property extensionRecommendationPromptTranslations (or something appropriate) to the schema

"localizations": [
      {
        "languageId": "abc",
        "languageName": "abc",
        "languageNameLocalized": "abc",
        "translations": [
          {
            "id": "vscode",
            "path": ""
          }
        ],
        "extensionRecommendationPromptTranslations": {
          "recommendatonMessage": "",
          "searchMarketplaceActionLabel": "Search MarketPlace in abc language",
          "installAndRestartActionLabel": ""
        }
      }
    ],

This property can be written into the vsix manifest and will be read while recommending the extension in VS Code. I think this will be clear to the author why and when this property is needed and will not overlap with general translations.

[ramya-rao-a]

@dbaeumer

if these strings aren't translated they will be replaced with their English version.

The vsce tool already has many other validations in place. We can add one more to check. If there are translations for the main vscode type, but they do not contain any for the vs/platform/node/minimalTranslations.ts file, the tool fails with something like: "The strings in the file vs/platform/node/minimalTranslations.ts are the bare minimum you need to translate for this extension to be a valid language pack"

@sandy081

How about having an optional property extensionRecommendationPromptTranslations?

If its optional, then we still don't avoid the case of the strings not being translated.
We can't make it mandatory as there can be language packs that just want to do translations for say the built-in extensions

The recommendation message differs based on whether we are asking the user to install or search. So, in total there are 5 strings. Install msg, Search msg, Install label, Install & Restart label, Search label. I have a feeling it will get complicated to describe each of these.

Say, the extension author hand-writes the package.json file, how do we communicate what each of these strings are for? Say the extension author uses yo code` to generate the extension initially and we pre-fill these strings, how do we ensure that the extension author updates these strings? In the current approach, the strings get translated without any extra thought process by the extension author. The translation system takes care of it

We could add checks in the vsce tool similar to the one that we have that ensures that the README has been updated. But, we could do the same for the current approach in the PR as well.

I am also worried that vsce will now have knowledge of these translation files for reading the strings.

If we have the strings directly added to the package.json by the extension author, the vsce tool will now have to be aware of each of the 5 strings and validate that they get translated/changed. How is that any better than being aware of a single special file which is deemed as the bare minimum translations?

Yes, I agree that the vsce tool is now aware of the vs/platform/node/minimalTranslations.ts file, but the advantage is that this solution scales. For any which reason, in the future we need to display a message that we deem important enough to be translated, there is no extra work to be done by us. We just put that string in this file, and when it gets translated, it is available for use.

[sandy081]

@ramya-rao-a

After thinking a lot and going through your comment, I would say I liked the approach of making these special strings part of general translations. I think if the language pack does not provide these language strings then showing them in default language (english) is a fair assumption. This gets applicable even to those language packs which are not up to date with latest.

I will go through the changes and push the changes by myself if there are any changes required since you have holiday today.

[dbaeumer]

@sandy081 and I discussed it offline and we decide to go with the current solution. If possible we should warn the user about the fact that these translations are mandatory or at least document it.

[sandy081]

@dbaeumer Looks like @ramya-rao-a is throwing an exception in vsce if these minimum translations are not found

[sandy081]

@ramya-rao-a I updated the implementation to read the core translations file from market place. Core translations file is added as an asset to the vsix.

[dbaeumer]

@sandy081 thanks.

[ramya-rao-a]

@dbaeumer @sandy081 Thanks

When do we stop shipping the core languages out of the box? June, July?
Do we keep this issue open till that time? Or create a new one for it later?

[sandy081]

@joaomoreno FYI: I have to remove the asset types filter and ask for all asset types so that I can get locale translations assets. Please see the change set 684b7e7

[sandy081]

@ramya-rao-a We would like to have insiders running without core languages for at least one milestone before going to stable. If translations team is ready, we can start 1.25.insiders without core languages and ship 1.25 stable without them.

[dbaeumer]

Agree with @sandy081 suggestion.

[dbaeumer]

I would close this issue and create a new one for the remaining work.

[ramya-rao-a]

Thanks @sandy081 @dbaeumer
Logged #50588 for the June milestone to cover the rest

[RMacfarlane]

@ramya-rao-a Can you provide a list of what should be verified as part of this issue? There are several scenarios mentioned here

[ramya-rao-a]

@RMacfarlane Sorry about that. The test plan item covers everything. I'll remove the verification-needed label