Update procedures and documentation for TranslateWiki merge and langfile rebuild

[cpcallen]

Background

The translatewiki branch diverged from develop one year ago, and has not had any updates (relevantly: of any of the gulpfiles) since then. As a result, it is not clear what procedures should be used to merge TranslateWiki updates into develop and rebuild the language files (or indeed even what order those should happen in.)

Separately, there is the question of whether we should even continue to have checked-in copies of the built langfiles in the blockly repository.

Proposed changes to repository

develop branch

I propose that we:

• remove msg/js/* from the repository,
• configure all applicable test harnesses, playgrounds, package scripts etc., to load from build/msg instead, and
• configure the npm prepare script, run as a side effect of npm install, to to run buildLangfiles.
  • This is very fast and can be done in parallel with the existing buildJavascript + buildDeps sequence.

translatewiki branch

At minimum we should update translatewiki from develop—preferably via a merge commit, to make it very clear in the history—and continue to do so on a regular basis.

We could consider deleting everything except msg/json from translatewiki. This would reduce the likelihood of an unauthorised code change being inadvertently merged from this branch in to develop.

Deleting everything except the translations would create some difficultly in easily merging back and forth between the two branches. To facilitate merges from translatewiki to develop, a subterfugous merge commit (in that direction) could be created, claiming to include the deletion without actually doing. This would mean that subsequent PRs would not attempt to delete bulk of the codebase from develop.

Updating in the reverse direction would probably entail manual copying between branches.

Proposed changes to procedures

We should create a new document explaining how to merge TranslateWiki update from translatewiki to develop and (if they have not been deleted from the repository) rebuild the compiled langfiles. This document should be linked from (and replace the existing) "merge in translate wiki updates" section of our internal "day of release" documentation.

Insofar as feasible, the procedure should:

• Ensure that any scripts/tooling that is needed is run from the develop branch if possible, or failing that from the translatewiki branch only after it has been updated from develop
• Include running npm ci at appropriate points to ensure that dependencies are up-to-date.

[cpcallen]

Description updated following discussion in team meeting.

[maribethb]

@BeksOmega do you have any context on why you re-added this to triage? Did this come up recently for a current issue?

[BeksOmega]

@BeksOmega do you have any context on why you re-added this to triage? Did this come up recently for a current issue?

I /think/ I put this back in triage because it seemed relevant to the monorepo stuff + release automation, and I knew y'all would be chatting about monorepo stuff after break. But I'm only 80% sure.

We also had an issue with the messages in the last release, but I don't think that's why I flagged this.

[maribethb]

Okay, thanks. I think we're just leaving this in the backlog for now. As I understand it, our process for merging develop back into translate-wiki may need some improvement but it doesn't seem like there's an urgent need to do anything and we don't have a solid plan for what that process improvement would be. I don't think the monorepo changes how we interact with translate-wiki other than possibly changing its permissions or how it interacts with branch protections.

[cpcallen]

Several parts of this issue have been dealt with. In partciular, in cl/503198373 (19 Jan 2023) I updated the release documentation with instructions on how to the translatewiki -> develop update in a "safe" way (i.e., one that does not risk pulling in code changes and does not depend on code in the translatewiki branch), but the question of what exactly should be in the translatewiki branch and how it should be updated from develop is still an open question, with updates currently being done ad-hoc.

Noting that msg/messages.json should be the source-of-truth for messages in English and for translation instructions, but that developers will want to be able to update msg/json/en.json in order to verify message changes locally, I think we might want to have:

1. A validation workflow that runs on every PR, and does checks to ensure that the only changes made to files in msg/json/*.json are ones that result from having made a change to msg/messages.js and then run the messages gulp task.
2. A workflow that updates the translatewiki branch by watching for commits on develop, and, if those commits touch msg/messages.js, applying the same change to that file in the translatewiki branch and then, using the version of the messages gulp task in the develop branch, apply the corresponding changes to msg/json/*.json in the translatewiki branch instantly via a (non-PR) commit.
3. A workflow, which can be run on demand (and which might also be run on some schedule) that creates a PR to update develop from translatewiki using the same steps as the currently-documented manual "day of release" procedure.

The purpose of the first workflow is to ensure that msg/messages.json remains the source of truth for English language messages, and that the translatewiki branch remains the source of truth for all others.

The second workflow ensures that new/changed messages will get sent for translation promptly. Note that, since files in msg/json/ (other than en.json and qqq.json et al.) will normally be behind the translatewiki branch, we must be careful that it does not blindly overwrite those files.

The third workflow will automate the existing procedure for creating a PR to manually reviewithe translation updates before they become eligible for publication.