Add translation helper shell script #2332

justinbeaty · 2022-07-20T14:13:33Z

Description (*)

Because in #2317 I lost track of which translation strings I introduced, I wrote a quick helper script for me to find strings used in the __ function or in XML files but not defined in a CSV file. I thought it would be useful so I fixed it up a bit and added it as shell/translations.php.

It has two modes, missing which will tell you which strings you must add to a CSV file, and unused which will print strings defined in CSV files but not apparently used anywhere in the source. Additionally, it lets you pipe in a list of files to only check, which is useful to test PRs.

Usage:  php -f translations.php -- [options]
        php -f translations.php -- missing --verbose

  missing           Display used translations strings that are missing from csv files
  unused            Display defined translations strings that are not used in templates
  --verbose         Include filename with output
  --lang <lang>     Specify which language pack to check in app/locale, default is en_US
  deprecated        Find deprecated usage of the global __() function
  --fix             Overwrite files to fix deprecated usage DO NOT RUN IN PRODUCTION!
  help              This help

Note: By default, this script will check all files in this repository. However,
      you can pipe a list of files to check for missing translations strings.
      This is useful for checking a specific commit. For example:

      # Check if last two commits may have introduced missing translations
      git diff --name-only HEAD~2 | php -f translations.php -- missing

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Run php shell/translations.php missing --verbose to see php, phtml, or xml files that used non-defined translations
Run php shell/translations.php unused --verbose to see CSV files that have unused translations

Questions or comments

# Returns 422 unused strings
php shell/translations.php unused | wc -l

# Returns 85 missing strings
php shell/translations.php missing | wc -l

It could be possible that the code to find usage of the __ function is not perfect, such as __('Total: ' . $total) would only pick up 'Total: ', but I don't think that's used often, or shouldn't be, as the %s usage is preferred.

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All automated tests passed successfully (all builds are green)
Add yourself to contributors list

justinbeaty · 2022-07-20T14:14:48Z

I did not do any sort of CSV clean up, but maybe someone else wants to take the lead on that. For example, adding missing translations or maybe removing translations that aren't used.

Edit: One warning about possibly removing unused translations, it's possible as mentioned in the previous comment that there could be some usages of calling __ with some sort of variable. For example "Google Checkout Shipping - Merchant Calculated" is not used anywhere, but it could be returned from Google's API and translated. There could be other unforeseen cases like this as well.

elidrissidev · 2022-07-20T14:19:43Z

Wow nice work! Will review when I get the time.

justinbeaty · 2022-07-20T14:37:40Z

I don't know much about github actions, but maybe this could also be used for one to check that translation strings have been included if necessary.

fballiano · 2022-07-21T12:02:47Z

wow this is great also for multistore development!
I always like to have verborse filenames since "translations.php" is pretty generic but since it does have multiple running modes...no better name comes to my mind

shell/translations.php

tmotyl · 2022-07-21T14:58:54Z

have you checked if n98-magerun has sth similar already?

justinbeaty · 2022-07-21T15:04:22Z

@tmotyl I did look but I don't see anything like it:

  dev:translate:admin                Toggle inline translation tool for admin          
  dev:translate:export               Export inline translations                         
  dev:translate:set                  Adds a translation to core_translate table. Globally for locale
  dev:translate:shop                 Toggle inline translation tool for shop

justinbeaty · 2022-07-21T17:38:59Z

As mentioned in #2333, I added a new mode to find deprecated usage of the global __() function:

# list deprecated usage only
php shell/translations.php deprecated

# fix deprecated usage, DO NOT RUN IN PRODUCTION
php shell/translations.php deprecated --fix

addison74 · 2022-08-15T15:51:29Z

This is an interesting PR and thank you for your contribution @justinbeaty. If I look at its description in the Questions or Comments section, I see that there are 422 unused strings and 85 missing strings. That is a problem. I'll take a look to see what it's all about.

For the deprecated option there is also a fix option. Running the command what is done cannot be reversed. We should evaluate if it is possible to make a backup of the files beforehand. If it is not possible, then a warning message must be mentioned to make a backup. We have to check where these changes are and find the parent directory of the files. If there are issues, either put all the files back in place, and if he uses Git he can do a revert.

justinbeaty · 2022-08-15T16:10:51Z

For the deprecated option there is also a fix option. Running the command what is done cannot be reversed. We should evaluate if it is possible to make a backup of the files beforehand. If it is not possible, then a warning message must be mentioned to make a backup. We have to check where these changes are and find the parent directory of the files. If there are issues, either put all the files back in place, and if he uses Git he can do a revert.

Ideally someone should not just blindly run the --fix option but should run php shell/translations.php deprecated first to see if there are any occurrences of the deprecated usage. Depending on the 3rd party module authors used, there could be zero occurrences. I think the warning is in the --help usage instructions where it says:

  --fix             Overwrite files to fix deprecated usage DO NOT RUN IN PRODUCTION!

I am not sure if we are responsible for backing up files for someone not using git and deploying directly to production. But if there's a better warning or other way to prevent accidental changes we can add it.

justinbeaty · 2022-08-15T16:19:08Z

As mentioned in #2429 I think a new mode can be added such as:

php shell/translations.php upgrade

This mode would allow us to change the first column in en_US csv files when we change a translation string. It would be required to add an entry to this shell script so that we can simply find/replace the changes.

In other words, take this example:

"Password must be at least of %d characters."

We can instead fix that in both columns and also put in this shell script something like:

$upgrades = array(
  "Password must be at least of %d characters." => "Password must be at least %d characters.",
);

And when the user runs php shell/translations.php upgrade it will search all other language packs for the first string to replace with the second.

Thoughts? Feedback from others who use non-english language packs is appreciated. This would be a new step users must take during upgrades.

addison74 · 2022-08-15T16:32:02Z

Column 1 is what is used in PHP/PHTML files.
Column 2 is what is used in CSV files and is displayed in Backend/Frontend.

It is obvious that we must pay attention to column 2. And then the modification should occur on column 1 of the CSV, but once we modified it must have a correspondent in the PHP/PHTML file.

PHP/PHTML FILE => COLUMN 1 CSV => COLUMN 2 CSV

To remain Column 2 then the process must be reversed.

justinbeaty · 2022-08-15T16:39:41Z

Let me try to re-explain what I propose.

When we want to change a string, we can do so in the PHTML files and also in both columns in the en_US csv files. This keeps everything consistent in our source code.

At the same time, we add an entry to translations.php shell script such that it will find/replace in all translation packs the old string with the new string.

So when a user upgrades to a new OpenMage version, they just have to run the script to fix what we broke in their translation files. The only negative is that they need to be aware of having to run that script.

addison74 · 2022-08-15T16:43:30Z

I understand what you mean. I haven't tested the script yet, but I will, including in destructive mode to see what happens and how such a "mistake" can be fixed.

sreichel · 2022-09-12T17:10:50Z

have you checked if n98-magerun has sth similar already?

I'm pretty sure something like this already exists. If not in n98-core, maybe as addon?

However, I'll test this later :)

Wait a sec ...

sreichel · 2022-12-25T23:40:05Z

@fballiano @Flyingmana

I'd like to have 3 further repos at OpenMage ...

openmage/dev-translations (for this PR)
openmage/dev-copyright (Add OpenMage Contributors Copyright #2295)
openmage/dev-meta-package

These shell scripts are not needed for production ... just install them with composer require --dev openmage/dev-meta-package

fballiano · 2022-12-25T23:54:06Z

created the repos and set you as admin

sreichel

We can approve this first (for better history) and then apply #2853

Add translation helper shell script

4bf075c

github-actions bot added the shell Relates to shell scripts label Jul 20, 2022

fix stdin array

c45dbcb

fballiano reviewed Jul 21, 2022

View reviewed changes

shell/translations.php Show resolved Hide resolved

justinbeaty mentioned this pull request Jul 21, 2022

Remove Deprecated Global Translate Function "__()" #2333

Closed

3 tasks

Add deprecated helper mode

c850f13

justinbeaty added 2 commits July 21, 2022 11:01

fix deprecated script

0bdd589

fix indent

8dfba6f

whitespace

54f4a06

fballiano previously approved these changes Sep 28, 2022

View reviewed changes

sreichel added 3 commits December 25, 2022 21:28

Merge branch '1.9.4.x' into topic-find-missing-translations

72aca19

CS-Fixes

103cf30

phpstan level 9

11bd397

sreichel dismissed fballiano’s stale review via 11bd397 December 25, 2022 21:45

sreichel previously approved these changes Dec 25, 2022

View reviewed changes

sreichel mentioned this pull request Dec 26, 2022

Load dev shell scripts as composer module #2853

Merged

4 tasks

sreichel approved these changes Dec 27, 2022

View reviewed changes

Merge branch '1.9.4.x' into topic-find-missing-translations

70d33eb

sreichel requested review from fballiano and addison74 December 27, 2022 03:12

fballiano approved these changes Dec 27, 2022

View reviewed changes

fballiano merged commit 514bb49 into OpenMage:1.9.4.x Dec 27, 2022

sreichel mentioned this pull request Jul 20, 2024

Missing translations in Mage_Sales.csv #4106

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add translation helper shell script #2332

Add translation helper shell script #2332

justinbeaty commented Jul 20, 2022 •

edited

Loading

justinbeaty commented Jul 20, 2022 •

edited

Loading

elidrissidev commented Jul 20, 2022

justinbeaty commented Jul 20, 2022

fballiano commented Jul 21, 2022

tmotyl commented Jul 21, 2022

justinbeaty commented Jul 21, 2022

justinbeaty commented Jul 21, 2022

addison74 commented Aug 15, 2022

justinbeaty commented Aug 15, 2022

justinbeaty commented Aug 15, 2022

addison74 commented Aug 15, 2022

justinbeaty commented Aug 15, 2022

addison74 commented Aug 15, 2022

sreichel commented Sep 12, 2022

sreichel commented Dec 25, 2022 •

edited

Loading

fballiano commented Dec 25, 2022

sreichel left a comment

Add translation helper shell script #2332

Add translation helper shell script #2332

Conversation

justinbeaty commented Jul 20, 2022 • edited Loading

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Questions or comments

Contribution checklist (*)

justinbeaty commented Jul 20, 2022 • edited Loading

elidrissidev commented Jul 20, 2022

justinbeaty commented Jul 20, 2022

fballiano commented Jul 21, 2022

tmotyl commented Jul 21, 2022

justinbeaty commented Jul 21, 2022

justinbeaty commented Jul 21, 2022

addison74 commented Aug 15, 2022

justinbeaty commented Aug 15, 2022

justinbeaty commented Aug 15, 2022

addison74 commented Aug 15, 2022

justinbeaty commented Aug 15, 2022

addison74 commented Aug 15, 2022

sreichel commented Sep 12, 2022

sreichel commented Dec 25, 2022 • edited Loading

fballiano commented Dec 25, 2022

sreichel left a comment

Choose a reason for hiding this comment

justinbeaty commented Jul 20, 2022 •

edited

Loading

justinbeaty commented Jul 20, 2022 •

edited

Loading

sreichel commented Dec 25, 2022 •

edited

Loading