-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
doc: design for atlas pull for the edx-platform and its plugins FC-0012
#33166
Merged
+362
−0
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
362 changes: 362 additions & 0 deletions
362
docs/decisions/0019-oep-58-atlas-translations-design.rst
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,362 @@ | ||
| Design for Refactoring Translations ``pull`` to use Atlas | ||
| ########################################################## | ||
|
|
||
| Status | ||
| ====== | ||
|
|
||
| Accepted | ||
|
|
||
| Context | ||
| ======= | ||
|
|
||
| OEP-58 Translation Management overview | ||
| -------------------------------------- | ||
|
|
||
| The `Translation Management update OEP-58`_ proposal has been merged with | ||
| the following changes to the way translations are managed in the Open edX platform: | ||
|
|
||
| - Move Translation Files to the `openedx-translations repo`_ | ||
| - Add the `Transifex GitHub App <https://github.com/apps/transifex-integration>`_ | ||
| to openedx Organization | ||
| - Connect the `openedx-translations repo`_ to the | ||
| `openedx-translations Transifex project`_ | ||
| - Copy `Transifex Translation Memory`_ into from the both of the | ||
| `edx-platform Transifex project`_ and the `xblocks Transifex project`_ into | ||
| the new `openedx-translations Transifex project`_ | ||
| - Utilize `openedx-atlas`_ to pull translations for development/deployment. | ||
|
|
||
| If you're new to the `OEP-58`_ proposal, please | ||
| review the `OEP-58 Specifications`_ in addition to the | ||
| Key Metrics and Expected Results section in the | ||
| `Approach Memo and Technical Discovery - Translations Infrastructure Implementation`_ | ||
| document before continuing. | ||
|
|
||
| Pre-OEP-58 Architecture/Implementation for XBlocks and Plugins | ||
| -------------------------------------------------------------- | ||
|
|
||
| Before `OEP-58`_, Open edX XBlocks and Open edX plugins had the following: | ||
|
|
||
| - Translations live in the GitHub repository. | ||
| - Translations are packaged with the rest of the code when published to pypi | ||
|
|
||
| Pros: | ||
|
|
||
| - Translations are always available after installation. | ||
|
|
||
| Cons: | ||
|
|
||
| - This can mean a complex integration with Transifex | ||
| - This can mean a lengthy manual PR review process up to a month such as in | ||
| the following example: | ||
| `Added French (Canada) and Japanese - xblock-drag-and-drop-v2 #220`_ | ||
|
|
||
| XBlockI18nService | ||
| ----------------- | ||
|
|
||
| The `XBlockI18nService`_ loads translations for installed XBlocks via its | ||
| ``__init__`` method. XBlock translations are only used during the | ||
| during the execution of the XBlock. | ||
|
|
||
| The `XBlockI18nService implementation pull request`_ (2016) introduced | ||
| support for XBlock translations in ``edx-platform`` and has the full | ||
| context of the implementation. | ||
|
|
||
| .. _js-translations: | ||
|
|
||
| JavaScript Translations for XBocks | ||
| ---------------------------------- | ||
|
|
||
| As of September 2023, there is no centralized method to bundle JavaScript | ||
| translations in XBlocks. Non-XBlock plugins lack JavaScript translation | ||
| support altogether. | ||
|
|
||
| The de-facto standard method for bundling JavaScript translations in XBlocks | ||
| is to use ``web_fragment`` and load the translations as part of the XBlock | ||
| frontend static files on every XBlock load. | ||
|
|
||
| The LTI Consumer XBlock embeds the translations in its ``web_fragment`` via | ||
| the `LtiConsumerXBlock._get_statici18n_js_url`_ and | ||
| `LtiConsumerXBlock.student_view`_ methods. | ||
|
|
||
| In order to separate the XBlock translations from the platform, it's isolated | ||
| in a separate ``gettext`` namespace. For example, the Drag and Drop XBlock | ||
| namespace is ``DragAndDropI18N`` which is hardcoded in multiple places such | ||
| as: | ||
|
|
||
| - `XBlock Makefile compile_translations rule`_ | ||
| - `XBlock compiled JavaScript text.js translations`_ | ||
| - `XBlock main JavaScript file`_ | ||
|
|
||
| `OEP-58`_ does not change this structure, it just makes the necessary changes | ||
| to pull translations from the `openedx-translations repo`_ via ``atlas`` | ||
| instead of having them live in the XBlock repository itself. | ||
|
|
||
| Decisions | ||
| ========= | ||
|
|
||
| Proposed Design for edX Platform ``conf/locale`` translations | ||
| ------------------------------------------------------------- | ||
|
|
||
| We're going to use ``atlas`` in ``make pull_translations`` like we do in | ||
| `course-discovery atlas integration`_ and | ||
| `frontend-app-learning atlas integration`_. | ||
|
|
||
| Proposed Design for XBlocks and Plugins | ||
| --------------------------------------- | ||
|
|
||
| Instead of storing translation files for each XBlock and Plugin in their | ||
| respective repositories, | ||
| we will use `openedx-atlas`_ to pull them from the | ||
| `openedx-translations repo`_. | ||
|
|
||
|
|
||
| New ``atlas_pull_plugin_translations`` command | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| Introduce new Django commands to the ``edx-platform``: | ||
|
|
||
| - ``manage.py lms atlas_pull_plugin_translations --list``: List all XBlocks and | ||
| Plugins installed in the ``edx-platform`` virtual environment. This will | ||
| list the Python *module names* (as opposed to git repository names) of the | ||
| installed XBlocks and Plugins e.g.:: | ||
|
|
||
| $ manage.py lms atlas_pull_plugin_translations --list | ||
| drag_and_drop_v2 | ||
| done | ||
| eox_tenant | ||
|
|
||
| This list doesn't include plugins that are bundled within the | ||
| ``edx-platform`` repository itself. Translations for bundled plugins | ||
| are included in the ``edx-platform`` translation files. | ||
|
|
||
| - ``manage.py lms atlas_pull_plugin_translations``: This command | ||
| will pull translations for installed XBlocks and Plugins by module name:: | ||
|
|
||
| $ atlas pull --expand-glob \ | ||
| 'translations/*/drag_and_drop_v2/conf/locale:conf/plugins-locale/drag_and_drop_v2' \ | ||
| 'translations/*/done/conf/locale:conf/plugins-locale/done' \ | ||
| 'translations/*/edx_proctoring/conf/locale:conf/plugins-locale/edx_proctoring' | ||
|
|
||
| Resulting in the following file tree:: | ||
|
|
||
| $ tree conf/plugins-locale/ | ||
| conf/plugins-locale/ | ||
| ├── done | ||
| │ ├── ar | ||
| │ │ └── LC_MESSAGES | ||
| │ │ └── django.po | ||
| │ ├── de | ||
| │ │ └── LC_MESSAGES | ||
| │ │ └── django.po | ||
| │ ├── en | ||
| │ │ └── LC_MESSAGES | ||
| │ │ └── django.po | ||
| │ └── fr_CA | ||
| │ └── LC_MESSAGES | ||
| │ └── django.po | ||
| ├── drag_and_drop_v2 | ||
| │ ├── ar | ||
| │ │ └── LC_MESSAGES | ||
| │ │ └── django.po | ||
| │ ├── en | ||
| │ │ └── LC_MESSAGES | ||
| │ │ └── django.po | ||
| │ └── fr_CA | ||
| │ └── LC_MESSAGES | ||
| │ └── django.po | ||
| └── edx_proctoring | ||
| ├── ar | ||
| │ └── LC_MESSAGES | ||
| │ └── djangojs.po | ||
| ├── de | ||
| │ └── LC_MESSAGES | ||
| │ └── djangojs.po | ||
| ├── en | ||
| │ └── LC_MESSAGES | ||
| │ ├── djangojs.po | ||
| │ └── django.po | ||
| └── fr_CA | ||
| └── LC_MESSAGES | ||
| ├── djangojs.po | ||
| └── django.po | ||
|
|
||
|
|
||
| BlockI18nService support for ``atlas`` Python translations | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| ``get_python_locale_directory`` will support two modes: | ||
|
|
||
| #. If translations for the XBlock/plugin has been pulled by ``atlas`` | ||
| from the `openedx-translations repo`_, it will be used. For example, if the | ||
| ``edx-platform/conf/plugins-locale/drag_and_drop_v2/ar/LC_MESSAGES/django.po`` | ||
| path exists, it will be used for the Drag and Drop XBlock. | ||
|
|
||
| #. Otherwise, the bundled translation files in the XBlock packages will be | ||
| used. The fallback path for the Drag and Drop XBlock will be | ||
| ``lib/python3.8/site-packages/drag_and_drop_v2/translations/ar/LC_MESSAGES/text.po``. | ||
|
|
||
| This fallback is used to maintain backwards compatibility with existing | ||
| XBlocks that may or may not be included in the `openedx-translations repo`_. | ||
| Third-party XBlocks that are not included in the | ||
| `xblocks Transifex project`_, such as the `Lime Survey XBlock`_, | ||
| will benefit from this backwards compatibility. | ||
|
|
||
| New ``compile_plugin_js_translations`` command | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| An ``XBlock.i18n_js_namespace`` property will be added for | ||
| the ``compile_plugin_js_translations`` to generate JavaScript translations | ||
| in a centrally managed manner for installed XBlocks. | ||
|
|
||
| A ``compile_plugin_js_translations`` command will loop over XBlock | ||
| modules that has the ``i18n_js_namespace`` | ||
| property set and compile the JavaScript translations via the `compilejsi18n`_ | ||
| command. | ||
|
|
||
| For example if the Drag and Drop XBlock has | ||
| ``i18n_js_namespace = 'DragAndDropI18N'``, the | ||
| ``compile_plugin_js_translations`` command will execute the following | ||
| commands:: | ||
|
|
||
| i18n_tool generate -v # Generate the .mo files | ||
| python manage.py compilejsi18n --namespace DragAndDropI18N --output conf/plugins-locale/drag_and_drop_v2/js/ | ||
|
|
||
|
|
||
| XBlockI18nService support for ``atlas`` JavaScript translations | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| A ``get_javascript_locale_path`` method will be added to the | ||
| ``XBlockI18nService`` to provide XBlocks the | ||
| appropriate path to ``django.js`` translation files. This method | ||
| will allow XBlocks to utilize legacy packaged translations | ||
| or ``atlas``. | ||
|
|
||
| A ``i18n_js_namespace`` property will be added | ||
| to generate JavaScript translations in a centrally managed manner for all | ||
| XBlocks as described in the :ref:`js-translations` section. | ||
|
|
||
| For example, the `Drag and Drop XBlock get_static_i18n_js_url`_ will need to | ||
| be updated to support the new ``XBlockI18nService`` | ||
| ``get_javascript_locale_path`` method and the namespace. | ||
|
|
||
| .. code:: diff | ||
|
|
||
| class DragAndDropBlock(XBlock): | ||
|
|
||
| + i18n_js_namespace = 'DragAndDropI18N' | ||
|
|
||
| @staticmethod | ||
| def _get_statici18n_js_url(): | ||
| """ | ||
| Returns the Javascript translation file for the currently selected language, if any found by | ||
| `pkg_resources` | ||
| """ | ||
| lang_code = translation.get_language() | ||
| if not lang_code: | ||
| return None | ||
|
|
||
| + # TODO: Make this the default once OEP-58 is implemented. | ||
| + if hasattr(self.i18n_service, 'get_javascript_locale_path'): | ||
| + atlas_locale_path = self.i18n_service.get_javascript_locale_path() | ||
| + if atlas_locale_path: | ||
| + return atlas_locale_path | ||
|
|
||
| text_js = 'public/js/translations/{lang_code}/text.js' | ||
| country_code = lang_code.split('-')[0] | ||
| for code in (translation.to_locale(lang_code), lang_code, country_code): | ||
| if pkg_resources.resource_exists(loader.module_name, text_js.format(lang_code=code)): | ||
| return text_js.format(lang_code=code) | ||
| return None | ||
|
|
||
|
|
||
| Dismissed Proposals | ||
| =================== | ||
|
|
||
| XBlocks and plugins have their own "atlas pull" command | ||
| ------------------------------------------------------- | ||
|
|
||
| This dismissed proposal intends to have each XBlock and Plugin have their | ||
| own ``make pull_translations`` and be responsible for managing pulling their | ||
| own translations from the `openedx-translations repo`_. | ||
|
|
||
| This proposal has been dismissed because it would require substantial work | ||
| to get into the details for the ``lib/python3.8/site-packages/`` directory | ||
| and ensure that the ``make pull_translations`` command won't corrupt the | ||
| virtual environment. | ||
|
|
||
| This is a non-trivial task and appears to add more complexity than necessary | ||
| due to the fact that XBlocks and plugins won't be used outside the | ||
| context of ``edx-platform``. | ||
|
|
||
|
|
||
| Goals | ||
| ===== | ||
| #. Use ``atlas pull`` for the ``edx-platform`` repo. | ||
| #. Use ``atlas pull`` for the XBlocks and Plugins. | ||
| #. Allow Tutor and other advanced uses to craft their own ``atlas pull`` | ||
| commands by making the the plugins list available via Django commands. | ||
| #. Allow ``atlas pull`` to use the Python module names instead of the | ||
| repository name of XBlocks and Plugins which is supported via the | ||
| `atlas pull --expand-glob`_ option. | ||
|
|
||
| .. _non-goals: | ||
|
|
||
| Non-Goals | ||
| ========= | ||
|
|
||
| The following are non-goals for this proposal, although some are going to | ||
| be tackled in the future as part of the | ||
| `Translation Management update OEP-58`_ proposal. | ||
|
|
||
| #. Provide a fool-proof method for managing named-release translations. | ||
| This will be a separate discussion. | ||
| #. Discuss the merge/segment strategy of the ``edx-platform``. This is being | ||
| discussed in the | ||
| `decision no. 0018 <https://github.com/openedx/edx-platform/pull/32994>`_. | ||
| #. Design a new XBlock frontend architecture. Instead this proposal works | ||
| with the existing architecture. | ||
| #. Provide a new translation method for theme translations. This will be | ||
| tackled later on. | ||
| #. Provide a new translation method for non-XBlock plugins such as | ||
| ``edx-val``. This will be tackled later on as part of the `OEP-58`_ | ||
| proposal. | ||
|
|
||
| .. _OEP-58 Specifications: https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0058-arch-translations-management.html#specification | ||
| .. _Translation Management update OEP-58: https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0058-arch-translations-management.html#specification | ||
| .. _OEP-58: https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0058-arch-translations-management.html#specification | ||
| .. _openedx-atlas: https://github.com/openedx/openedx-atlas | ||
| .. _openedx-translations repo: https://github.com/openedx/openedx-translations | ||
| .. _extract-translation-source-files.yml: https://github.com/openedx/openedx-translations/blob/2566e0c9a30d033e5dd8d05d4c12601c8e37b4ef/.github/workflows/extract-translation-source-files.yml#L36-L43 | ||
| .. _openedx-translations Transifex project: https://app.transifex.com/open-edx/openedx-translations/dashboard/ | ||
|
|
||
| .. _Approach Memo and Technical Discovery - Translations Infrastructure Implementation: https://docs.google.com/document/d/11dFBCnbdHiCEdZp3pZeHdeH8m7Glla-XbIin7cnIOzU/edit | ||
| .. _Added French (Canada) and Japanese - xblock-drag-and-drop-v2 #220: https://github.com/openedx/xblock-drag-and-drop-v2/pull/220 | ||
| .. _XBlockI18nService: https://github.com/openedx/edx-platform/blob/6e28ba329e0a5354d7264ea834861bf0cae4ceb3/xmodule/modulestore/django.py#L359-L395 | ||
| .. _XBlockI18nService implementation pull request: https://github.com/openedx/edx-platform/pull/11575/files#diff-0bbcc6c13d9bfc9d88fbe2fdf4fd97f6066a7a0f0bfffb82bc942378b7cf33e0R248 | ||
|
|
||
| .. _course-discovery atlas integration: https://github.com/openedx/course-discovery/pull/4037 | ||
| .. _frontend-app-learning atlas integration: https://github.com/openedx/frontend-app-learning/pull/1093 | ||
| .. _edx-platform pull_translations: https://github.com/openedx/edx-platform/blob/0137881b8199701b2af7d07c9a01200e358e3d86/Makefile#L55-L64 | ||
|
|
||
| .. _drag-and-drop-v2 xblock: https://github.com/openedx/xblock-drag-and-drop-v2/ | ||
| .. _LTI Consumer XBlock: https://github.com/openedx/xblock-lti-consumer/ | ||
| .. _edx-val: https://github.com/openedx/edx-val | ||
|
|
||
| .. _LtiConsumerXBlock._get_statici18n_js_url: https://github.com/openedx/xblock-lti-consumer/blob/7a142310a78ac393286c1e9e77c535ea520ab90b/lti_consumer/lti_xblock.py#L663-L677 | ||
| .. _LtiConsumerXBlock.student_view: https://github.com/openedx/xblock-lti-consumer/blob/7a142310a78ac393286c1e9e77c535ea520ab90b/lti_consumer/lti_xblock.py#L1215C24-L1217 | ||
| .. _Drag and Drop XBlock get_static_i18n_js_url: https://github.com/openedx/xblock-drag-and-drop-v2/blob/66e8d3517fe8c0db55c1a3907ff253c2a4562a7e/drag_and_drop_v2/drag_and_drop_v2.py#L318-L332 | ||
|
|
||
| .. _XBlock compiled JavaScript text.js translations: https://github.com/openedx/xblock-drag-and-drop-v2/blob/b8ab1ecd9168ab1dba21f994ee4bfedb6a57d11f/drag_and_drop_v2/public/js/translations/tr/text.js#L3 | ||
| .. _XBlock Makefile compile_translations rule: https://github.com/openedx/xblock-drag-and-drop-v2/blob/66e8d3517fe8c0db55c1a3907ff253c2a4562a7e/Makefile#L41 | ||
| .. _XBlock main JavaScript file: https://github.com/openedx/xblock-drag-and-drop-v2/blob/b8ab1ecd9168ab1dba21f994ee4bfedb6a57d11f/drag_and_drop_v2/public/js/drag_and_drop.js#L6 | ||
|
|
||
|
|
||
| .. _translations/xblock-drag-and-drop-v2 directory: https://github.com/openedx/openedx-translations/tree/8a01424fd8f42e9e76aed34e235c82ab654cdfc5/translations/xblock-drag-and-drop-v2 | ||
| .. _atlas pull --expand-glob: https://github.com/openedx/openedx-atlas/blob/main/docs/decisions/0001-support-glob-pattern.rst | ||
|
|
||
| .. _compilejsi18n: https://django-statici18n.readthedocs.io/en/latest/commands.html#compilejsi18n | ||
| .. _Transifex Translation Memory: https://help.transifex.com/en/articles/6224636-introduction-to-translation-memory | ||
| .. _edx-platform Transifex project: https://www.transifex.com/open-edx/edx-platform/ | ||
| .. _xblocks Transifex project: https://www.transifex.com/open-edx/xblocks/ | ||
|
|
||
| .. _Lime Survey XBlock: https://github.com/eduNEXT/xblock-limesurvey | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe so, yes.