Skip to content

Commit

Permalink
WIP - firmware translations
Browse files Browse the repository at this point in the history
WIP - refactor and extend font generation for non-ascii characters

WIP - add czech characters mapping between UTF8 value and index

WIP - regenerate font files with czech characters

WIP - shorten czech button text, it was causing SHUTDOWN for some reason

WIP - support UTF8 characters in fonts.c

WIP - account for translation in tests

WIP - small fixes

WIP - fix last test

WIP - support UTF8 also in Rust font operations

WIP - add a script to find non-translated english strings in micropython code

WIP - add a validator script for checking missing micropython translations

WIP - translate remaining altcoins and other apps in core (fido, sdcard, TT layouts, ...)

WIP - generate czech glyphs for TT fonts

WIP - modify gen_font.py to account for negative bearing czech characters

WIP - extend translation validation scripts, move them into core/tools

WIP - translate TT layouts in Rust

WIP - fix tests

WIP - fix inverse coloring of nonprintable glyph

WIP - add build and test pipelines for Czech language

WIP - merge both JSON files together

WIP - run new isort

WIP - unify all the translation in Rust, expose to micropython

TEMP - leave en_merged.json file, so it is accessible by translators with old link

WIP - fixes

WIP - add french characters and translation via Google Translator

WIP - skip rustfmt in mako-created files

WIP - revert all the font height changes causing false-positive UI diff

WIP - fixes after rebase

WIP - fix broken translations

WIP - revert some wording changes causing UI diff

WIP - improve validation and translate scripts, translate missing strings

WIP - sort all keys alphabetically

WIP - remove any usage of translation in bootloader

WIP - add newline at the end of JSON file

WIP - fix bitcoin-only strings check

WIP - fix python support check

WIP - add some missing translations

WIP - fix SD card device test

WIP - fix pystyle

WIP - fix rust unittests

WIP - fix click tests

WIP - flag errors in french translations

WIP - add script transferring translations data into a byte blob

WIP - regenerate fr.rs

WIP - store and read language translations from flash

WIP - storing language name in storage

WIP - sending language_data in apply_settings protobuf message

WIP - separate protobuf message for translations, fixes

WIP - set up translations area for TT as well

WIP - get rid of TREZOR_LANG env variable during build

WIP - make the firmware buildable for TT

WIP - add basic device tests

WIP - set language for tests

WIP - counting with language when writing fixtures

WIP - add todos

WIP - fix CI

WIP - unify translations, make titles CAPITAL

WIP - translate missing english

WIP - skip translations messages for T1

WIP - not changing tests names for english

WIP - fix flake8

WIP - no test language setting for T1

WIP - clippy lint about complex data type

WIP - fix some english UI diff for TR

WIP - fix cstyle

WIP - minimize the usage of #[cfg(feature = "micropython")] outside translations module

WIP - minimize TT's UI diff

WIP - fix ruststyle

WIP - fix TR build

WIP - advanced Shamir text change

WIP - storing the language name as the first item in the translation data

WIP - modify and extend tests after storing language name

WIP - modify checklist sentence

WIP - add TEST_LANG into Makefile for all the emu tests

WIP - default arguments

WIP - reimplement default arguments

remove unneeded pub from get_info function

WIP - Rust handling of object attributes lookups from upy - thanks Matejcik!

WIP - generate mock interface for attribute-based translations lookups

WIP - change function calls to object attributes

WIP - symbolic link for unix/translations.c

WIP - fix and improve the reading of translations - thanks Matejcik!

WIP - add support for multiple languages in removing missing tests

WIP - fix multiple-accounts warning in tests

WIP - fix encoding of newlines in translations

WIP - fix czech tutorial text

WIP - fix czech click tests

WIP - do not translate wire error messages

WIP - add language options to click tests as well

WIP - setup czech device tests in CI

WIP - setup czech click tests in CI

WIP - record czech device tests for TR

WIP - record czech click tests for TR

WIP - record czech device tests for TT

WIP - record czech click tests for TT

WIP - pystyle

WIP - cstyle

WIP - fix Rust micropython import dependency

WIP - fix czech recordings

WIP - support french translations in tests

WIP - shorten some french words to fix the tests

WIP - fix micropython cfg compilation

WIP - record french click tests for TR

WIP - record french device tests for TR

WIP - record french device tests for TT

WIP - record french click tests for TT

WIP - fix french translations - shorten them

WIP - translate missing french words

WIP - fix click tests

WIP - add french tests into CI

WIP - pystyle

WIP - allow for czech/french tests in update script

WIP - update czech fixtures

WIP - update french fixtures

WIP - ruststyle

WIP - disallow MPU to run it on hardware

WIP - cstyle

WIP - change translations delimiter from * to \x00

WIP - change translations protobufs

WIP - remove language handling from storage

WIP - add header into JSON files

WIP - count with header in translations blob

WIP - yml style fixes

WIP - fix proto gen

WIP - verify version and data hash

WIP - fix loading test translations

feat(core): allow access to translations area in firmware

[no changelog]

WIP - fixes after rebase

WIP - increase the TT's translations area to 3 sectors

WIP - dynamically read the maximum translations size

WIP - record non-english tests from CI

WIP - loading font data from translations blob

WIP - bump translations version

WIP - include czech and french glyph data

WIP - whitelist another negative-bearing glyph

WIP - remove czech/french glyphs from common font files

WIP - fix language tests

WIP - specific fonts for specific models

WIP - revert the non-ascii font hardcoding

WIP - include missing BIG font into nonprintable logic

WIP - minor Rust code improvements

WIP - include newlines at the end of json files

WIP - move glyph Rust function to librust_fonts.h

WIP - add all fonts into translations file

WIP - move fonts into its own dir

WIP - reflect separate dir for fonts

WIP - not putting translations trezorhal into bootloader

WIP - write and read multiple fonts into translations data

WIP - silence pyright issue/notissue

WIP - delete no more used translations/*.py imports

WIP - fix bootloader builds by introducing translations feature and TRANSLATIONS flag

WIP - fix TT's bootloader Rust build

WIP - fix tests in non-english languages

WIP - not search for UTF-8 when there are no translations data

WIP - add colons to strings where missing

WIP - fix language loading in tests

WIP - fix signmessage input flow to work in all languages

WIP - create offset table for translation strings

WIP - code improvements

WIP - record foreign language fixtures + sync with main in english

WIP - do alignment check before reading u16 data

WIP - allocate blob in RAM for translations data

WIP - add TODO for blob generation

WIP - record non-english device tests

WIP - use bytes.align_to instead of messing with pointers

WIP - fixtures

WIP - remove unused import

WIP - add order.py

WIP - add order.json

WIP - take order.json into account in creating general.rs

WIP - take order.json into account in generating the blob

WIP - style

WIP - sort the language files

WIP - remove unused file

WIP - code improvements

WIP - add TODO for homescreen notification

WIP - translate plural forms

WIP - translate time intervals

WIP - sign translations with dev keys, validate signatures, improve robustness

WIP - improve tests for translations

WIP - add `trezorctl utils sign-translations` for production signing of the blob

WIP - pyright fix

WIP - changing TR progress loader offset - it was colliding with title

WIP - show indeterminate loader when loading translations data

WIP - record new and updated language tests

WIP - show the change language title/prompt in the target language

WIP - sort keys

WIP - add crowdin-cli into shell.nix

WIP - add crowdin sync script
  • Loading branch information
grdddj committed Jan 2, 2024
1 parent c09f9e8 commit 7f1a5ac
Show file tree
Hide file tree
Showing 238 changed files with 20,236 additions and 2,475 deletions.
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,6 @@ vendorheader: ## generate vendor header
vendorheader_check: ## check that vendor header is up to date
./core/embed/vendorheader/generate.sh --quiet --check

gen: mocks icons templates protobuf ci_docs vendorheader solana_templates ## regenerate auto-generated files from sources
gen: templates mocks icons protobuf ci_docs vendorheader solana ## regenerate auto-generated files from sources

gen_check: mocks_check icons_check templates_check protobuf_check ci_docs_check vendorheader_check solana_templates_check ## check validity of auto-generated files
gen_check: templates_check mocks_check icons_check protobuf_check ci_docs_check vendorheader_check solana_templates_check ## check validity of auto-generated files
1 change: 1 addition & 0 deletions ci/shell.nix
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ stdenvNoCC.mkDerivation ({
bash
bloaty # for binsize
check
crowdin-cli # for translations
curl # for connect tests
editorconfig-checker
gcc-arm-embedded
Expand Down
252 changes: 252 additions & 0 deletions ci/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,70 @@ core device test:
reports:
junit: tests/junit.xml

core device test czech:
stage: test
<<: *gitlab_caching
needs:
- core unix frozen debug build
variables:
TREZOR_PROFILING: 1 # so that we get coverage data
MULTICORE: 4 # more could interfere with other jobs
TEST_LANG: "cs" # czech
script:
- $NIX_SHELL --run "poetry run make -C core test_emu_ui_multicore | ts -s"
after_script:
- mv core/src/.coverage.* core # there will be more coverage files (one per core)
- mv tests/ui_tests/reports/test/ test_ui_report
- $NIX_SHELL --run "poetry run python ci/prepare_ui_artifacts.py | ts -s"
- diff -u tests/ui_tests/fixtures.json tests/ui_tests/fixtures.suggestion.json
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_SHORT_SHA"
paths:
- ci/ui_test_records/
- test_ui_report
- tests/ui_tests/screens/
- tests/ui_tests/fixtures.suggestion.json
- tests/ui_tests/fixtures.results.json
- tests/junit.xml
- tests/trezor.log
- core/.coverage.*
when: always
expire_in: 1 week
reports:
junit: tests/junit.xml

core device test french:
stage: test
<<: *gitlab_caching
needs:
- core unix frozen debug build
variables:
TREZOR_PROFILING: 1 # so that we get coverage data
MULTICORE: 4 # more could interfere with other jobs
TEST_LANG: "fr" # french
script:
- $NIX_SHELL --run "poetry run make -C core test_emu_ui_multicore | ts -s"
after_script:
- mv core/src/.coverage.* core # there will be more coverage files (one per core)
- mv tests/ui_tests/reports/test/ test_ui_report
- $NIX_SHELL --run "poetry run python ci/prepare_ui_artifacts.py | ts -s"
- diff -u tests/ui_tests/fixtures.json tests/ui_tests/fixtures.suggestion.json
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_SHORT_SHA"
paths:
- ci/ui_test_records/
- test_ui_report
- tests/ui_tests/screens/
- tests/ui_tests/fixtures.suggestion.json
- tests/ui_tests/fixtures.results.json
- tests/junit.xml
- tests/trezor.log
- core/.coverage.*
when: always
expire_in: 1 week
reports:
junit: tests/junit.xml

core device R test:
stage: test
<<: *gitlab_caching
Expand Down Expand Up @@ -114,6 +178,72 @@ core device R test:
reports:
junit: tests/junit.xml

core device R test czech:
stage: test
<<: *gitlab_caching
needs:
- core unix frozen R debug build
variables:
TREZOR_PROFILING: "1"
TREZOR_MODEL: "R"
MULTICORE: 4 # more could interfere with other jobs
TEST_LANG: "cs" # czech
script:
- $NIX_SHELL --run "poetry run make -C core test_emu_ui_multicore | ts -s"
after_script:
- mv core/src/.coverage.* core # there will be more coverage files (one per core)
- mv tests/ui_tests/reports/test/ test_ui_report
- $NIX_SHELL --run "poetry run python ci/prepare_ui_artifacts.py | ts -s"
- diff -u tests/ui_tests/fixtures.json tests/ui_tests/fixtures.suggestion.json
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_SHORT_SHA"
paths:
- ci/ui_test_records/
- test_ui_report
- tests/ui_tests/screens/
- tests/ui_tests/fixtures.suggestion.json
- tests/ui_tests/fixtures.results.json
- tests/junit.xml
- tests/trezor.log
- core/.coverage.*
when: always
expire_in: 1 week
reports:
junit: tests/junit.xml

core device R test french:
stage: test
<<: *gitlab_caching
needs:
- core unix frozen R debug build
variables:
TREZOR_PROFILING: "1"
TREZOR_MODEL: "R"
MULTICORE: 4 # more could interfere with other jobs
TEST_LANG: "fr" # french
script:
- $NIX_SHELL --run "poetry run make -C core test_emu_ui_multicore | ts -s"
after_script:
- mv core/src/.coverage.* core # there will be more coverage files (one per core)
- mv tests/ui_tests/reports/test/ test_ui_report
- $NIX_SHELL --run "poetry run python ci/prepare_ui_artifacts.py | ts -s"
- diff -u tests/ui_tests/fixtures.json tests/ui_tests/fixtures.suggestion.json
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_SHORT_SHA"
paths:
- ci/ui_test_records/
- test_ui_report
- tests/ui_tests/screens/
- tests/ui_tests/fixtures.suggestion.json
- tests/ui_tests/fixtures.results.json
- tests/junit.xml
- tests/trezor.log
- core/.coverage.*
when: always
expire_in: 1 week
reports:
junit: tests/junit.xml

core device asan test:
stage: test
<<: *gitlab_caching
Expand Down Expand Up @@ -327,6 +457,68 @@ core click test:
expire_in: 1 week
when: always

core click test czech:
stage: test
<<: *gitlab_caching
needs:
- core unix frozen debug build
variables:
TREZOR_PROFILING: 1
TEST_LANG: "cs" # czech
script:
- $NIX_SHELL --run "poetry run make -C core test_emu_click_ui | ts -s"
after_script:
- mv core/src/.coverage core/.coverage.test_click
- mv tests/ui_tests/reports/test/ test_ui_report
- $NIX_SHELL --run "poetry run python ci/prepare_ui_artifacts.py | ts -s"
- diff -u tests/ui_tests/fixtures.json tests/ui_tests/fixtures.suggestion.json
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_SHORT_SHA"
paths:
- ci/ui_test_records/
- test_ui_report
- tests/ui_tests/screens/
- tests/ui_tests/fixtures.suggestion.json
- tests/ui_tests/fixtures.results.json
- tests/trezor.log
- tests/junit.xml
- core/.coverage.*
reports:
junit: tests/junit.xml
expire_in: 1 week
when: always

core click test french:
stage: test
<<: *gitlab_caching
needs:
- core unix frozen debug build
variables:
TREZOR_PROFILING: 1
TEST_LANG: "fr" # french
script:
- $NIX_SHELL --run "poetry run make -C core test_emu_click_ui | ts -s"
after_script:
- mv core/src/.coverage core/.coverage.test_click
- mv tests/ui_tests/reports/test/ test_ui_report
- $NIX_SHELL --run "poetry run python ci/prepare_ui_artifacts.py | ts -s"
- diff -u tests/ui_tests/fixtures.json tests/ui_tests/fixtures.suggestion.json
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_SHORT_SHA"
paths:
- ci/ui_test_records/
- test_ui_report
- tests/ui_tests/screens/
- tests/ui_tests/fixtures.suggestion.json
- tests/ui_tests/fixtures.results.json
- tests/trezor.log
- tests/junit.xml
- core/.coverage.*
reports:
junit: tests/junit.xml
expire_in: 1 week
when: always

# Click tests.
# See [docs/tests/click-tests](../tests/click-tests.md) for more info.
core click R test:
Expand Down Expand Up @@ -358,6 +550,66 @@ core click R test:
expire_in: 1 week
when: always

core click R test czech:
stage: test
<<: *gitlab_caching
needs:
- core unix frozen R debug build
variables:
TREZOR_PROFILING: 1
TEST_LANG: "cs" # czech
script:
- nix-shell --run "poetry run make -C core test_emu_click_ui | ts -s"
after_script:
- mv core/src/.coverage core/.coverage.test_click
- mv tests/ui_tests/reports/test/ test_ui_report
- nix-shell --run "poetry run python ci/prepare_ui_artifacts.py | ts -s"
- diff -u tests/ui_tests/fixtures.json tests/ui_tests/fixtures.suggestion.json
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_SHORT_SHA"
paths:
- ci/ui_test_records/
- test_ui_report
- tests/ui_tests/screens/
- tests/ui_tests/fixtures.suggestion.json
- tests/ui_tests/fixtures.results.json
- tests/trezor.log
- tests/junit.xml
reports:
junit: tests/junit.xml
expire_in: 1 week
when: always

core click R test french:
stage: test
<<: *gitlab_caching
needs:
- core unix frozen R debug build
variables:
TREZOR_PROFILING: 1
TEST_LANG: "fr" # french
script:
- nix-shell --run "poetry run make -C core test_emu_click_ui | ts -s"
after_script:
- mv core/src/.coverage core/.coverage.test_click
- mv tests/ui_tests/reports/test/ test_ui_report
- nix-shell --run "poetry run python ci/prepare_ui_artifacts.py | ts -s"
- diff -u tests/ui_tests/fixtures.json tests/ui_tests/fixtures.suggestion.json
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_SHORT_SHA"
paths:
- ci/ui_test_records/
- test_ui_report
- tests/ui_tests/screens/
- tests/ui_tests/fixtures.suggestion.json
- tests/ui_tests/fixtures.results.json
- tests/trezor.log
- tests/junit.xml
reports:
junit: tests/junit.xml
expire_in: 1 week
when: always

core click asan test:
stage: test
<<: *gitlab_caching
Expand Down
33 changes: 32 additions & 1 deletion common/protob/messages-management.proto
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ message EndSession {
}

/**
* Request: change language and/or label of the device
* Request: change some property of the device, e.g. label or homescreen
* @start
* @next Success
* @next Failure
Expand All @@ -180,6 +180,37 @@ message ApplySettings {
optional bool hide_passphrase_from_host = 11; // do not show passphrase coming from host
}

/**
* Request: change the device language via translation data
* Does not send the translation data itself, as they are too large for one message.
* Device will request the translation data in chunks.
* @start
* @next TranslationDataRequest
* @next Failure
*/
message ChangeLanguage {
required uint32 data_length = 1; // byte length of the whole translation blob (set to 0 for default language)
}

/**
* Response: Device asks for more data from transaction payload.
* @end
* @next TranslationDataAck
*/
message TranslationDataRequest {
required uint32 data_length = 1; // Number of bytes being requested (<= 1024)
required uint32 data_offset = 2; // Offset of the first byte being requested
}

/**
* Request: Translation payload data.
* @next TranslationDataRequest
* @next Success
*/
message TranslationDataAck {
required bytes data_chunk = 1; // Bytes from translation payload (<= 1024 bytes)
}

/**
* Request: set flags of the device
* @start
Expand Down
3 changes: 3 additions & 0 deletions common/protob/messages.proto
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,9 @@ enum MessageType {
MessageType_UnlockBootloader = 96 [(bitcoin_only) = true, (wire_in) = true];
MessageType_AuthenticateDevice = 97 [(bitcoin_only) = true, (wire_out) = true];
MessageType_AuthenticityProof = 98 [(bitcoin_only) = true, (wire_in) = true];
MessageType_ChangeLanguage = 990 [(bitcoin_only) = true, (wire_in) = true];
MessageType_TranslationDataRequest = 991 [(bitcoin_only) = true, (wire_out) = true];
MessageType_TranslationDataAck = 992 [(bitcoin_only) = true, (wire_in) = true];

MessageType_SetU2FCounter = 63 [(wire_in) = true];
MessageType_GetNextU2FCounter = 80 [(wire_in) = true];
Expand Down
5 changes: 5 additions & 0 deletions common/tools/cointool.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,10 @@ def ascii_filter(s: str) -> str:
return re.sub("[^ -\x7e]", "_", s)


def utf8_str_filter(s: str) -> str:
return '"' + repr(s)[1:-1] + '"'


def make_support_filter(
support_info: SupportInfo,
) -> Callable[[str, Coins], Iterator[Coin]]:
Expand All @@ -126,6 +130,7 @@ def supported_on(device: str, coins: Coins) -> Iterator[Coin]:


MAKO_FILTERS = {
"utf8_str": utf8_str_filter,
"c_str": c_str_filter,
"ascii": ascii_filter,
"black_repr": black_repr_filter,
Expand Down
Loading

0 comments on commit 7f1a5ac

Please sign in to comment.