Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to latest nixpkgs-unstable, rustc #4374

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

mmilata
Copy link
Member

@mmilata mmilata commented Nov 21, 2024

Fixes #3337. Replaces #3412.

Reduces the number of different nixpkgs used from 3 to 2, we're keeping the older one for:

  • pyright nodejs environment, it's a PITA to build it with new nodejs
  • protobuf3_19 because legacy is using some internal API that no longer exists in v 21 - needs poetry.lock changes, TBD separately Update protobuf #4394
  • python 3.8 for python tests

MacOS switches from system clang to GCC 14 from nixpkgs, which requires micropython/micropython#14126 backported. Linux keeps GCC 12 due to #4393.

@mmilata mmilata force-pushed the mmilata/nixpkgs-bump2 branch 4 times, most recently from 0916a1d to 20a13b6 Compare November 22, 2024 00:13
@trezor trezor deleted a comment from github-actions bot Nov 22, 2024
@trezor trezor deleted a comment from github-actions bot Nov 22, 2024
Copy link

github-actions bot commented Nov 22, 2024

core UI changes device test click test persistence test
T2T1 Model T test(screens) main(screens) test(screens) main(screens) test(screens) main(screens)
T3B1 Safe 3 test(screens) main(screens) test(screens) main(screens) test(screens) main(screens)
T3T1 Safe 5 test(screens) main(screens) test(screens) main(screens) test(screens) main(screens)
All main(screens)

@mmilata mmilata force-pushed the mmilata/nixpkgs-bump2 branch from 20a13b6 to 6b1aa72 Compare November 22, 2024 00:41
Copy link

github-actions bot commented Nov 22, 2024

legacy UI changes device test(screens) main(screens)

@mmilata mmilata force-pushed the mmilata/nixpkgs-bump2 branch 2 times, most recently from 74ebe54 to edd2bfa Compare November 22, 2024 19:13
@mmilata mmilata marked this pull request as ready for review November 22, 2024 20:06
@cepetr
Copy link
Contributor

cepetr commented Nov 25, 2024

There’s an issue with the Unix build on macOS:

  1. After compiling each C file, I see a new compiler note:
    cc1: note: ‘-g3’ is not supported by the debug linker in use (set to 2)

  2. The emulator crashes:

...
...
REZOR_UDP_PORT=21324
TREZOR_FIDO2_UDP_PORT=21326
TREZOR_SRC=/Users/petrcernin/repo/trezor-firmware/core/src
Waiting for emulator to come up...
1732520747519688 trezor.loop DEBUG finish: <generator object 'bootscreen' at 113086b20>
ERROR: Assert
FILE: qstr.c:133
SHUTDOWN

@cepetr
Copy link
Contributor

cepetr commented Nov 25, 2024

When you run poetry install on macOS you can see this:

Consider moving TOML configuration files to /Users/{user}/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.

and this:

Warning: The current project could not be installed: No file/folder found for package trezor-firmware
If you do not want to install the current project use --no-root.
If you want to use Poetry only for dependency management but not for packaging, you can disable package mode by setting package-mode = false in your pyproject.toml file.
In a future version of Poetry this warning will become an error!

(maybe these're not new)

@mmilata
Copy link
Member Author

mmilata commented Nov 25, 2024

cc1: note: ‘-g3’ is not supported by the debug linker in use (set to 2)

Probably a result of switching from MmacOS clang to NixOS gcc, seems harmless.

The emulator crashes:

Looks like a micropython bug: micropython/micropython#16300, will backport it to our fork later. I guess this is the reason we switched to the system compiler in the first place?

Consider moving TOML configuration files to /Users/{user}/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.

I can't see this one, perhaps I don't have anything in the legacy location wherever it is?

Warning: The current project could not be installed ...

Fixed in 404fa9a.

@mmilata
Copy link
Member Author

mmilata commented Nov 25, 2024

Still, even with the micropython fix device tests crash on something. I'm giving up for now.

``` $ make test_emu /Users/mm/work/trezor-firmware/core/emu.py --disable-animation --headless --output=/Users/mm/work/trezor-firmware/core/../tests/trezor.log --temporary-profile -c pytest --junitxml=/Users/mm/work/trezor-firmware/core/../tests/junit.xml /Users/mm/work/trezor-firmware/core/../tests/device_tests --lang="en" TREZOR_PATH=udp:127.0.0.1:21324 TREZOR_PROFILE_DIR=/private/tmp/nix-shell-45162-0/trezor-emulator-5esd7p_9 TREZOR_UDP_PORT=21324 TREZOR_FIDO2_UDP_PORT=21326 TREZOR_SRC=/Users/mm/work/trezor-firmware/core/src Waiting for emulator to come up... Emulator ready after 0.417 seconds ================================================================================================ test session starts ================================================================================================= platform darwin -- Python 3.10.10, pytest-8.3.2, pluggy-1.5.0 Using --random-order-bucket=module Using --random-order-seed=389422

rootdir: /Users/mm/work/trezor-firmware
configfile: setup.cfg
plugins: xdist-2.5.0, forked-1.4.0, random-order-1.0.4, flaky-3.8.1, timeout-2.1.0, hypothesis-6.36.1, ordering-0.6
collecting ... Fatal Python error: Aborted

Current thread 0x00000002022af840 (most recent call first):
File "/nix/store/r23sw9iy2jdrh59dsq8mic6ahkcijijr-python3-3.10.10/lib/python3.10/ctypes/init.py", line 374 in init
File "/nix/store/r23sw9iy2jdrh59dsq8mic6ahkcijijr-python3-3.10.10/lib/python3.10/ctypes/init.py", line 452 in LoadLibrary
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/bitcoin/core/key.py", line 34 in
File "", line 241 in _call_with_frames_removed
File "", line 883 in exec_module
File "", line 688 in _load_unlocked
File "", line 1006 in _find_and_load_unlocked
File "", line 1027 in _find_and_load
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/bitcoin/wallet.py", line 33 in
File "", line 241 in _call_with_frames_removed
File "", line 883 in exec_module
File "", line 688 in _load_unlocked
File "", line 1006 in _find_and_load_unlocked
File "", line 1027 in _find_and_load
File "/Users/martinmilata/work/trezor-firmware/tests/device_tests/bitcoin/signtx.py", line 8 in
File "", line 241 in _call_with_frames_removed
File "", line 883 in exec_module
File "", line 688 in _load_unlocked
File "", line 1006 in _find_and_load_unlocked
File "", line 1027 in _find_and_load
File "/Users/martinmilata/work/trezor-firmware/tests/device_tests/bitcoin/test_authorize_coinjoin.py", line 29 in
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 174 in exec_module
File "", line 688 in _load_unlocked
File "", line 1006 in _find_and_load_unlocked
File "", line 1027 in _find_and_load
File "", line 1050 in _gcd_import
File "/nix/store/r23sw9iy2jdrh59dsq8mic6ahkcijijr-python3-3.10.10/lib/python3.10/importlib/init.py", line 126 in import_module
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/pathlib.py", line 582 in import_path
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/python.py", line 493 in importtestmodule
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/python.py", line 546 in _getobj
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/python.py", line 284 in obj
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/python.py", line 562 in _register_setup_module_fixture
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/python.py", line 549 in collect
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 389 in collect
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 391 in pytest_make_collect_report
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in call
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 567 in collect_one_node
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/main.py", line 835 in _collect_one_node
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/main.py", line 970 in genitems
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/main.py", line 975 in genitems
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/main.py", line 975 in genitems
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/main.py", line 809 in perform_collect
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/main.py", line 347 in pytest_collection
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in call
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/main.py", line 336 in _main
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/main.py", line 283 in wrap_session
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in call
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/config/init.py", line 175 in main
File "/Users/martinmilata/work/trezor-firmware/.venv/lib/python3.10/site-packages/_pytest/config/init.py", line 201 in console_main
File "/Users/martinmilata/work/trezor-firmware/.venv/bin/pytest", line 8 in

Extension modules: PIL._imaging (total: 1)
Terminating emulator...
Emulator shut down after 0.001 seconds
make: *** [Makefile:191: test_emu] Error 250
(trezor-firmware-py3.10) Mac:core martinmilata$

</details>

@mmilata mmilata marked this pull request as draft November 25, 2024 18:52
@mmilata
Copy link
Member Author

mmilata commented Nov 25, 2024

Ok, this seems to happen on main as well and can be considered out of scope for the PR.

The message WARNING: python is loading libcrypto in an unsafe way preceding the abort() leads to some clues but most of the solutions involve linking/copying libraries to /usr/local/lib which seems deranged and also doesn't work for me.

@mmilata mmilata marked this pull request as ready for review November 25, 2024 23:19
@mmilata
Copy link
Member Author

mmilata commented Nov 26, 2024

Looks like a micropython bug

Turns out I can't read and it's probably a GCC problem. Need to try again with GCC 14 or a clang version from nixpkgs.

@mmilata mmilata marked this pull request as draft November 26, 2024 10:04
@onvej-sl
Copy link
Contributor

The failing valgrind crypto test appears similar to what we dealt with here. I also discovered this is caused by updating gcc12 to gcc14. I will look into it later.

@mmilata
Copy link
Member Author

mmilata commented Nov 27, 2024

Yep, documented what I found in #4393. Not sure how useful is testing this on x86_64 but it probably can catch some cases which are not dependent on the compiler like this one. For the time being I think we can use GCC 14 on macOS only and stay on 12 on Linux (cross compiler is currently GCC 12, with 13 available).

Fixes the following warning:

Warning: The current project could not be installed: No file/folder found for package trezor-firmware
If you do not want to install the current project use --no-root.
If you want to use Poetry only for dependency management but not for packaging, you can disable package mode by setting package-mode = false in your pyproject.toml file.
In a future version of Poetry this warning will become an error!
@mmilata mmilata force-pushed the mmilata/nixpkgs-bump2 branch from 5dcbf9c to de6babf Compare November 27, 2024 15:44
@mmilata mmilata mentioned this pull request Nov 27, 2024
@mmilata mmilata marked this pull request as ready for review November 27, 2024 16:52
@mmilata mmilata requested a review from prusnak as a code owner November 27, 2024 16:52
@mmilata
Copy link
Member Author

mmilata commented Nov 27, 2024

Seems to be in somewhat workable state now.

@cepetr when you have time can you please test on macOS and review the micropython changes?

@mmilata mmilata marked this pull request as draft November 28, 2024 15:22
@mmilata
Copy link
Member Author

mmilata commented Nov 28, 2024

  • ERROR: Assert FILE: qstr.c:133 macOS crash is not fixed by GCC14, i just screwed up the testing, it needs PYOPT=0 to manifest
  • SDL2 update somehow breaks HIGHDPI support on macOS making the emulator unusable
edit: this seems to fix the second issue???
diff --git a/core/embed/io/display/unix/display_driver.c b/core/embed/io/display/unix/display_driver.c
index bdb25c787..e77a1e194 100644
--- a/core/embed/io/display/unix/display_driver.c
+++ b/core/embed/io/display/unix/display_driver.c
@@ -117,7 +117,7 @@ void display_init(display_content_mode_t mode) {
 #ifdef TREZOR_EMULATOR_RASPI
                        SDL_WINDOW_SHOWN | SDL_WINDOW_FULLSCREEN
 #else
-                       SDL_WINDOW_SHOWN | SDL_WINDOW_ALLOW_HIGHDPI
+                       SDL_WINDOW_SHOWN
 #endif
       );
   free(window_title_alloc);

@cepetr
Copy link
Contributor

cepetr commented Dec 2, 2024

Seems to be in somewhat workable state now.

@cepetr when you have time can you please test on macOS and review the micropython changes?

nix-shell --arg devTools true seems to be broken again:

error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:34:12:
           33|
           34|   strict = derivationStrict drvAttrs;
             |            ^
           35|

       … while evaluating derivation 'trezor-firmware-env'
         whose name attribute is located at /nix/store/n1g84klfb0h3bpwyvc59lcy5ca58h36w-source/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'buildInputs' of derivation 'trezor-firmware-env'
         at /nix/store/n1g84klfb0h3bpwyvc59lcy5ca58h36w-source/pkgs/stdenv/generic/make-derivation.nix:383:7:
          382|       depsHostHost                = elemAt (elemAt dependencies 1) 0;
          383|       buildInputs                 = elemAt (elemAt dependencies 1) 1;
             |       ^
          384|       depsTargetTarget            = elemAt (elemAt dependencies 2) 0;

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: Package ‘kcachegrind-23.08.5’ in /nix/store/n1g84klfb0h3bpwyvc59lcy5ca58h36w-source/pkgs/applications/kde/kcachegrind.nix:12 is not available on the requested hostPlatform:
         hostPlatform.config = "aarch64-apple-darwin"
         package.meta.platforms = [
           "aarch64-linux"
           "armv5tel-linux"
           "armv6l-linux"
           "armv7a-linux"
           "armv7l-linux"
           "i686-linux"
           "loongarch64-linux"
           "m68k-linux"
           "microblaze-linux"
           "microblazeel-linux"
           "mips-linux"
           "mips64-linux"
           "mips64el-linux"
           "mipsel-linux"
           "powerpc64-linux"
           "powerpc64le-linux"
           "riscv32-linux"
           "riscv64-linux"
           "s390-linux"
           "s390x-linux"
           "x86_64-linux"
         ]
         package.meta.badPlatforms = [ ]
       , refusing to evaluate.

@mmilata mmilata marked this pull request as ready for review December 2, 2024 20:55
@mmilata
Copy link
Member Author

mmilata commented Dec 2, 2024

nix-shell --arg devTools true seems to be broken again

Fixed in 311906c, please test when you have spare cycles.

Also brought back micropython/micropython#16300 and added questionable fix for emulator image scaling on macOS.

@cepetr
Copy link
Contributor

cepetr commented Dec 9, 2024

nix-shell --arg devTools true seems to be broken again

Fixed in 311906c, please test when you have spare cycles.

It seems, it's ok now. Thanks.

@cepetr
Copy link
Contributor

cepetr commented Dec 9, 2024

cc1: note: ‘-g3’ is not supported by the debug linker in use (set to 2)

Probably a result of switching from macOS clang to NixOS gcc, seems harmless.

Yes, it seems harmless but is very annoying since the message appears for every compiled C file. We should either modify the SCons script for the Unix emulator or investigate the root cause to resolve it.

@cepetr
Copy link
Contributor

cepetr commented Dec 9, 2024

Consider moving TOML configuration files to /Users/{user}/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.

I can't see this one, perhaps I don't have anything in the legacy location wherever it is?

Looks like the problem is specific to my machine. I fixed it by manually moving config.toml to the expected folder.

@cepetr
Copy link
Contributor

cepetr commented Dec 9, 2024

It seems the emulator is still crashing on macOS, but now with a different error than before:

./emu.py
TREZOR_PATH=udp:127.0.0.1:21324
TREZOR_PROFILE_DIR=/private/var/tmp
TREZOR_UDP_PORT=21324
TREZOR_FIDO2_UDP_PORT=21326
TREZOR_SRC=/Users/petrcernin/repo/trezor-firmware/core/src
Waiting for emulator to come up...
Traceback (most recent call last):
  File "/Users/petrcernin/repo/trezor-firmware/core/./emu.py", line 313, in <module>
    cli()
  File "/Users/petrcernin/repo/trezor-firmware/.venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/petrcernin/repo/trezor-firmware/.venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/petrcernin/repo/trezor-firmware/.venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/petrcernin/repo/trezor-firmware/.venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/petrcernin/repo/trezor-firmware/core/./emu.py", line 274, in cli
    emulator.start()
  File "/Users/petrcernin/repo/trezor-firmware/python/src/trezorlib/_internal/emulator.py", line 168, in start
    self.wait_until_ready()
  File "/Users/petrcernin/repo/trezor-firmware/python/src/trezorlib/_internal/emulator.py", line 118, in wait_until_ready
    raise RuntimeError("Emulator process died")
RuntimeError: Emulator process died

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 🔎 Needs review
Development

Successfully merging this pull request may close these issues.

Update nixpkgs
3 participants