Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tests fail on Mac M1 #92

Closed
kaos opened this issue Feb 1, 2023 · 8 comments
Closed

Tests fail on Mac M1 #92

kaos opened this issue Feb 1, 2023 · 8 comments

Comments

@kaos
Copy link
Member

kaos commented Feb 1, 2023

I have some issue running the tests as below.

This on current main 4f06c9d

$ cargo run -p package -- test --tools-pex-mismatch-warn
   Compiling package v0.2.0 (/Users/andreas.stenius/src/github/kaos/scie-pants/package)
    Finished dev [unoptimized + debuginfo] target(s) in 2.18s
     Running `target/debug/package test --tools-pex-mismatch-warn`
 1.) Loading the `ptex` v0.6.0 binary from the cache...
 2.) Loading the `jump` v0.10.0 binary from the cache...
 3.) Executing scie-jump boot-pack of the `pbt` helper binary...
/Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/pbt/lift.json: /Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/pbt/pbt
 4.) Building the scie_pants `tools.pex`...
 5.) Building the scie-pants Rust binary....
  Installing scie-pants v0.3.2 (/Users/andreas.stenius/src/github/kaos/scie-pants)
    Updating crates.io index
   Compiling scie-pants v0.3.2 (/Users/andreas.stenius/src/github/kaos/scie-pants)
    Finished release [optimized] target(s) in 6.67s
   Replacing /Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/bin/scie-pants
    Replaced package `scie-pants v0.3.2 (/Users/andreas.stenius/src/github/kaos/scie-pants)` with `scie-pants v0.3.2 (/Users/andreas.stenius/src/github/kaos/scie-pants)` (executable `scie-pants`)
 6.) Building the `scie-pants` scie...
/Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/scie-pants/lift.json: /Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/scie-pants/scie-pants
 7.) Running smoke tests...
>> Linting, testing and packaging the tools codebase
21:22:27.27 [INFO] Initializing scheduler...
21:22:27.63 [INFO] Scheduler initialized.
21:22:30.20 [INFO] Completed: Format with Black - black made no changes.
21:22:31.19 [INFO] Completed: Format with isort - isort made no changes.

✓ black made no changes.
✓ isort made no changes.

✓ black succeeded.
✓ isort succeeded.
21:22:31.47 [INFO] Completed: Typecheck using MyPy - mypy - mypy succeeded.
Success: no issues found in 13 source files



✓ mypy succeeded.
21:22:31.50 [INFO] Completed: Run Pytest - tools/tests/test_bootstrap_tools.py - succeeded.
21:22:31.50 [INFO] Completed: Run Pytest - tools/tests/test_pants_bootstrap.py - succeeded.

✓ tools/tests/test_bootstrap_tools.py succeeded in 1.08s (cached locally).
✓ tools/tests/test_pants_bootstrap.py succeeded in 0.48s (cached locally).
21:22:31.51 [INFO] Wrote dist/tools/tools.pex
>> Checking .pants.bootstrap handling ignores bash functions
2.15.0rc0
>> Verifying the tools.pex built by the package crate matches the tools.pex built by Pants
The tools.pex generated by Pants does not match ours:
Ours:  /Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/tools.pex
->     50dc7700a3e0372a096b2a425c0cad857b2026224940a18d48ca14cd0edb4369
Pants: /Users/andreas.stenius/src/github/kaos/scie-pants/dist/tools/tools.pex
->     03a341070cfe94b408b90074330ca54fe825f9b2966b0a0320f4e69606fa0717

>> Verifying PANTS_BOOTSTRAP_TOOLS works
python_distribution_hash=64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab pants_version=2.15.0rc0
>> Verifying PANTS_SHA is respected
Bootstrapping Pants 2.14.0a0+git8e381dbf using cpython 3.9.15
Installing pantsbuild.pants==2.14.0a0+git8e381dbf into a virtual environment at /Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.14.0a0+git8e381dbf

[notice] A new release of pip available: 22.3.1 -> 23.0
[notice] To update, run: python -m pip install --upgrade pip
New virtual environment successfully created at /Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.14.0a0+git8e381dbf.
21:22:58.25 [INFO] Initializing scheduler...
21:22:58.40 [INFO] Scheduler initialized.
2.14.0a0+git8e381dbf
>> Verifying --python-repos-repos is used prior to Pants 2.13 (no warnings should be issued by Pants)
Bootstrapping Pants 2.12.1 using cpython 3.9.15
Installing pantsbuild.pants==2.12.1 into a virtual environment at /Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1

[notice] A new release of pip available: 22.3.1 -> 23.0
[notice] To update, run: python -m pip install --upgrade pip
New virtual environment successfully created at /Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1.
Traceback (most recent call last):
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 115, in main
    PantsLoader.main()
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 111, in main
    cls.run_default_entrypoint()
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 93, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 64, in run
    options_bootstrapper = OptionsBootstrapper.create(
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/option/options_bootstrapper.py", line 175, in create
    alias = CliAlias.from_dict(alias_dict)
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/option/alias.py", line 79, in from_dict
    definitions = {key: tuple(shlex.split(value)) for key, value in aliases.items()}
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/option/alias.py", line 79, in <dictcomp>
    definitions = {key: tuple(shlex.split(value)) for key, value in aliases.items()}
  File "/Users/andreas.stenius/Library/Caches/nce/64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab/cpython-3.9.15+20221106-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/shlex.py", line 315, in split
    return list(lex)
  File "/Users/andreas.stenius/Library/Caches/nce/64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab/cpython-3.9.15+20221106-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/shlex.py", line 300, in __next__
    token = self.get_token()
  File "/Users/andreas.stenius/Library/Caches/nce/64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab/cpython-3.9.15+20221106-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/shlex.py", line 109, in get_token
    raw = self.read_token()
  File "/Users/andreas.stenius/Library/Caches/nce/64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab/cpython-3.9.15+20221106-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/shlex.py", line 140, in read_token
    nextchar = self.instream.read(1)
AttributeError: 'dict' object has no attribute 'read'
Error: Command "/Users/andreas.stenius/src/github/kaos/scie-pants/target/debug/build/package-1709cf9c213231d9/out/dist/scie-pants/scie-pants-macos-aarch64" "--no-verify-config" "-V" failed with exit code: Some(1)
STDOUT not captured.
STDERR not captured.
@kaos
Copy link
Member Author

kaos commented Feb 1, 2023

Is there something to clean up to start fresh.. in case I have gotten into a broken state?

@jsirois
Copy link
Contributor

jsirois commented Feb 1, 2023 •
edited
Loading

Well, so, you in particular are an interesting user here. Forgetting the immediate context and focusing on just Pants:

    alias = CliAlias.from_dict(alias_dict)
  File "/Users/andreas.stenius/Library/Caches/nce/b85e31749a229329c058bb98bd5875ae0621ebe38e6e15f1fb89631f449cc327/bindings/venvs/2.12.1/lib/python3.9/site-packages/pants/option/alias.py", line 79, in from_dict
    definitions = {key: tuple(shlex.split(value)) for key, value in aliases.items()}
...
  File "/Users/andreas.stenius/Library/Caches/nce/64dc7e1013481c9864152c3dd806c41144c79d5e9cd3140e185c6a5060bdc9ab/cpython-3.9.15+20221106-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/shlex.py", line 140, in read_token
    nextchar = self.instream.read(1)
AttributeError: 'dict' object has no attribute 'read'

Do you have .pants.rc or some other similar config that might have CLI alias definitions not compatible with Pants 2.12 / exposing an old bug?

@jsirois
Copy link
Contributor

jsirois commented Feb 1, 2023

To answer your question though, no. I've never seen anything like this at all and neither has the Mac ARM CI job.

@jsirois
Copy link
Contributor

jsirois commented Feb 1, 2023

This is Python; so the easiest debug step is to modify the py file in the nce cache directly to print the CLI alias dict just prior to stacktrace. That should help move things forward. Just don't forget to blow away (that portion of) the nce when all debugging is done.

@jsirois
Copy link
Contributor

jsirois commented Feb 1, 2023

IOW: scie-pants is just executing pants from a venv it built earlier under the nce. You can experiment and debug in that venv directly.

@kaos
Copy link
Member Author

kaos commented Feb 1, 2023

Thanks for the pointers @jsirois

Indeed I have custom aliases in a ~/.pants.rc file. I did notice this was in the vicinity of cli aliases, but didn't connect (too late at night again?). Also the issue reported by the unexpected tools.pex checksum had me thinking my setup could be borked.

I'll dig into this 👍🏽

@jsirois
Copy link
Contributor

jsirois commented Feb 1, 2023

Ok, great. FWIW the checksum issue is well known and fully understood: #2

@kaos
Copy link
Member Author

kaos commented Feb 2, 2023

This was caused by my ~/.pants.rc having configuration that was incompatible with pants 2.12.

Adding PANTS_PANTSRC=False globally for all smoke tests in #75

@kaos kaos closed this as completed Feb 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kaos @jsirois