Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OTP25 bus error on Mac OS Sonoma #7687

Closed
rafaelgaspar opened this issue Sep 26, 2023 · 34 comments
Closed

OTP25 bus error on Mac OS Sonoma #7687

rafaelgaspar opened this issue Sep 26, 2023 · 34 comments
Assignees
Labels
bug Issue is reported as a bug team:VM Assigned to OTP team VM

Comments

@rafaelgaspar
Copy link

rafaelgaspar commented Sep 26, 2023

Describe the bug
This have been known for a while since the beta, but I don’t see any issue here and the final version was released today.

To Reproduce
On Mac OS Sonoma, run erlc and you get a bus error.

While rebuilding it from source with XCode 15, it also fails in the same manner, after building erlc it fails running it to build ebin’s.

Expected behavior
It would be nice to know if there is any plans of fixing this for OTP25, or if only OTP26 will be supported going forward.

Affected versions
As far as I’m aware all versions below OTP26.

Additional context
https://elixirforum.com/t/bus-error-after-upgrading-to-sonoma-beta/56354?page=2
kerl/kerl#450

@rafaelgaspar rafaelgaspar added the bug Issue is reported as a bug label Sep 26, 2023
@carflynn2009
Copy link

I am getting the same issue installing Erlang OTP 25 using asdf

To Reproduce
On Mac OS Sonoma run asdf install erlang 25.0.4

If you already had OTP 25 before upgrade you get bus error if you run iex and if you remove and try to reinstall the asdf build fails

Expected behavior
It would be nice to know if there is any plans of fixing this for OTP25, or if only OTP26 will be supported going forward.

Affected versions
As far as I’m aware all versions below OTP26.

@westonlit
Copy link

+1

I didn't know about the issue with macOS Sonoma, and I installed it today and cannot run apps using OTP 25 or build it from source.

@starbelly
Copy link
Contributor

starbelly commented Sep 27, 2023

This is fixed on maint. You can confirm this with asdf install erlang ref:maint (or compile as you normally would).

Specifically, this was fixed up per 82dd4a5 and 291f3ed .

To note, I tested OTP 24, no issues there, only with OTP 25.

@IngelaAndin IngelaAndin added the team:VM Assigned to OTP team VM label Sep 27, 2023
@carflynn2009
Copy link

@starbelly does this fix only cover 25.0? I will give this a test and see if that fixes my issue

@carflynn2009
Copy link

This is fixed on maint. You can confirm this with asdf install erlang ref:maint (or compile as you normally would).

Specifically, this was fixed up per 82dd4a5 and 291f3ed .

To note, I tested OTP 24, no issues there, only with OTP 25.

I have successfully been able to install ref:maint using asdf but how to i use a specific version of OTP25 that my asdf tool-versions require?

25.0.4 to be exact but any version of 25 will do

After i install ref:maint and run erl i get OTP26

@rafaelgaspar
Copy link
Author

I have successfully been able to install ref:maint using asdf but how to i use a specific version of OTP25 that my asdf tool-versions require?

That's not possible maint is the branch containing the latest released commits(see it as similar to a production branch), while the OTP25 lives in maint-25, so it's two separate branches. It's not an option you pass when compiling.

Specifically, this was fixed up per 82dd4a5 and 291f3ed .

@starbelly Do you know if those are cherry-pickable into maint-25? I'll give it a try anyway.

@carflynn2009
Copy link

I have successfully been able to install ref:maint using asdf but how to i use a specific version of OTP25 that my asdf tool-versions require?

That's not possible maint is the branch containing the latest released commits(see it as similar to a production branch), while the OTP25 lives in maint-25, so it's two separate branches. It's not an option you pass when compiling.

Specifically, this was fixed up per 82dd4a5 and 291f3ed .

@starbelly Do you know if those are cherry-pickable into maint-25? I'll give it a try anyway.

maint will not help me in that case as I have always been able to install OTP26 but I need OTP25

@rafaelgaspar
Copy link
Author

25.0.4 to be exact but any version of 25 will do

@carflynn2009 I doubt they will do a new release for each non-major version, if this is fixed, you will probably get only on the latest OTP25 version. But compatibility wise, that should be good enough for almost everyone.

@carflynn2009
Copy link

25.0.4 to be exact but any version of 25 will do

@carflynn2009 I doubt they will do a new release for each non-major version, if this is fixed, you will probably get only on the latest OTP25 version. But compatibility wise, that should be good enough for almost everyone.

Yes latest OTP25 will work for what I need, hopefully this can get cherry picked into that version

@rafaelgaspar
Copy link
Author

rafaelgaspar commented Sep 27, 2023

Specifically, this was fixed up per 82dd4a5 and 291f3ed .

I just checked and both commits changes are present in maint-25.

But still I can't compile it.

@carflynn2009
Copy link

Specifically, this was fixed up per 82dd4a5 and 291f3ed .

I just checked and both commits changes are present in maint-25.

But still I can't compile it.

+1

Does not compile for me either

@garazdawi
Copy link
Contributor

We are not as eager to update our MacOS you guys, so we don't have anyway to test this yet. Can you do a git bisect start --first-parent maint OTP-25.3.2 and see which commit fixes the problem?

@Hermanverschooten
Copy link

I ran into the same issues just now. erlang 26 is no problem, all 25 are failing with a Bus error, reinstalling fails. I am using rtx, rtx install erlang@ref:maint works, rtx install erlang@ref:main-25 crashes with

MiniM2➜  api git:(main) ✗ rtx install erlang@ref:maint-25
make[3]: *** Waiting for unfinished jobs....
/bin/sh: line 1: 57453 Bus error: 10           erlc -W -Werror +debug_info -I/Users/herman/.local/share/rtx/plugins/erlang/kerl-home/builds/asdf_maint-25/otp_src_git/lib/stdlib/include -Werror -o../ebin leex.erl
make[3]: *** [../ebin/leex.beam] Error 138
/bin/sh: line 1: 57454 Bus error: 10           erlc -W -Werror +debug_info -I/Users/herman/.local/share/rtx/plugins/erlang/kerl-home/builds/asdf_maint-25/otp_src_git/lib/stdlib/include -Werror -o../ebin yeccparser.erl
make[3]: *** [../ebin/yeccparser.beam] Error 138
/bin/sh: line 1: 57455 Bus error: 10           erlc -W -Werror +debug_info -I/Users/herman/.local/share/rtx/plugins/erlang/kerl-home/builds/asdf_maint-25/otp_src_git/lib/stdlib/include -Werror -o../ebin yeccscan.erl
make[3]: *** [../ebin/yeccscan.beam] Error 138
make[2]: *** [opt] Error 2
make[1]: *** [opt] Error 2
make: *** [secondary_bootstrap_build] Error 2
APPLICATIONS DISABLED (See: /Users/herman/.local/share/rtx/plugins/erlang/kerl-home/builds/asdf_maint-25/otp_build_git.log)
 * jinterface     : Java compiler disabled by user
 * odbc           : User gave --without-odbc option

APPLICATIONS INFORMATION (See: /Users/herman/.local/share/rtx/plugins/erlang/kerl-home/builds/asdf_maint-25/otp_build_git.log)
 * wx             : Can not link wx program are all developer packages installed?

make[3]: *** Waiting for unfinished jobs....
/bin/sh: line 1: 57453 Bus error: 10           erlc -W -Werror +debug_info -I/Users/herman/.local/share/rtx/plugins/erlang/kerl-home/builds/asdf_maint-25/otp_src_git/lib/stdlib/include -Werror -o../ebin leex.erl
make[3]: *** [../ebin/leex.beam] Error 138
/bin/sh: line 1: 57454 Bus error: 10           erlc -W -Werror +debug_info -I/Users/herman/.local/share/rtx/plugins/erlang/kerl-home/builds/asdf_maint-25/otp_src_git/lib/stdlib/include -Werror -o../ebin yeccparser.erl
make[3]: *** [../ebin/yeccparser.beam] Error 138
/bin/sh: line 1: 57455 Bus error: 10           erlc -W -Werror +debug_info -I/Users/herman/.local/share/rtx/plugins/erlang/kerl-home/builds/asdf_maint-25/otp_src_git/lib/stdlib/include -Werror -o../ebin yeccscan.erl
make[3]: *** [../ebin/yeccscan.beam] Error 138
make[2]: *** [opt] Error 2
make[1]: *** [opt] Error 2
make: *** [secondary_bootstrap_build] Error 2
rtx erlang@ref:maint-25 [ERROR] /Users/herman/.local/share/rtx/plugins/erlang/bin/install failed                                                                                                                                                                                                                                                                                 ✗ 44srtx ~/.local/share/rtx/plugins/erlang/bin/install exited with non-zero status: exit code 1
rtx Run with RTX_DEBUG=1 for more information

@rafaelgaspar
Copy link
Author

rafaelgaspar commented Sep 27, 2023

@garazdawi This is the result of the git bisect:

b0c30263a0ef698244d898a95c9b9a40e67eac9c is the first bad commit
commit b0c30263a0ef698244d898a95c9b9a40e67eac9c
Author: Henrik Nord <henrik@erlang.org>
Date:   Wed May 18 09:57:03 2022 +0200

    Prepare for development

 OTP_VERSION              |   2 +-
 make/otp_version_tickets | 191 +----------------------------------------------
 2 files changed, 2 insertions(+), 191 deletions(-)
bisect found first bad commit

Not sure what to do with that though.

@kpy3
Copy link
Contributor

kpy3 commented Sep 27, 2023

@garazdawi This is result of the git bisect:

b0c30263a0ef698244d898a95c9b9a40e67eac9c is the first bad commit
commit b0c30263a0ef698244d898a95c9b9a40e67eac9c
Author: Henrik Nord <henrik@erlang.org>
Date:   Wed May 18 09:57:03 2022 +0200

    Prepare for development

 OTP_VERSION              |   2 +-
 make/otp_version_tickets | 191 +----------------------------------------------
 2 files changed, 2 insertions(+), 191 deletions(-)
bisect found first bad commit

Not sure what to do with that though.

Hm, looks like there is no code changes in that commit... 🤔

@Sgoettschkes
Copy link

Shouldn't git bisect take the bad commit ref first? So it would be git bisect start --first-parent OTP-25.3.2 maint?

@garazdawi
Copy link
Contributor

Odd, try it without the --first-parent option and see if that work better.

Also as @Sgoettschkes noticed, the order of good and bad are switched because (in my git) bad has to be after good. So when doing bisect you need to switch the meaning of good and bad so that good = broken and bad = works.

@rafaelgaspar
Copy link
Author

rafaelgaspar commented Sep 27, 2023

Shouldn't git bisect take the bad commit ref first? So it would be git bisect start --first-parent OTP-25.3.2 maint?

$ git bisect start --first-parent OTP-25.3.2 maint
Some good revs are not ancestors of the bad rev.
git bisect cannot work properly in this case.
Maybe you mistook good and bad revs?

Odd, try it without the --first-parent option and see if that work better.

Already tried, same commit is returned.

@rafaelgaspar
Copy link
Author

I see my mistake, I also need to invert the test script, running it now with !compile.

Also, somewhere in the middle of these commits the error message changes from bus error to Internal error: jit: Cannot allocate executable memory. Use the interpreter instead.

=== Entering application parsetools
erlc -W  -Werror +debug_info -I/Users/rafael.gaspar/Projects/ThirdParty/otp/lib/stdlib/include -Werror -o../ebin leex.erl
beam/jit/beam_jit_main.cpp:190:pick_allocator(): Internal error: jit: Cannot allocate executable memory. Use the interpreter instead.
make[3]: *** [../ebin/leex.beam] Abort trap: 6
make[2]: *** [opt] Error 2
make[1]: *** [opt] Error 2
make: *** [secondary_bootstrap] Error 2

@rafaelgaspar
Copy link
Author

So this is how a renamed things:

git bisect start --term-new=fixed --term-old=unfixed --first-parent maint OTP-25.3.2

And this is resulting commit:

3508ea39f8d144cb78bda5a2c326b4b6a9f0ce58 is the first fixed commit
commit 3508ea39f8d144cb78bda5a2c326b4b6a9f0ce58
Merge: 69f95cb444 69cd11ae60
Author: John Högberg <john@erlang.org>
Date:   Fri Mar 17 12:22:00 2023 +0100

    Merge pull request #7018 from jhogberg/john/erts/refactor-code-modification/OTP-18513

    erts: Refactor code modification

 erts/emulator/beam/beam_bif_load.c             |  22 +-
 erts/emulator/beam/beam_bp.c                   | 283 ++++++++++++++++++-------
 erts/emulator/beam/beam_bp.h                   |   7 +-
 erts/emulator/beam/beam_debug.c                |   7 +-
 erts/emulator/beam/emu/emu_load.c              |   2 +
 erts/emulator/beam/erl_bif_trace.c             |  32 ++-
 erts/emulator/beam/erl_lock_check.c            |   2 +-
 erts/emulator/beam/erl_nif.c                   | 144 ++++++++-----
 erts/emulator/beam/jit/arm/beam_asm.hpp        |   6 +-
 erts/emulator/beam/jit/arm/beam_asm_global.cpp |  12 +-
 erts/emulator/beam/jit/asm_load.c              |  92 ++++----
 erts/emulator/beam/jit/beam_asm.h              |  27 ++-
 erts/emulator/beam/jit/beam_jit_common.cpp     |  28 ++-
 erts/emulator/beam/jit/beam_jit_main.cpp       |  54 +++--
 erts/emulator/beam/jit/load.h                  |   4 +-
 erts/emulator/beam/jit/x86/beam_asm.hpp        |   6 +-
 erts/emulator/beam/jit/x86/beam_asm_global.cpp |  12 +-
 erts/emulator/beam/module.c                    |  91 ++++----
 erts/emulator/beam/module.h                    |  28 ++-
 erts/emulator/internal_doc/BeamAsm.md          |  14 +-
 erts/emulator/test/jit_SUITE.erl               |  18 +-
 21 files changed, 567 insertions(+), 324 deletions(-)
bisect found first bad commit

@garazdawi
Copy link
Contributor

Seems like the way we allocate JIT code in 25 has been disallowed in MacOS Sonoma. I don't think we can just apply the commit you found to 25 as that would also introduce other changes that I'm not sure we want on the 25 track.

What happens if you set ERL_FLAGS="+JPperf map" before building?

If you want to have a quick fix so that you can get on with your work you can disable the jit when compiling by setting: KERL_CONFIGURE_OPTIONS="--disable-jit".

@Sgoettschkes
Copy link

If you want to have a quick fix so that you can get on with your work you can disable the jit when compiling by setting: KERL_CONFIGURE_OPTIONS="--disable-jit".

I can confirm that this is working with asdf. Thanks for unblocking me (and others) 😊!

@rafaelgaspar
Copy link
Author

What happens if you set ERL_FLAGS="+JPperf map" before building?

=== Entering application parsetools
erlc -W  -Werror +debug_info -I/Users/rafael.gaspar/Projects/ThirdParty/otp/lib/stdlib/include -Werror -o../ebin leex.erl
+JPperf is not supported on this platform

If you want to have a quick fix so that you can get on with your work you can disable the jit when compiling by setting: KERL_CONFIGURE_OPTIONS="--disable-jit".

Using this:

export KERL_CONFIGURE_OPTIONS="--disable-jit"
asdf install erlang 25.3.2.6

Does compile it fine.

Thanks for the help, and let me know if you need help further investigating the root cause and possible solution for it without disabling JIT.

@jhogberg
Copy link
Contributor

Thanks for the help, and let me know if you need help further investigating the root cause and possible solution for it without disabling JIT.

Sure, can you check if the problem comes back when you remove this line on maint?

@rafaelgaspar
Copy link
Author

Sure, can you check if the problem comes back when you remove this line on maint?

I commented that line in maint(1e09a38) and unfortunately it builds fine.

@starbelly
Copy link
Contributor

starbelly commented Sep 27, 2023

So this is how a renamed things:

git bisect start --term-new=fixed --term-old=unfixed --first-parent maint OTP-25.3.2

And this is resulting commit:

3508ea39f8d144cb78bda5a2c326b4b6a9f0ce58 is the first fixed commit
commit 3508ea39f8d144cb78bda5a2c326b4b6a9f0ce58
Merge: 69f95cb444 69cd11ae60
Author: John Högberg <john@erlang.org>
Date:   Fri Mar 17 12:22:00 2023 +0100

Beautiful! So what I can do is reset back to this commit on maint-25 and it works.

@jhogberg
Copy link
Contributor

I’m not sure that’ll work well, if nothing else you’ll need to apply this too: 74eee75

@starbelly
Copy link
Contributor

I’m not sure that’ll work well, if nothing else you’ll need to apply this too: 74eee75

Oh, no one said anything about working well 😄 Merely just getting it to compile and run and finding a point in time for that.

@rafaelgaspar
Copy link
Author

Beautiful! So what I can do is reset back to this commit on maint-25 and it works.

That wouldn't work because you would be reseting forward into OTP26 and not backwards.

rafaelgaspar added a commit to rafaelgaspar/homebrew-core that referenced this issue Sep 27, 2023
This basically disables JIT compiling for now since the way erlang@25 does JIT compiling is not supported by Mac OS Sonoma.

See this for more information: erlang/otp#7687
@iAchilles
Copy link

I faced with the same issue.
MacOS Sonoma 14.0 (Mac Studio 2022 Apple M1 Max) OTP 25 - bus error,

but
MacOS Sonoma 14.0 (Macbook Pro 2019 2,6 GHz 6-Core Intel Core i7) - no any troubles. OTP 25 works

@rafaelgaspar
Copy link
Author

rafaelgaspar commented Sep 28, 2023

I faced with the same issue. MacOS Sonoma 14.0 (Mac Studio 2022 Apple M1 Max) OTP 25 - bus error,

but MacOS Sonoma 14.0 (Macbook Pro 2019 2,6 GHz 6-Core Intel Core i7) - no any troubles. OTP 25 works

Makes sense that this is an ARM64 specific issue since it’s related to JIT compilation.

Thanks for the extra info on the issue, I did not had access to an Intel Mac running Sonoma at the moment to test that.

p-linnane pushed a commit to rafaelgaspar/homebrew-core that referenced this issue Sep 29, 2023
This basically disables JIT compiling for now since the way erlang@25 does JIT compiling is not supported by Mac OS Sonoma.

See this for more information: erlang/otp#7687

erlang@25: disable JIT on arm64 Sonoma only
fschoenfeldt added a commit to fschoenfeldt/fotohaecker that referenced this issue Oct 5, 2023
- mostly to prevent issues with latest macos update
- see asdf-vm/asdf-erlang#284
- see erlang/otp#7687
fschoenfeldt added a commit to fschoenfeldt/fotohaecker that referenced this issue Oct 5, 2023
- mostly to prevent issues with latest macos update
- see asdf-vm/asdf-erlang#284
- see erlang/otp#7687
bjorng added a commit to bjorng/otp that referenced this issue Oct 10, 2023
On macOS Sonoma, the JIT causes the runtime system to crash. This
problem is eliminated in OTP 26, but the fix touches too much code for
us trying to backport it.

Fixes erlang#7687
bjorng added a commit that referenced this issue Oct 10, 2023
* bjorn/jit/macos/GH-7687/OTP-18792:
  Disable JIT by default on Apple Silicon Macs
@bjorng bjorng closed this as completed in ac591a5 Oct 10, 2023
garazdawi pushed a commit that referenced this issue Oct 12, 2023
* bjorn/jit/macos/GH-7687/OTP-18792:
  Disable JIT by default on Apple Silicon Macs
fschoenfeldt added a commit to fschoenfeldt/fotohaecker that referenced this issue Oct 12, 2023
* refactor UserLive.Index

- use auth0 cache
- refactor components

* wip stripe

* wip stripe

* refactor e2e auth and prepare photographer test

* comment out details

* update donation banner, improve a11y

* add e2e for photographer

* add note, update translations

* bump to OTP 26

- mostly to prevent issues with latest macos update
- see asdf-vm/asdf-erlang#284
- see erlang/otp#7687

* regenerate gettext

* fix a11y on non stripe user settings page

* fix edge case that was thrown on profile

* add behaviour

* make payment/donations optional

* skip some tests if env var isn't set

* bump translations

* pass down stripe secrets

* add todo

* move donation banner so no errors occur when user not found

* fix error when user is in session despite user management not being enabled

* update docs

* add `UserManagment` and `UserShow` tests

* format

* update gettext

* format again…

* fix credo

* make more tests async

* improve styling

* refactor e2e tests

- move login details into env variables
- document it

* add variables to uberspace ini

* remove stripe callback base url and use endpoint url instead

* fix callback on prod
@jdmarshall
Copy link

jdmarshall commented Mar 8, 2024

disabling jit doesn't work for me. It just changes the error:

make[5]: *** [../../ebin/megaco_flex_scanner_handler.beam] Error 1

@bjorng I am still seeing this issue with 26. It has not been eliminated and I don't think you should have closed it.

@garazdawi
Copy link
Contributor

disabling jit doesn't work for me. It just changes the error:

make[5]: *** [../../ebin/megaco_flex_scanner_handler.beam] Error 1

@bjorng I am still seeing this issue with 26. It has not been eliminated and I don't think you should have closed it.

@jdmarshall From your comments in kerl/kerl#514 it seems like you found out what the problem was and everything is working for you now?

@jdmarshall
Copy link

jdmarshall commented Mar 11, 2024

@garazdawi Yes! Make absolutely sure you've scrubbed ErlangInstaller off your machine. They are mutually incompatible. Check your .rc .profile and .login files. Otherwise you get cryptic error messages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issue is reported as a bug team:VM Assigned to OTP team VM
Projects
None yet
Development

No branches or pull requests