Bootsnap causes kamal setup to hang in Ruby 3.3.5

[iiewad]

I have a new Rails app, There are some env

ruby 3.3.5
rails 8.0.0
kamal 2.3.0

Dockerfile

# Install application gems
RUN bundle install
RUN rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git
RUN rm -rf tmp/cache/bootsnap*
RUN bundle exec bootsnap precompile --gemfile

# Copy application code
COPY . .

# Precompile bootsnap code for faster boot times
RUN bundle exec bootsnap precompile app/ lib/

# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile

When I run kamal setup

 DEBUG [c688ef86]       #15 /bin/sh -c bundle install
 DEBUG [c688ef86]       #15 CACHED
 DEBUG [c688ef86]
 DEBUG [c688ef86]       #16 /bin/sh -c rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git
 DEBUG [c688ef86]       #16 CACHED
 DEBUG [c688ef86]
 DEBUG [c688ef86]       #17 /bin/sh -c rm -rf tmp/cache/bootsnap*
 DEBUG [c688ef86]       #17 CACHED
 DEBUG [c688ef86]
 DEBUG [c688ef86]       #18 /bin/sh -c bundle exec bootsnap precompile --gemfile

[iiewad]

I changed ruby version to 3.2.5. bootsnap precompile is ok!

[mattgibson]

Same issue using Ruby 3.3.6 and Rails 8.0.1.

I'm on an Intel mac and the build works if I leave the architecture as default. When targeting arm64 however, it hangs with no output from bootsnap.

Steps to reproduce

1. Install Ruby 3.3.6
2. install rails 8.0.1 gem with gem install rails
3. rails new docker_test
4. cd docker_test
5. docker buildx build --platform linux/arm64 .

Expected:
The docker build completes with no errors.
Actual:
Docker build hangs with the output from bundle install as the last lines on the console.

If I use docker buildx build . then it works fine.

The problematic line in the dockerfile is:

RUN bundle install && \
    rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git && \
    bundle exec bootsnap precompile --gemfile

Breaking it apart into this shows that it's bootsnap, but gives no ouput:

RUN bundle install
RUN rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git
RUN bundle exec bootsnap precompile --gemfile --verbose

[mattgibson]

Switching to Ruby 3.2.6 fixes it for me too

[mihael]

Also stuck on ruby 3.3.7. and latest bootsnap release.

With the --verbose flag it shows paths being processed, then at some point it just stops outputting. The order of the file paths is not deterministic, so it is hard to pinpoint anything specific.

While it says here rails/rails#54039, that the issue is with arm64, I am using amd64 and still getting stuck:

docker buildx build --push --platform linux/amd64 --builder kamal-local-docker-container -t kitschmaster/eternal:... -t kitschmaster/eternal:latest --label service="eternal" --file Dockerfile .

[byroot]

I am using amd64 and still getting stuck:

from and amd64 machine or from an ARM one?

[mihael]

from and amd64 machine or from an ARM one?

Am deploying to a dedicated server using kamal (docker).

The machine is an AMD processor running: Ubuntu 24.04.1 LTS (GNU/Linux 6.8.0-41-generic x86_64)

I have set kamal's instruction to use amd64:

deploy.yml:

# Configure the image builder.
builder:
  arch: amd64

[byroot]

I'm asking about the machine on which you build the image (your dev machine most likely)

[mihael]

I'm asking about the machine on which you build the image (your dev machine most likely)

Dev machine is arm64.

I am fairly new to docker, are you saying the "docker image" build machine, the machine where the image gets built, is somehow the cause for the bootsnap to stall?

[byroot]

Dev machine is arm64.

Right, that's what I suspected.

the machine where the image gets built, is somehow the cause for the bootsnap to stall?

Yes. From what was reported, my current suspicion is a problem with QEMU. The initial report was building an ARM64 image from an x86 machine, yours is the opposite.

When you build an non-native image, docker uses QEMU, and I think that's what cause the hang.

[mihael]

Yes. From what was reported, my current suspicion is a problem with QEMU. The initial report was building an ARM64 image from an x86 machine, yours is the opposite.

Yes, so it seems!

[mihael]

When you build an non-native image, docker uses QEMU, and I think that's what cause the hang.

These are the settings I am running with:

Does it still use QEMU, even if it's set to Apple Virtualization Network?

UPDATE:
Switching to Docker VMM and get the same result.
Using legacy QEMU setting also.

[byroot]

This is Docker desktop?

I'm not too sure how Apple Virtualization Framework works, but I think it's just a native (ARM64 in your case) Linux VM, and QEMU is used inside that VM.

[mihael]

This is Docker desktop?

Yes

[byroot]

I'll try to find some time to try to repro, but I use https://orbstack.dev/ which uses Rosetta under the hood, so I may not run into it.

But I guess I can always install Docker Desktop.

[mihael]

Thanks @byroot

Let me know, if I can be of any help.

[byroot]

Unfortunately, I was unable to repro using either Orbstack or official Docker Desktop.

I followed the steps from #495 (comment), but on a M3 mac (so ARM) and using docker buildx build --push --platform linux/amd64 .

[pglombardo]

I've hit an example of this here in this build which is still hanging.

The build usually takes ~1 hour but as of now it's up to 2 hours stuck on the line below.

This is a Docker build for amd64 and via QEMU arm64 which is where it hangs on

RUN bundle exec bootsnap precompile --gemfile

This is the Dockerfile (ruby:3.4-alpine) and Github workflow file.

Background: Builds have been stable for > 1 year but 2 days ago hit this binfmt bug causing random segfaults while compiling extensions.

Once I added the work-around, segfaults are fixed but now bootsnap precompile is hanging. :-(

I'm not saying this is bootsnap's fault but any input/advice/ideas would be appreciated.

Update/FWIW*: I deleted all Github Action caches and kicked off new Docker builds but the problem remains. For the time being, I've disabled the use of QEMU and arm64 builds entirely. bootsnap on native amd64 works without issue.

[byroot]

I just thought of something. For people who manage to make it hang in local, if you can try to enter the container from another shell, and kill the ruby process with kill -S ABRT -p <PID>.

That should hopefully cause Ruby to crash and print a crash report with a backtrace. Short of having a proper repro, that may give clues as to where it's stuck.

[byroot]

sorry -s, not -S

[ximus]

I have been able to get my builds to run past bootsnap precompile with the following setting:

Use Rosetta for x86_64/amd64 emulation on Apple Silicon specifically.