Errno::EROFS thrown when run from a docker container with a RO filesystem

[smedrick]

Describe the bug

When running our Rails app on a docker container with a read-only Filesystem mount, we get

`/home/overlay` is not writable.
/usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:117:in `rescue in filesystem_access': There was an error accessing `/app`. (Bundler::GenericSystemCallError)
The underlying system error is Errno::EROFS: Read-only file system @ dir_s_mkdir - /app/bundler20240702-392135-36ubeq392135
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:102:in `filesystem_access'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:681:in `tmp_home_path'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:241:in `user_home'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:40:in `default_bundle_dir'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:459:in `default_bundle_dir'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:33:in `<class:StacktraceBuilder>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:25:in `<module:ElasticAPM>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:23:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/agent.rb:24:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm.rb:37:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic-apm.rb:21:in `<top (required)>'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `block in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:175:in `require'
	from /app/config/application.rb:15:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require_application!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:14:in `require_application_and_environment!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands/console/console_command.rb:95:in `perform'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/base.rb:69:in `perform'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command.rb:46:in `invoke'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:8:in `require'
	from bin/rails:8:in `<main>'
/usr/share/ruby/tmpdir.rb:89:in `mkdir': Read-only file system @ dir_s_mkdir - /app/bundler20240702-392135-36ubeq392135 (Errno::EROFS)
	from /usr/share/ruby/tmpdir.rb:89:in `block in mktmpdir'
	from /usr/share/ruby/tmpdir.rb:135:in `create'
	from /usr/share/ruby/tmpdir.rb:87:in `mktmpdir'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:315:in `tmp'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:682:in `block in tmp_home_path'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:103:in `filesystem_access'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:681:in `tmp_home_path'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:241:in `user_home'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/shared_helpers.rb:40:in `default_bundle_dir'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:459:in `default_bundle_dir'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:33:in `<class:StacktraceBuilder>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:25:in `<module:ElasticAPM>'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/stacktrace_builder.rb:23:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm/agent.rb:24:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic_apm.rb:37:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `block in require'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:257:in `load_dependency'
	from /app/bundle/ruby/2.6.0/gems/activesupport-5.2.8.1/lib/active_support/dependencies.rb:291:in `require'
	from /app/bundle/ruby/2.6.0/gems/elastic-apm-4.7.3/lib/elastic-apm.rb:21:in `<top (required)>'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:55:in `block in require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `each'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:44:in `require'
	from /usr/local/share/gems/gems/bundler-2.2.33/lib/bundler.rb:175:in `require'
	from /app/config/application.rb:15:in `<top (required)>'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:22:in `require_application!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/actions.rb:14:in `require_application_and_environment!'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands/console/console_command.rb:95:in `perform'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /app/bundle/ruby/2.6.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command/base.rb:69:in `perform'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/command.rb:46:in `invoke'
	from /app/bundle/ruby/2.6.0/gems/railties-5.2.8.1/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:8:in `require'
	from bin/rails:8:in `<main>'

I suspect this is the commit that breaks compatibility with a RO FS:
23b4e4d

and browsing Issue history on the Bundler repo, it appears default_bundle_dir was not designed with the intent that it would be used outside of the install process.

Steps to reproduce

1. Create Rails app with latest version of apm agent
2. Build docker image with pre-bundled Rails app
3. Deploy docker image to Kubernetes cluster using helm directive

        securityContext:
          readOnlyRootFilesystem: true

Expected behavior

Able to run bundle exec rails c or bundle exec rails s without encountering exception

Environment

• OS: Red Hat Universal Base Image 8
• Ruby version: 2.6.4
• Framework and version: Rails 5.2.8, Bundler 2.2.33
• APM Server version:
• Agent version: 4.7.3

Additional context

Add any other context about the problem here.

• Agent config options

  Click to expand

    enabled: false
    log_path: log/elastic-apm-<%= ENV['HOSTNAME'] %>.log
    service_node_name: <%= ENV['HOSTNAME'] %>