Berkshelf downloader doesn't work to download file from URI in ruby 3.0

[lytao]

Description

We are having a internal supermarket with cookbook files stored on a web server. The location of cookbooks are provided in universe.json as:

"hostsfile": {
  "3.0.1": {
    "endpoint_priority": 0,
    "platforms": { "all": ">= 0.0.0" },
    "dependencies": {},
    "location_type": "uri",
    "location_path": "https://chef-supermarket-internal.domain/cookbooks/hostsfile-3.0.1.tar.gz"
  }
},

When we run berks install, we get the error:

[normal output removed]
Errno::ENOENT No such file or directory @ rb_sysopen - https://chef-supermarket-unstable.ubisoft.org/cookbooks/hostsfile-3.0.1.tar.gz
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/berkshelf-7.2.2/lib/berkshelf/downloader.rb:155:in `initialize'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/berkshelf-7.2.2/lib/berkshelf/downloader.rb:155:in `open'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/berkshelf-7.2.2/lib/berkshelf/downloader.rb:155:in `try_download'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/berkshelf-7.2.2/lib/berkshelf/downloader.rb:42:in `block in download'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/berkshelf-7.2.2/lib/berkshelf/downloader.rb:41:in `each'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/berkshelf-7.2.2/lib/berkshelf/downloader.rb:41:in `download'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/berkshelf-7.2.2/lib/berkshelf/installer.rb:108:in `install'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/berkshelf-7.2.2/lib/berkshelf/installer.rb:181:in `block (2 levels) in install_from_universe'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24:in `block in execute'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:22:in `execute'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/ivar.rb:169:in `safe_execute'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/future.rb:55:in `block in execute'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
	/opt/chef-workstation/embedded/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'

Chef Workstation Version

Any version later than 21.6.467 that has ruby 3.0 bundled. It should start from 21.6.479 according to https://docs.chef.io/release_notes_workstation/.

Platform Version

CentOS 7, but it doesn't matter.

Root Cause

The bug is related to different behavior between open-uri in Ruby 2.x and 3.0. open-uri used to (before Ruby 3.0) overwrite Kernel#open to support downloading files from http URI. However, it is no more the case in Ruby 3.0. We should switch from open method to URI.open instead:

lib/berkshelf/downloader.rb#155

# The line is not working:
# open(url, "rb") do |remote_file|
# Changed to:
URI.open(url, "rb", :ssl_ca_cert => '/path/to/ca-bundles.crt') do |remote_file|
  archive_path.open("wb") { |local_file| local_file.write remote_file.read }
end

You can also refer to the following link for details:

https://localcoder.org/no-such-file-or-directory-rb-sysopen-for-external-url-rails-6-11-ruby-3