Fix to work with UTF-8 Gem $HOME

[davispuh]

What was the end-user or developer problem that led to this PR?

I'm using RVM and my $HOME is /home/Dāvis so Gem.ruby returns /home/Dāvis/.rvm/rubies/ruby-3.3.4/bin/ruby which is with UTF-8 encoding and that causes failure

Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
  /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/cli/exec.rb:85:in `start_with?'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/cli/exec.rb:85:in `block in ruby_shebang?'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/cli/exec.rb:85:in `any?'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/cli/exec.rb:85:in `ruby_shebang?'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/cli/exec.rb:22:in `run'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/cli.rb:455:in `exec'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/cli.rb:35:in `dispatch'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/cli.rb:29:in `start'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bundler-2.5.22/exe/bundle:28:in `block in <top (required)>'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bundler-2.5.22/exe/bundle:20:in `<top (required)>'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/bin/bundle:25:in `load'
          /home/Dāvis/.rvm/rubies/ruby-3.3.4/bin/bundle:25:in `<main>'

What is your fix for the problem, implemented in this PR?

This PR fixes this issue by setting ASCII-8BIT and using that for comparison instead.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[duckinator]

Hi!

I'm not going to merge this because it seems like it would cause more problems than it solves, but I can try to help you figure out another solution.

It seems like something is up with your environment — ASCII only defines 128 characters, and they don't include ā. But your environment is telling Ruby (and thus RubyGems) that it's ASCII, despite you clearly having a non-ASCII character in your HOME and GEM_PATH.

I added a user with the same username on my system, and with a UTF-8 $LANG it works:

Dāvis@cerberus:~$ gem install boop
Fetching boop-0.5.0.gem
Successfully installed boop-0.5.0
Parsing documentation for boop-0.5.0
Installing ri documentation for boop-0.5.0
Done installing documentation for boop after 0 seconds
1 gem installed
Dāvis@cerberus:~$ echo $LANG
en_US.UTF-8
Dāvis@cerberus:~$ ruby -e 'p Gem.ruby'
"/home/Dāvis/.rubies/ruby-3.3.5/bin/ruby"
Dāvis@cerberus:~$

Can you provide the output of running env | grep LANG and env | grep LC_?

[davispuh]

It's not related with environment. I'm only using UTF-8

$ env | grep -E 'LANG|LC_'
LANG=lv_LV.UTF-8
LANGUAGE=

Gem.ruby contains Unicode and is correctly encoded as UTF-8 but when you read first line for shebang you read it in binary mode rb

      first_line = File.open(file, "rb") {|f| f.read(possibilities.map(&:size).max) }
      possibilities.any? {|shebang| first_line.start_with?(shebang) }

first_line is in ASCII-8BIT encoding which is not ASCII but just means it's binary encoded.
This is why we get such failure when trying to compare first_line with UTF-8 encoded shebang

[davispuh]

Hmm something's off, I don't understand why it was failing, I thought that you can't use start_with? with ASCII-8BIT encoding and UTF-8 encoding but it works...

first_line = 'test'.force_encoding('ASCII-8BIT')
shebang = '/home/Dāvis/.rvm/rubies/ruby-3.3.4/bin/ruby'
first_line.start_with?(shebang)
# No exception

but then why did it show up like that...

Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
  /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/site_ruby/3.3.0/bundler/cli/exec.rb:85:in `start_with?'

[davispuh]

Now also it doesn't seem to be happening anymore, weird...

[duckinator]

I dug into this a bit more.

Turns out, string interpolation putting an ASCII-8BIT string into a UTF-8 string results in a UTF-8 string.

That means that shebang in first_line.start_with?(shebang) is guaranteed to be UTF-8, no matter what.

This means it is impossible for this to have fixed anything, on any system.

irb(main):025> gem_ruby = Gem.ruby.force_encoding('ASCII-8BIT')
=> "/home/puppy/.rubies/ruby-3.3.5/bin/ruby"
irb(main):026> gem_ruby.encoding
=> #<Encoding:ASCII-8BIT>
irb(main):027> positions = ["#!#{gem_ruby}\n"]
=> ["#!/home/puppy/.rubies/ruby-3.3.5/bin/ruby\n"]
irb(main):028> positions[0].encoding
=> #<Encoding:UTF-8>
irb(main):029> 

[deivid-rodriguez]

I think this error may be real, and not a misconfiguration, and although I'm no encoding expect, I think @davispuh fix may do the trick. I used the same trick for a similar error at #8196.

@duckinator If you run gem install rake under your new user with a UTF-8 home, you should get a rake binstub installed with a shebang like: #!/home/Dāvis/.rubies/ruby-3.3.5/bin/ruby. If you then run bundle exec rake, I think the error should be triggered.

I didn't fully simulate the situation myself but I can recreate the error like @davispuh suggested below, just slightly modified:

$ irb
irb(main):001> first_line = '#!/home/Dāvis/.rvm/rubies/ruby-3.3.4/bin/ruby'.force_encoding('ASCII-8BIT')
irb(main):002> shebang = '/home/Dāvis/.rvm/rubies/ruby-3.3.4/bin/ruby'
irb(main):003> first_line.start_with?(shebang)
(irb):3:in `start_with?': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
	from (irb):3:in `<main>'
	from <internal:kernel>:187:in `loop'
	from /Users/deivid/.asdf/installs/ruby/3.3.5/lib/ruby/gems/3.3.0/gems/irb-1.13.1/exe/irb:9:in `<top (required)>'
	from /Users/deivid/.asdf/installs/ruby/3.3.5/bin/irb:25:in `load'
	from /Users/deivid/.asdf/installs/ruby/3.3.5/bin/irb:25:in `<main>'