GBK and UTF-8 (Encoding::CompatibilityError)

[yanlongguo]

When I use cucumber to execute the specification, I encounter a problem, could you please help me out?
And if you can give me a clue to feedback problems, I will be very appreciated.

The problem is very strange…

All the feature files are encoded with UTF-8.
English Filename with English content —— OK
English Filename with Chinese content —— OK
Chinese Filename with English content —— OK
Chinese Filename with Chinese content —— Failed.

save the following content into two different file test.feature and test_中文.feature, and encode the two files with UTF-8.
Feature: a
Scenario: a
Given ab
When b
Then 中文

when execute cucumber .\features\test.feature, it's can parse the GWT well.
but when execute cucumber .\features\test_中文.feature, it reports error;
D:\temp>cucumber .\features\test_中文.feature
Feature: a

Scenario: s # .\features\test_中文.feature:3
Given a # .\features\test_中文.feature:4
When b # .\features\test_中文.feature:5
incompatible character encodings: GBK and UTF-8 (Encoding::CompatibilityError)
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/has_steps.r
b:45:in backtrace_line' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/step.rb:85: inbacktrace_line'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/step_invoca
tion.rb:98:in failed' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/step_invoca tion.rb:85:inrescue in find_step_match!'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/step_invoca
tion.rb:82:in find_step_match!' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/step_invoca tion.rb:56:ininvoke'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/step_invoca
tion.rb:38:in accept' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker .rb:106:inblock in visit_step'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker
.rb:170:in broadcast' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker .rb:105:invisit_step'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/step_collec
tion.rb:19:in block in accept' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/step_collec tion.rb:18:ineach'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/step_collec
tion.rb:18:in accept' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker .rb:100:inblock in visit_steps'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker
.rb:170:in broadcast' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker .rb:99:invisit_steps'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker
.rb:15:in block in execute' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/runtime.rb:83:i nblock (2 levels) in with_hooks'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/runtime.rb:99:i
n before_and_after' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/runtime.rb:82:i nblock in with_hooks'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/runtime/support
_code.rb:117:in call' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/runtime/support _code.rb:117:inaround'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/runtime.rb:94:i
n around' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/runtime.rb:81:i nwith_hooks'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker
.rb:13:in execute' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/scenario.rb :32:inblock in accept'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/scenario.rb
:79:in with_visitor' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/scenario.rb :31:inaccept'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker
.rb:58:in block in visit_feature_element' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker .rb:170:inbroadcast'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker
.rb:57:in visit_feature_element' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/feature.rb: 38:inblock in accept'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/feature.rb:
37:in each' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/feature.rb: 37:inaccept'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker
.rb:27:in block in visit_feature' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker .rb:170:inbroadcast'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker
.rb:26:in visit_feature' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/features.rb :28:inblock in accept'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/features.rb
:17:in each' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/features.rb :17:ineach'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/features.rb
:27:in accept' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker .rb:21:inblock in visit_features'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker
.rb:170:in broadcast' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/ast/tree_walker .rb:20:invisit_features'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/runtime.rb:49:i
n run!' D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/lib/cucumber/cli/main.rb:47: inexecute!'
D:/Ruby193/lib/ruby/gems/1.9.1/gems/cucumber-1.3.10/bin/cucumber:13:in <top (re quired)>' D:/Ruby193/bin/cucumber:23:inload'
D:/Ruby193/bin/cucumber:23:in `

'

[os97673]

I do not know the root cause of the problem, but I'd print encoding of all strings in https://github.com/cucumber/cucumber/blob/v1.3.x-bugfix/lib/cucumber/ast/has_steps.rb#L44:L46 to see which one has a problematic encoding.

[mattwynne]

Hi @yanlongguo. Can you create a little github project with some examples files in it, so we can easily reproduce the problem?

[yanlongguo]

hi @mattwynne . I have created a very little project to reproduce the problem. you can access the project via: https://github.com/yanlongguo/cucumber. If you have any question about this, please let me know. thank you.

[os97673]

Hi @yanlongguo I've played with the test a little bit (on my Russian WIndows) and here is what I've found. If you will pass -EUTF-8 to ruby (I used "ruby cucumber ..." form of command for this) then everything work ok. As far as I can see the problem is that in your environment ruby uses GBK as default encoding but cucumber uses utf-8. So you can force utf-8 encoding to work.
I'm not sure we are ready to change our code to work with "default" ruby encoding instead of utf-8 :(
@mattwynne @tooky what do you think about this? Perhaps we should specify that we can use with UTF-8 compatible encodings only?

[os97673]

Also I've noticed problem with feature which has GBK encoding (we try to convert some parts to "default encoding" which is hard-coded to utf-8). But I've not investigated this problem deeper.

[sabind]

@os97673 What do you mean by (I used "ruby cucumber ..." form of command for this)?

I'm having the same issue, but I liked the idea of controlling ruby's execution.

[os97673]

@os97673 What do you mean by (I used "ruby cucumber ..." form of command for this)?

instead of running cucumber .... I was running ruby <path-to-cucumber>/cucumber ...
I.e. I've executed cucumber as regular ruby script.

[sabind]

@os97673 Thank you. That's what I was looking for. It's a good tip, because the other option is to .force_encoding in the code and I'd rather not ruin the code like that.

[mattwynne]

What if we did something like this in the main codebase?

https://github.com/cucumber/cucumber/blob/master/spec/spec_helper.rb#L6

Would that work do you think @os97673?

[os97673]

@mattwynne I think it could work.

[mattwynne]

OK I've pushed a fix to master. Can someone please try this and tell me if it's fixed?

[mattwynne]

I'm going to close this ticket in expectation that 64e90e1 has fixed it. If it's not fixed, please reopen the ticket.

We should have a release out shortly that includes this fix.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.