cucumber2: incorrect selection of scenarios to execute

[akostadinov]

Hello, I'm running cucumber like:
cucumber example.feature:lineX example.feature:lineY

With cucumber 1.3 I see scenarios executed in the order specified once. With cucumber 2 beta3 I see that first it runs the scenario with lower line number first and then it runs the same two scenarios a second time.

The order of execution is a problem because we are using an external test management system to synchronize scenario execution across multiple runners. But it is not ideal for the purpose and when having scenarios in same order on all runners, there are duplicate runs. To avoid that we reorder scenarios randomly and collisions are unlikely. With cucumber reordering execution, this may not work well.
Executing scenarios multiple times just doesn't make sense to me.

There is one more thing that I'm not sure is a bug but is a difference between 1.3 and 2.x. When specifying a line number of a scenario step, then 1.3 would execute the whole scenario. With 2.0 nothing gets executed.

To summarize:

• order of scenarios is not preserved
• scenarios executed more than once
• 2.x would not execute a scenario specified by the line number of a step from that scenario

[mattwynne]

Confirmed.

bundle exec cucumber features/docs/cli/run_specific_scenarios.feature:9:25

=> 2 scenarios (2 passed)

vs

bundle exec cucumber features/docs/cli/run_specific_scenarios.feature:9 features/docs/cli/run_specific_scenarios.feature:25

=> 4 scenarios (4 passed)

[jdks]

Hi @mattwynne I've made a naïve attempt to fix the duplication issue stated above. It may only be a partial solution to the overall problem so please let me know if any other improvements can be made.

[akostadinov]

That may sound stupid but my gut feeling tells me, it would be more reliable to build @file_specs in the first place, instead of trying to correct it after the fact.

Additionally I see reordering of executed scenarios is happening even when selection is like this: features/3scenarios.feature:9:6

[brasmusson]

@akostadinov I cannot reproduce "2.x would not execute a scenario specified by the line number of a step from that scenario".
All of
bundle exec cucumber features/docs/cli/run_specific_scenarios.feature:10
to
bundle exec cucumber features/docs/cli/run_specific_scenarios.feature:23
result in

1 scenario (1 passed)
4 steps (4 passed)

[mattwynne]

@akostadinov thanks so much for reporting this, and all the other bugs you've raised.

I've closed the ticket, but if the problem of selecting a step still seems to be an issue for you, can you open a new ticket?

[akostadinov]

As you can see, the bugs I've reported are happening when 2 or more scenarios from a file are specified. I can't understand why the issue has been closed? Was it fixed? 2 days ago it was not.

[mattwynne]

@akostadinov as I understand it, there are two bugs reported in this ticket:

1. selecting two scenarios from the same feature by mentioning the file twice caused the scenarios to be run multiple times
2. specifying the line number of a step does not run the scenario

I think (1) has been fixed by a8050ee which is in the master branch, and will be released soon. Do you have the facility to test from the master branch?

@brasmusson has said he can't reproduce (2).

That's why I closed the ticket. Does that make sense? 😄

[akostadinov]

Thanks, I understand better now. But the third one I reported - reordering of scenarios executed (see end of my original post) has not been addressed. Or I don't see how it would be. But I'll be able to test after the holidays.

[mattwynne]

Oh sorry I didn't clearly understand about that one. OK. Can you create a separate ticket for that with a specific example? If I get a chance I'll test it myself too.