Would like JUnit ErrorCollector support or a Cucumber-jvm equivalent

[JasonSmiley]

I think this would be a common enough concern that it would work, however, I saw a post elsewhere saying this is not supported. (https://groups.google.com/forum/#!msg/cukes/qMwgAVzWmR0/GSkRUgJ8f4EJ)

I am ok with this not being supported at the moment, however, it seems like Error Collecting is something that any test developer would want.

Can we get this support inserted or get a cucumber equivalent?

[dkowis]

I haven't ever had to use any ErrorCollector functionality with Cucumber. Perhaps a pull request with some examples?

[JasonSmiley]

ErrorCollector is a JUnit API class which handles error collecting.

This means if a test fails and I want to continue execution, I can. And then, at the end of the test (in a @after hook I believe), the test would be marked as failing and would provide teh stack trace for each of the errors captured.

So, in a simple case:

public class TestClass{

@Rule 
ErrorCollector handler = new ErrorCollector();

@When("this test will (.*)")
public void test(String type){
  switch(type){
    case "pass":
    case "still pass"
       handler.checkThat("This worked", true, equalTo(true));   //Similar to Assert.that
       break;
    case "fail"
      handler.checkThat("this failed", false, equalTo(true));      //Similar to Assert.that
      break;
     }
}
}

---

Feature: Proof of Concept

  Scenario: Example Testing environment.
    Given this test will pass
    And this test will pass
    When this test will fail
    And this test will fail
    Then this test will still pass
    And this test will still pass

---

In a normal JUnit framework, I can get the above scenario to FAIL as described.

Right now, since JUnit Rules are never checked, this test will pass 100% of the time (even though there should be failures).

[dkowis]

If a step fails, but you want it to continue executing, that's a different scenario. I would say you're not using Cucumber properly. Your step should not fail, if you want it to continue executing.

It looks as though you brought this up over a year ago: https://groups.google.com/forum/#!searchin/cukes/steps$20continue/cukes/odXSUY9cbK0/FdUl1CgrTCwJ

And the answer was, you're not righting your tests correctly. I'm going to go with the same answer, and say that you're not using cucumber the way cucumber is intended to be used, and unless you can offer a compelling reason as to why we should break the fundamental logic of "A step failed, your test failed" I don't think this is going to happen.

[JasonSmiley]

Well, in that post, I actually wanted the test to skip based on outside forces (real bug in the system, just don't run this specific check without having to comment it out of the feature file).

But I will give you an actual test case and let me know if you still think this is bad.

URL = http://www.ebay.com/itm/1x-Demonic-Tutor-MTG-mint-/151396030150?pt=Trading_Card_Games_US&hash=item233fe816c6

----
  Scenario: Page Load Test
    Given I navigate to a "Purchasable Item description page on ebay"  
    Then the "item title" is appearing on the page
    And the "item description" is appearing on the page
    And the "number of bids" is appearing on the page.
----

This is a more practical example, are you still saying I am using cucumber incorrectly? Even though these tests can essentially run in parallel?

If it is intended that this be 3 separate scenarios, wouldn't that be bad from an execution and resource need perspective?

EDIT: comment returned

Shoot - can you get my comment back? I tried to cancel edit, not delete post...

[dkowis]

So cucumber isn't about optimizing execution, it's about optimizing communication. This example scenario indicates that those three things are required to be on the page, and so if any one of those items is not there, have it fail.

How would the scenario be valuable if item title, description, or number of bids doesn't exist, but you expect them to exist?

You could refactor the test, and use a background to grab the page (if you're worried about grabbing the page too many times, which I think shouldn't be a problem for a system under test.) and then

Background:
Given I load the page to optimize my tests.

Scenario: Item title is set
Then the "item title" is set

Scenario: Description is set
Then the "item description" is set

You can do some internal caching if page load times are very slow, where it'll cache it once for a given period of time, or maybe for the entire run, cache eviction strategies are really outside the scope of cucumber.

[JasonSmiley]

what you are proposing is fine for a simple case, but starts to break down (in my opinion) when the setup is harder (solution isn't scalable).

Scenario: Publish Test
  Given I login as a dealer
  And I publish a new test item
  When I navigate to the published item
  Then the "item title" is set
  And the "item description" is set
  And the "number of bids" is set

In a scenario like this, having a big background slows down the test considerably. Not to mention that I now have 3 test items instead of 1. I don't think breaking up the test is the ideal solution (I know the DBA would hate me for example). And when you consider all the services like Sauce Labs that charge based on execution times, breaking up the tests actually become a bad idea since it will cost us more money and we will have to wait longer for results.

However, based on your comment that cucumber is about communication and not execution, I see this as a valid counter point.

I do want to make something clear though. You said, "How would the scenario be valuable if item title, description, or number of bids doesn't exist, but you expect them to exist?"

The problem I am trying to solve is that if the title doesn't exist, I don't know if the description or number of items exist (is it just the title?) or if everything on the page missing. Sure, a screen shot can help reduce the amount of investigative work, but, it isn't always complete (I am looking at you Chrome Webdriver).

Also, just for the sake of mentioning, one of my QA co-workers actually asked for this functionality because she said that the cucumber reports weren't that helpful - they didn't tell her only what failed. it told her what was failed and what was skipped and that made her NOT TRUST the tests. I know its just a matter of perception (I trust the tests after all), but it is still upsetting when people aren't looking at your test results because of all the blue.

[dkowis]

Perhaps you could write a step that "Required elements are on the page" If you wanted to assert mutliple things per single step.

And then fail that step and indicate what elements aren't on the page, basically grouping your assertions into one "pass/fail" and outputting what was causing that grouping to fail. Make sense?

[JasonSmiley]

Thats what I am trying to do :) I just thought this thread become more of a philosophical discussion. I actually thought I was doing something wrong...

JUnit does this for me. Hence the request. My thought was "Why re-invent the wheel". Also, I would think this feature would be used by lots of people. Maybe its just me though, so I could be wrong.

[dkowis]

Fair enough, I don't think it'll be resolved at the cucumber level though, since that would require a fundamental change to the Step failure methodology. However, if you were to provide a pull request that can do optionally the ErrorCollecting, I'm confident it would be fairly reviewed.

[JasonSmiley]

Well, I already re-invented the wheel on my end since it didn't seem like this would happen at first.

I guess I can compare the two different runner classes and see what JUnit is doing instead of you guys. Would take me a bit though.

[slaout]

Hello,

I've seen a lot of threads on the Web about people wanting to continue steps execution if one failed.
I've discussed with Cucumber developers: they think this is a bad idea: https://groups.google.com/forum/#!topic/cukes/xTqSyR1qvSc
Many times, scenarios can be reworked to avoid this need: scenarios must be split into several smaller and independent scenarios, or several checks can be aggregated into one, providing a more human scenario and a less script-like scenario.

But if you REALLY need this feature, like our project do, we've done a fork of Cucumber-JVM.
This fork let you annotate steps so that when they fail with a determined exception, they will let let next steps execute anyway (and the step itself is marked as failed).

The fork is available here:
https://github.com/slaout/cucumber-jvm/tree/continue-next-steps-for-exceptions-1.2.4
It's published on the OSSRH Maven repository.
See the README.md for usage, explanation screenshot and Maven dependency.
It's only available for the Java language, tough: any help is welcome to adapt the code to Ruby, for instance. I don't think it will be a lot of work.

Best regards,
Sébastien Laoût.

[lunivore]

Hi @JasonSmiley , I hope that the suggestions from the team have provided you with some different ways around your problem, or that the code mentioned here by @slaout is suitable for solving it. Since the team have indicated that they don't intend Cucumber-JVM to work in the way you describe (I support them in this), I am closing this ticket as a "won't fix". I hope you're enjoying your work with Cucumber-JVM!

[JasonSmiley]

No worries, I found a suitable workaround. Thanks for responding!

…

Sent from my iPhone On Jun 28, 2017, at 8:40 AM, Liz Keogh <notifications@github.com<mailto:notifications@github.com>> wrote: Hi @JasonSmiley<https://github.com/jasonsmiley> , I hope that the suggestions from the team have provided you with some different ways around your problem, or that the code mentioned here by @slaout<https://github.com/slaout> is suitable for solving it. Since the team have indicated that they don't intend Cucumber-JVM to work in the way you describe (I support them in this), I am closing this ticket as a "won't fix". I hope you're enjoying your work with Cucumber-JVM! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#771 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ADEcgh85Rkwj-d-QVij4y7fhD83qmyohks5sIkmqgaJpZM4CenMx>.

[Dyadko]

Hi @JasonSmiley , could you share your workaround with me? I also stuck with this problem

[JasonSmiley]

I basically just wrote my own implementation by copy and pasting the error collector into my project, and then writing an after step to call the verify function (made public), and used the error collector as needed

…

Sent from my iPhone On Oct 19, 2017, at 6:46 AM, Andrii Dvoinik <notifications@github.com<mailto:notifications@github.com>> wrote: Hi @JasonSmiley<https://github.com/jasonsmiley> , could you share your workaround with me? I also stuck with this problem — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#771 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ADEcgs9ihVl9QXtbmLVa500sKoVd0sXkks5styiSgaJpZM4CenMx>.

[slaout]

You mean an AfterStep like this: https://github.com/cucumber/cucumber/wiki/Hooks#step-hooks ?

So it only works with Ruby. No support in Java: see this issue: #604

[JasonSmiley]

Just for clarify, here is what I did:

1. I copy and paste this file into my project locally - https://github.com/junit-team/junit4/blob/master/src/main/java/org/junit/rules/ErrorCollector.java

2. In my local file, I make verify() a public function

3. In a @Before hook, I would instantiate the ErrorCollector, during my test I would use the checkThat() method, in the @After method I would call the verify().

private ErrorCollector collector;
@Before
public void setup(){
  collector = new ErrorCollector();
 // anything else you need in setup
}

@When("^Something should happen$")
public void stepExpample(){
  collector.checkThat("Something didn't happen", "something", equalTo("not something"));
}

@After
public void teardown(){
  collector.verify();
 // anything else you need in teardown
}

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.