Issue #207 UsageFormatter #214

klausbayrhammer · 2012-02-19T18:16:08Z

A usage formatter displaying duration of steps and calculating average and median of step durations

Sample Output:

0.000135 (median) : Given  precondition
0.025980 (average) : Given  precondition
    0.000025 : Given  precondition
    0.000108 : Given  precondition
    0.000163 : Given  precondition
    0.103624 : Given  precondition
0.069714 (median) : Then  I expect following
0.069737 (average) : Then  I expect following
    0.069334 : Then  I expect following
    0.069668 : Then  I expect following
    0.069674 : Then  I expect following
    0.069754 : Then  I expect following
    0.069842 : Then  I expect following
    0.070154 : Then  I expect following
0.000025 (median) : When  I do the action with param
0.000050 (average) : When  I do the action with param
    0.000011 : When  I do the action with param
    0.000012 : When  I do the action with param
    0.000023 : When  I do the action with param
    0.000025 : When  I do the action with param
    0.000025 : When  I do the action with param
    0.000026 : When  I do the action with param
    0.000140 : When  I do the action with param
    0.000141 : When  I do the action with param

aslakhellesoy · 2012-02-19T18:45:39Z

Nice work!

The original usage formatter in Cucumber displays the location of each step definition, and all matching steps with their location. Your example above doesn't show any locations, so I'm not sure what kind of usage your formatter produces.

There is already a --dotcucumber flag that produces what I would call an improved version of Cucumber's existing usage formatter. It has more information and a more versatile format (JSON). Here is an example: https://github.com/cucumber/cucumber-jvm/blob/master/picocontainer/src/test/resources/.cucumber/stepdefs.json

Were you aware of this? It looks like your motivation for writing this formatter is to get some more time stats. Do you think it would make sense to expand the .cucumber/stepdefs.json format include times instead?

klausbayrhammer · 2012-02-19T20:32:53Z

Your right, the intention was to get some more time stats.

The usage-formatter in cuke4duke generates something like:

12.4327500 ^precondition$                                                                                       #PerformanceSteps.preconditionsellVehicle()
  12.4650000 Given precondition                                                                                   # features\performancetest\performance.feature:18
  12.6340000 Given precondition                                                                                   # features\performancetest\performance.feature:33
  12.7520000 Given precondition                                                                                   # features\performancetest\performance.feature:48
  11.8800000 Given precondition                                                                                   # features\performancetest\performance.feature:63

What is the target of the --docucumber flag? I did like the approach of different formatters, so we could activate a specific usage/performance-formatter

aslakhellesoy · 2012-02-19T22:42:56Z

The argument to the --dotcucumber option is a file path. While it generates "output" it isn't implemented via the Formatter and Reporter interfaces, because the original intent wasn't to report at all.

What it really is is metadata about the structure of gherkin and stepdefs, and how they relate. This is used for code completion in IDEs: https://github.com/cucumber/gherkin/wiki/Code-Completion. The generated file is meant to be committed to source control to make it easily available for IDEs. I think it would be a mistake to add runtime info to it (run results, time stats etc). It would cause a lot of merge problems for developers.

So I change my opinion about merging your work into it. I think your formatter should be a proper formatter/reporter. However, it would be nice if you made it output a similar format as --dotcucumber - perhaps just with extra fields. Having this information available in JSON would make it a lot more useful than plain text.

I suggest you take a look at the classes in the cucumber.runtime.autocomplete package. You could pregenerate a structure with List<MetaStepdef> StepdefGenerator.generate(List<StepDefinition> stepDefinitions, List<CucumberFeature> features). Then, as results come in you can add data about that in the same structure. You'd have to be able to lool up steps from a map, and add some attributes to the support classes (MetaStep perhaps).

klausbayrhammer · 2012-02-20T08:01:14Z

Sounds like a good idea. I'll code it today or tomorrow.

klausbayrhammer · 2012-02-20T12:27:27Z

Now the UsageFormatter generates a json report. I didn't use the StepdefGenerator because the output generated by the usageFormatter is much simpler. Furthermore I didn't have to re-create StepDefinitions and CucumberFeatures.

Sample output:

[
  {
    "stepName": "Given  precondition",
    "aggregatedResults": [
      {
        "strategy": "median",
        "value": "0.000119"
      },
      {
        "strategy": "average",
        "value": "0.030210"
      }
    ],
    "durations": [
      "0.120591",
      "0.000182",
      "0.000056",
      "0.000013"
    ]
  },
  {
    "stepName": "Then  I expect following",
    "aggregatedResults": [
      {
        "strategy": "median",
        "value": "0.069292"
      },
      {
        "strategy": "average",
        "value": "0.069335"
      }
    ],
    "durations": [
      "0.069264",
      "0.069281",
      "0.069388",
      "0.069511",
      "0.069304",
      "0.069267"
    ]
  },
  {
    "stepName": "When  I do the action with param",
    "aggregatedResults": [
      {
        "strategy": "median",
        "value": "0.000013"
      },
      {
        "strategy": "average",
        "value": "0.000049"
      }
    ],
    "durations": [
      "0.000128",
      "0.000193",
      "0.000015",
      "0.000013",
      "0.000013",
      "0.000013",
      "0.000012",
      "0.000012"
    ]
  }
]

aslakhellesoy · 2012-02-20T14:55:40Z

core/src/main/java/cucumber/formatter/UsageFormatter.java

+        StringBuffer buffer = new StringBuffer();
+        buffer.append(step.getKeyword()).append(" ");
+        Format format = getFormat(result.getStatus());
+        Format argFormat = getArgFormat(result.getStatus());


Format is for ANSI-coloured output. Did you really intend to put colours in the JSON?

I did use the MonochromeFormat in order to reuse the StepPrinter for printing Steps and their corresponding arguments.

aslakhellesoy · 2012-02-20T16:39:27Z

I'm still puzzled by a few things:

The step definition is not in the report. Instead of "stepName": "When I do the action with param" - wouldn't it make more sense to have stepdef: "^I do the action with param$".

The durations don't seem to have much value if I don't know where they come from.

klausbayrhammer · 2012-02-20T17:02:59Z

Good point on the stepdefs. I'll put them in the report.

What I want to achieve by using the step-names with their corresponding arguments can be explained by the following example:

Given a stepdef like "^I create (\d+) entries$"
It wouldn't make much sense to compute the average runtime if one step is "I create 2 entries" and the second step is "I create 5 entries". But it would make sense if I would compare all "I create 2 entries" and "I create 5 entries" separately.

Maybe we should include the stepLocations of the executed steps as well?

aslakhellesoy · 2012-02-20T17:25:01Z

Yeah, I think the structure that is used for --docucumber is pretty close to what we want - except it has more info about execution times.

I also think the averages can be simplified:

"aggregatedResults": {
    "median": 0.000013
    "average": 0.000049
}

klausbayrhammer · 2012-02-20T17:47:54Z

Yes your right.

So the output could be like the following:

[
  {
    "source": "^I have (\\d+) (.*) in my belly$",
    "flags": "",
    "steps": [
      {
        "name": "I have 12 cukes in my belly",
        "aggregatedResults": {
          "median": 0.000013
          "average": 0.000049
        },
        "duration" : [
          0.000013, 
          0.000063
        ]
      },
      {
        "name": "I have 3 cukes in my belly",
        "aggregatedResults": {
          "median": 0.000008
          "average": 0.000009
        },
        "duration" : [
          0.000006, 
          0.000012
        ]
      },
    ]
  },
]

we could also add the stepLocations e.g.

[
  {
    "source": "^I have (\\d+) (.*) in my belly$",
    "flags": "",
    "steps": [
      {
        "name": "I have 12 cukes in my belly",
        "aggregatedResults": {
          "median": 0.000013
          "average": 0.000013
        },
        "durations" : [
          {
            "duration" : 0.000013, 
            "location" : "/features/usage.feature:3"
          },
          {
            "duration" : 0.000013, 
            "location" : "/features/usage.feature:7"
          },
        ]
      }
    ]
  }
]

Would you suggest to use the StepdefGenerator.generate? In this case we would have to do some modifications so we can create the MetaStepDef-Structur with the information available in the reporter/formatter.

* added stepDefinition (pattern) * grouped steps under stepdefinitions

klausbayrhammer · 2012-02-23T12:55:23Z

I modified the code to according to your feedback:

This is the current output:

[
  {
    "source": "^the result should be (.+)$",
    "steps": [
      {
        "name": "the result should be 1",
        "aggregatedResults": {
          "median": 0.05107028,
          "average": 0.05107028
        },
        "durations": [
          {
            "duration": 0.102112843,
            "location": "C:\\fake\\feature.feature:3"
          },
          {
            "duration": 0.000027717,
            "location": "C:\\fake\\feature.feature:4"
          }
        ]
      },
      {
        "name": "the result should be 4",
        "aggregatedResults": {
          "median": 0.000017708,
          "average": 0.000020017
        },
        "durations": [
          {
            "duration": 0.000025022,
            "location": "C:\\fake\\feature.feature:5"
          },
          {
            "duration": 0.000017708,
            "location": "C:\\fake\\feature.feature:6"
          },
          {
            "duration": 0.000017323,
            "location": "C:\\fake\\feature.feature:7"
          }
        ]
      }
    ]
  }
]

aslakhellesoy · 2012-02-29T16:13:03Z

I need some more time to go over this - haven't forgotten about it!

aslakhellesoy · 2012-03-18T22:29:50Z

Hi there. I'm afraid this no longer merges cleanly due to recent refactorings. Any chance you can update this pull request?

klausbayrhammer · 2012-03-19T08:08:30Z

Of course. I'll go for it tomorrow

Conflicts: core/src/main/java/cucumber/formatter/FormatterFactory.java core/src/test/java/cucumber/formatter/FormatterFactoryTest.java

lock · 2018-10-25T06:27:54Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

klausbayrhammer added 2 commits February 19, 2012 18:58

Issue cucumber#207 UsageFormatter

9c93be9

Added Keyword for Output

e6d6e4b

Issue cucumber#270 output in json

3a7ca5f

aslakhellesoy reviewed Feb 20, 2012
View reviewed changes

removed unnecessary space (keyword already contains a space)

3d93be7

* added steplocation to report

2e56457

* added stepDefinition (pattern) * grouped steps under stepdefinitions

klausbayrhammer added 2 commits February 23, 2012 18:45

removed unnecessary StepPrinter

3503811

Merge branch 'master', remote-tracking branch 'origin/usageFormatter'

05cc113

retrieve stepname from match instead of a stepStack

7530377

klausbayrhammer added 2 commits March 22, 2012 10:50

ignore failed and skipped steps

a865428

Merge remote-tracking branch 'upstream/master' into usageFormatter

3302b9c

Conflicts: core/src/main/java/cucumber/formatter/FormatterFactory.java core/src/test/java/cucumber/formatter/FormatterFactoryTest.java

aslakhellesoy added a commit that referenced this pull request Mar 26, 2012

Reformat code, shuffle things around a little. Closes #207, #214.

5771b4a

aslakhellesoy merged commit 3302b9c into cucumber:master Mar 26, 2012

lock bot locked as resolved and limited conversation to collaborators Oct 25, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue #207 UsageFormatter #214

Issue #207 UsageFormatter #214

klausbayrhammer commented Feb 19, 2012

aslakhellesoy commented Feb 19, 2012

klausbayrhammer commented Feb 19, 2012

aslakhellesoy commented Feb 19, 2012

klausbayrhammer commented Feb 20, 2012

klausbayrhammer commented Feb 20, 2012

aslakhellesoy Feb 20, 2012

klausbayrhammer Feb 20, 2012

aslakhellesoy commented Feb 20, 2012

klausbayrhammer commented Feb 20, 2012

aslakhellesoy commented Feb 20, 2012

klausbayrhammer commented Feb 20, 2012

klausbayrhammer commented Feb 23, 2012

aslakhellesoy commented Feb 29, 2012

aslakhellesoy commented Mar 18, 2012

klausbayrhammer commented Mar 19, 2012

lock bot commented Oct 25, 2018

Issue #207 UsageFormatter #214

Issue #207 UsageFormatter #214

Conversation

klausbayrhammer commented Feb 19, 2012

aslakhellesoy commented Feb 19, 2012

klausbayrhammer commented Feb 19, 2012

aslakhellesoy commented Feb 19, 2012

klausbayrhammer commented Feb 20, 2012

klausbayrhammer commented Feb 20, 2012

aslakhellesoy Feb 20, 2012

Choose a reason for hiding this comment

klausbayrhammer Feb 20, 2012

Choose a reason for hiding this comment

aslakhellesoy commented Feb 20, 2012

klausbayrhammer commented Feb 20, 2012

aslakhellesoy commented Feb 20, 2012

klausbayrhammer commented Feb 20, 2012

klausbayrhammer commented Feb 23, 2012

aslakhellesoy commented Feb 29, 2012

aslakhellesoy commented Mar 18, 2012

klausbayrhammer commented Mar 19, 2012

lock bot commented Oct 25, 2018