Inconsistent TestStepResult for pending steps across implementations

[davidjgoss]

(Not an issue with CCK as such but have to raise it somewhere.)

Recent work on CCK has highlighted differences in how various Cucumber implementations output a TestStepResult message for a step that was marked pending by the user.

It's worth noting that the mechanism for marking a step pending also varies between implementations - Ruby and JVM do it by throwing, whereas JS and fake-cucumber return from the step function. This influences the current behaviour a little, I think.

Here's the current state of things (with status and timing fields omitted as these are all fine):

field	reference	cucumber-ruby	cucumber-jvm	cucumber-js
message	(omitted)	Name of pending exception	Stack trace from pending exception	(omitted)
exception	(omitted)	Pending exception	Pending exception	(omitted)

[luke-hill]

Not sure if we want to comment here or in messages but the user supplied reason for both pending/skipped in cucumber ruby gets lost in the ether, it's not fully implemented to go through all the steps to message building

There is a message prop but it's kind of redundant

[davidjgoss]

@luke-hill what gets populated on the message prop then for the CCK to be failing on it?

[luke-hill]

There is a prop. Which is why it fails. The CCK has no message prop. So the failing test states "Too many keys produced"

[davidjgoss]

Ah so a prop with no value?

[luke-hill]

It says something like message: " ()Cucumber::Pending" but the devkit generates no message (If you check the ndjsons they're all just featuring status and time)

[davidjgoss]

This was discussed at the community call today. Upshot:

• message should appear on result messages for pending steps - no hard requirements on the content, but it may be a user-authored string explaining why the step isn't implemented yet, or just a default value from Cucumber (suggestion: TODO).
• exception should not appear for pending steps - an exception may have been thrown but only as a vehicle to mark the step pending, and this should not be exposed to reporting.

So the work needed now is:

1. Change fake-cucumber to include a message value for pending steps . Release.
2. Regenerate the reference messages in compatibility-kit. Release.
3. Make any required changes to cucumber-(js|jvm|ruby). Release.