added Test stats to execution context

[pierreh]

#1485
#1692

I would like to propose this pull request. I built a framework on top of AVA that allows you to execute individual steps within a Test. However I wanted to have more control in order to stop steps from executing when previous steps have failed and to give debug feedback about which steps have executed and which have not. In order to do this I had to hack Object.bind to get to the Test object, however this patch would expose the necessary information that tests can use to inspect the current state of testing.

[novemberborn]

Hi @pierreh, thanks for the PR. It's an interesting approach, could you share more about your framework? I want to be careful about how much internal state we expose to the test implementations.

t.try() (#1692) is how we're looking to address use cases like yours. Would that work for you?

(I'm closing this issue for housekeeping purposes, but let's keep the conversation going. Either in this PR or in a new issue.)

[pierreh]

Hi @novemberborn, thanks for your reply. I think the t.try() solution can work for my use case. Is there any implementation for it yet?
What I have created uses ava-spec to create features and scenario's, but on top of that each scenario is built up from individual steps on order to create a BDD style spec file. The steps are implemented in a separate file and are reusable. The steps are a collection of given when then actions. So for my particular case I would need to run t.try() on each step. So the implementation will need to allow me to execute it multiple times during a test. It also integrates supertest/superagent because we are using it to test our API's.

This is what a spec file looks like:

feature("Log in ›", scenario => {
	scenario("a bureau service user", t => steps(t)
		.step(given.aBureauServiceUser)
		.step(when.theUserLogsIn)
		.step(expect.loginIsSuccessful)
		.step(when.theUserLogsOut)
		.step(expect.userIsLoggedOut));
}

This is what the results look like (with my patch)

✖ Log out › a bureau service user Unexpected response code (401): Username or password incorrect
    ℹ step: aBureauServiceUser
    ℹ step: theUserLogsIn
    ℹ failed: loginIsSuccessful
    ℹ skipped: theUserLogsOut
    ℹ skipped: userIsLoggedOut

[pierreh]

The reason why I like my solution is that you can query the results at any time during testing. Also the t.try() solution would have to spawn a nested test context which seems somewhat illogical. The stats don't necessarily need to expose that much, perhaps just assertCount and assertError

[novemberborn]

@pierreh there's an open PR, it's just taking a while to land it: #1947

I like your use case, and I think t.try() is a good fit for it especially since it comes with its own DSL. I'd like to try that first before seeing if we need to expose other details within the main test implementation.

[pierreh]

Thank you, look forward to it landing!