Add ability to define multiple test cases for same endpoint/response

[jekhardt]

Currently the test runner generates a test case per endpoint/response defined in the raml. It would be great to be able to define multiple test cases in situations where more thorough testing is desired to feed in slightly modified sample inputs via hooks.

There are several ways to approach this, but here's one thought I had to keep it minimalistic for the test author... We could add test instances during hooks file processing just by defining additional before hooks.

For example, if we have...
GET /endpoint -> 200

Then in the hooks file, to achieve multiple test executions we could specify additional hooks:
before GET /endpoint -> 200 ...
before GET /endpoint -> 200 arbitrary workflow 1 ...
before GET /endpoint -> 200 arbitrary workflow 2 ...

And when the hooks file is processed it could detect there are additional tests and add those into mocha.

[cybertk]

It's good

[cybertk]

Do you think it will be a little unclear if there are too much test case in hooks. Can we order the case in levels? such as

before("API1")
    .case('case 1')
    .case('case 2')

[jekhardt]

I think the names would be clear enough for me with the current format with just differing the names as I described, but I guess it could be put into levels also. I was thinking the current style makes it easy to refactor lines when a single test case endpoint looks the same as a multi test case endpoint. What do you think would look best considering some multi test cases could have considerable amount of before logic (eg. your fixture examples x5 in a before("API"1").case... format vs. just repeated blocks of the current syntax having differing test name.

[cybertk]

You can have a look at #7, what about this style?

hook ('GET /path', 'scene 1')
.before (test, done) ->
  done()
.after (test, done) ->
  done()

hook ('GET /path', 'scene 2')
.before (test, done) ->
  done()
.after (test, done) ->
  done()

[jekhardt]

That works!

[jekhardt]

Is there anything we can do to help? Just curious what your estimate would be for completing this change (planning purposes)?

[jekhardt]

Hey @cybertk any news on implementing this?

[cybertk]

I started development on previous weekend. Have not much time recently, so the progress is not good.

[bpytlik]

@cybertk If a pull request was offered, would you be willing to take it? I'm not sure whether I'll be able to get to it soon, but I might.

[cybertk]

PR is welcome, I'm very appreciated for your contribution.

[bpytlik]

I've been thinking about this issue for a bit and I'll layout what I think the issues are and some potential solutions.

Issues:

1. We want to automatically generate tests based on the raml, but we also need to provide ways for users to modify those automatically generated tests.
2. There's no hierarchical organization of tests currently, but adding hierarchical organization would make associating user defined tests with their automatically generated counterparts difficult.
3. Using the syntax like mocha's or jasmine's would be a departure and might be a challenge to implement.

I can see a couple of paths forward. The simplest one is to use something like the hook construction described in the comment above, but perhaps modified slightly to be like this:

hook('GET /path -> 200', [
    new Case('case 1').before(before_func).after(after_func).test(test_func),
    new Case('case 2').before(before_func2).after(after_func2).test(test_func2)
]).beforeAll(before_all_func
).beforeEach(before_each_func).afterAll(after_all_func).afterEach(after_each_func)

That follows the same conventions that exist today. We could use suite instead of hook but I'd be concerned that would cause confusion w/ suite from mocha or jasmine which would have different constructions and semantics. I'd note that the .before .after .test etc are all optional. The before functions on the hook are run before any of the case's methods are run and the after functions on the hook are run after all of the case's methods are run.

It doesn't provide a fully hierarchical set of suites. An example where this is an issue:
In my project, we have an http server that talks to another server that provides some data. If the second server is down, then the http server should send back a 404 (or a 500 or...). Right now, if I wanted to test my http server with the other server both up and down, I would need to use separate hook files or start up and shutdown the other server in each test case.

I think that's ok for now as putting it in a second hook file isn't a terrible solution (though I'd probably need a 'skip' function or something similar b/c we might never get a 200 code in that scenario).

How does the approach outlined above sound?

[bpytlik]

So as I've continued working with abao, I've rethought some things. I think the right way to do this is to adopt an approach like mocha and others have where you have separate suites, and you expose those suites to users.

For example:

suite("media", () ->
    beforeAll(() -> ...)
    beforeEach(() -> ...)
    suite("song", () ->
         suite("{name}", () ->
             test("GET 200", () -> ... )
             suite("PATCH", () ->
                 suite("200", () ->
                     test("case 1", () ->
                     )
                     test("case 2", () ->
                     )
                 )
                 test("403", () ->)
             )
         )
    )
    suite("books", () ->
    )
)

Suites would be optional so one could do something like:

before("media/song/{name} GET 200", () ...)

and that would be equivalent to

suite("media", () ->
    suite("song", () ->
       test("{name} GET 200", () ->
       )
    )
)

or something similar.

Unfortunately, I think this means a large rewrite of the hooks infrastructure. I may tackle this in the future but probably not anytime soon.

[bpytlik]

Thinking about this some more over the weekend, I think you'd really want to be able to note whether the suite name is or is not part of the route url.

[cybertk]

I think it should be.

[ferrylauda]

Hi guys, we (with @mik01aj) did some workaround in order to have multiple tests on the same endpoints.

https://github.com/lovelybooks/abao

it a quick fix while we are waiting for your solution.

[cybertk]

Thanks @ferrylauda

[MMore]

Hi guys,
just want to emphasize the importance of this feature too. We described all APIs with all possible response codes. So when I run the test I have exactly one successful test for one endpoint, but 3 or 4 failing ones.
So for us the current implementation is completely useless.

[cybertk]

+1

[agrover8]

Any update on this issue or any workaround. I have tried the workaround mentioned by @ferrylauda but it didn't work for me.

hooks.after('GET /org/{orgid}/sample -> 200', function(test, done) {

   console.log(test.response.body);
   done();
});


hooks.before('GET /org/{orgid}/test/{sample} -> 200 second test case', function(test, done) {

    test.request.headers.cookie = session;
    test.request.params.orgid= 17128359;
    test.request.params.sample = 20719951;
    //console.log(test.request);

 done();
});

[ferrylauda]

Hi @agrover8,

Unfortunately we did not implement the feature for "after"-hooks, because we need this temporary solution for multiple test on "before"-hooks only.

This is the example how to use multiple tests within "before"-hooks

hooks.before('GET /org/{orgid}/test/{sample} -> 200 first test case', function(test, done) {
    // your first test case code
});

hooks.before('GET /org/{orgid}/test/{sample} -> 200 second test case', function(test, done) {
    // your second test case code
});

[agrover8]

@ferrylauda Thanks for the reply, but this also not working for me. I did update abao last week. Its v0.4-beta.1

[plroebuck]

+1. Have immediate need for this! Perhaps just implement @ferrylauda's solution for now,
and rethink the architecture for v2. Inability to test more than one route at time is killing me...

[martnst]

+1 I guess it's a highly common need.
@plroebuck did you get it to work with the approach of two before hooks by @ferrylauda on Nov 16, 2015? I cloned the git repo on the very last master commit (which includes #63). However it doesn't work for me. It just ignores the hooks (as the names do not match any test, i guess).

[bpytlik]

@maremmle are your before and after hooks working for the test? The test function uses the same name as those do. Here's an example I used:

test("GET /cfg/admin/user -> 200", (response, body, done) ->

If that's not working, please post an example of what's not working and I'll take a look.

[agrover8]

Can anybody please post an working example? This is the only issue which is stopping us to use ABAO as API framework.

[plroebuck]

Until we get this done, multiple tests per endpoint can be simulated by using different hookfile(s) and rerunning the command. A "less than" solution, but a workaround.

[agrover8]

I have tried the multiple hooks files, but look like abao is reading only
first hook file.

On Thu, Mar 3, 2016 at 10:04 AM, plroebuck notifications@github.com wrote:

Until we get this done, multiple tests per endpoint can be simulated by
using different hookfile(s) and rerunning the command. A "less than"
solution, but a workaround.

—
Reply to this email directly or view it on GitHub
#35 (comment).

• Thanks
  Amit Grover

[plroebuck]

Say if you prefixed your normal run hookfile names with 'basic_', then duplicate one and prefix it with 'alt_'. Put your alternate endpoint test in that file. Do first run specifying --hookfiles=basic*_hooks.js. Then specify --hookfiles=alt*_hooks.js --hooks-only for second run. Make sense?

[agrover8]

Means I have run separate command for each hook file. I will try this.

[plroebuck]

@agrover8, were you able to test multiple cases using the [edited] workaround described above?