🚀 Feature: Test metadata

[lightsofapollo]

99% of the time grep serves my needs but recently we have a need for running/not running tests (the same tests) under many different conditions (think like a selenium situation on multiple browsers) and only some tests apply in some situations...

Similar to what rspec does this is what I would like to do:

// the metadata itself is irrelevant but there are some uses in my 
// particular instance(related to multiple different "host" environments like browsers 
suite('my cool thing', { browsers: ['not-ie6'] }, function() {
});

My initial reaction is to put this in some external repo that extends mocha but I wonder if this is useful enough to put in the core lib so others can use that metadata to extend mocha's behaviors in their extension libraries/projects.

Is this something that would be accepted into core mocha?

[park9140]

I've been using a form of tagging that seems to work for situations like this.

I'll add a "tag" to my suite definition like below.

suite('my cool thing @awesome @monkey @not-ie',function(){});

You can --grep "@awesome" to get all @awesome tests.
Or for no tests that contain @not-ie use -i --grep "@not-ie".
If you need to use multiple tags you have to use regex. For example if you want all @awesome tests but no @monkey tests you would --grep "@awesome|(?!@monkey)"

It would be nice if we didn't have to pollute the test suite name with the tags though and have them in a hidden value that is also searched by grep.

[tj]

what about:

if (!ie6) {
  stuff
}

[rprieto]

Agreed, there's a lot of cases where I'd like to do something like:

• --tags "not:network" when running tests locally
• --tags "is:integration" to run integration tests only
• --tags "is:integration not:slow" when running tests on Travis
• --tags "not:business-hours" to exclude certain tests when running scheduled/hourly tests

This can be achieved with --grep, or a conditional if in code, but it usually becomes quite convoluted.

@lightsofapollo I also tried creating this as a external module, but no luck without monkey-patching quite a lot of methods.... so I have a proof of concept PR and keen to get feedback.

@tj is this something you're open about, if the syntax is clean enough? For now it uses the describe.tag('tag1, 'tag2', 'title', function() {...}) syntax, but could easily be switched to something like describe('@tag1 @tag2 rest of the title', function() {}).

describe('cool api', function() {
  it('returns a list', function() {});
  describe.tag('integration', 'runs as a service', function() {
    describe.tag('fast', 'some quick tests', function() {
      it('can get a status page', function() {});
    });
    it.tag('slow', 'calls the backend', function() {});
  });
});

By default all tests will run as usual. If you set --tags, only tests matching the predicate will be run, and the others will be marked as "pending". For example:

$mocha --tags "is:integration not:slow"

  cool api
    - returns a list
    web service
      - calls the backend
      some quick tests
        ✓ can get a status page

[lightsofapollo]

@rprieto Randomly (wow this is pretty old now) the --ui option now accepts the ability to specify third party modules (I have no idea if this would help entirely but would be a potential way to handle this). That said In general people I have talked to like the idea of having tags.

[rprieto]

Haha yes sorry to revive an old issue, I thought it'd be better than opening a new one. I'll have a look into --ui, thanks!

The main issues I had with building a module was accessing a suite's context to set & test tags. The easiest place to add this logic was the actual definition of it and describe, hence the proof-of-concept PR (not submitted yet).

[boneskull]

A new interface may be the best way to go, because you have several interfaces this would need to be implemented in.

But, if this was going to be done, I would prefer a third parameter to describe, beforeEach, it (etc) which is an array of tags.

The implementation would be pretty trivial at that point.

[rprieto]

This makes sense, I actually implemented it in interfaces/bdd so it would be trivial to extract a new interface (and potentially release it separately from Mocha). And 👍 to the third parameter.

Or is there an easy way in Mocha to apply something like this across interfaces? (even if it means a different syntax, like tag('foo', 'bar').describe(...)).

[boneskull]

There's really no way to automatically provide a function to all interfaces; everything has to be explicitly placed in the context.

This would actually not be that trivial without some refactors.

[rprieto]

Hi, just a note I submitted a tentative PR to add tags to the BDD interface with the feedback above.

It's barely any code at all, and I believe it would be very useful - however I still couldn't think of a way of extracting it as a module that can be shipped separately (especially if we want a nicer integration like mocha --tags).

All of it is optional though, so it doesn't break any existing behaviour.

[christopheranderson]

I have a scenario that I think is similar to this.

I have functional tests for libraries across several languages. There is a large amount of tests in each library and what they are testing tends to drift overtime. There isn't a common naming scheme we've followed and adoption of proposed schemes has some reasonable debate. We use XUnit as our standard test output format, which allows for arbitrary properties on each test/suite/failures. Many of our other test frameworks allow adding arbitrary metadata to test which is then added to the XUnit style output.

I believe I could use #1445 for this purpose, with some modifications to the XUnit reporter, but I don't explicitly need the ability to filter on --tag in the cli (though that could be useful occasionally). I'd still need some mechanism where I mapped a tag like "::" into separate attributes.

What would be optimal for my use case is the ability to do something like:

describe('connectivity tests' function() {
    this.metadata('component', 'connectivity')
    it('should error on http without override set', function() {
        this.metadata('requirement', 'http/requireOverride')
        // ...

Then reporters with structured output could include the metadata on their output, if it is present.

Downsides of this would be that since the metadata is added in the function, it would have limited ability to add metadata to tests for filtering, but would work fine for the "suite" level filtering (assuming you wrote a custom test startup which did the filtering or there was some kind of arbitrary metadata filtering mechanism added to the cli)