Support component integration tests

[ef4]

This implements template-driven component integration tests. See previous discussion in #25.

TestModuleForComponent now offers both a unit test mode (which is unchanged from the present behavior) and an integration test mode (which subsumes previous ModuleForIntegration).

The default is integration test mode, because I am 100% convinced that is the way most people should be testing most components most of the time. We switch to unit test mode if you provide an explicit needs argument, or say unit: true, or say integration:false.

In integration mode, tests are template-driven, giving you full and intuitive access to all features of components, including passing them block templates, yielding block params, and action handling.

People with existing component unit tests that already have a needs parameter (which I expect is most of them) will not be affected by this change. But people with component unit tests and no needs parameter will get failing tests, until they add unit: true to their module. This is a very low-cost upgrade pain, so I think it's worth it for achieving a simpler API that nudges people toward the right choice by default.

Edited to add: after feedback, I changed this to not be a breaking change. Unit test mode is still the default, but we will hit you with a deprecation warning unless you add an explicit needs or unit:true, which will protect you from a future change that makes integration mode the default.

I have tested this change with both ember-mocha and ember-qunit, and no changes are actually required in either of them to use this functionality. We will want to update their ember-cli blueprint generators however, so that they can generate both unit and integration tests for components, and so they place them under an appropriate path (like tests/integration instead of tests/unit).

[ef4]

This makes the isolated case consistent with the integrated case, which already provides this.registry.

[rwjblue]

I have a few tiny comments, but the approach looks good to me.

[ef4]

Based on feedback from @rwjblue I have made this a less jarring upgrade. For now we still default to unit test mode, but will fire a deprecation warning if you are in danger of getting the new behavior when we finally do switch the default.

[rwjblue]

LGTM.

@dgeb - Any objections?

[dgeb]

I was never crazy about the name TestModuleForIntegration, but it clearly provide a free-form template sandbox into which you can render any elements. You can integration test a single component or multiple components acting together. You can even test multiple configurations of the same component. None of that really goes away with these changes, but the naming gives the appearance that template-driven integration tests are tied to a single component.

This is not just an issue with component integration tests. The integration flag on our modules allows for sharing of an application instance's registry/container, but still leads the developer to focusing tests on a single thing: model, component or otherwise.

I think I'd rather see us introduce integration test modules that provide the semantics to test actual integrations. Perhaps IntegrationTestModule, ModelIntegrationTestModule, and ComponentIntegrationTestModule? The basic IntegrationTestModule could provide access to just an ApplicationInstance, the ModelIntegrationTestModule could also automatically provide a store, and the ComponentIntegrationTestModule could provide a free-form template. None of them would be tied to a single subject however.

@ef4 - I really appreciate the hard work you've done to enable these integration capabilities. I would like to make sure we expose these capabilities in the most useful light to users via the higher level libs (ember-qunit and ember-mocha) and through the blueprints in ember-cli.

[ef4]

I'm not 100% clear on what your position is on this.

I realize "unit" and "integration" mean different things in different
places, and I don't necessarily care which interpretation we pick as long
as we're consistent.

But it is clear that testing against public API is better than testing
against private API. And the main public API components expose is via
templates.
And it is also clear that component tests -- even the existing ones -- are
actually testing integration between your component and many other moving
parts: htmlbars, ember, any built-in helpers or keywords, the long list of
things inside isolatedContainer. That is the kind of integration I'm
talking about.

If we continue to generate component "unit" tests by default when people
run ember g component, I think we are actively leading them down the
wrong path, by encouraging them to test against private API and by not
giving them easy access to all the template features like blocks and block
params.

[stefanpenner]

in general i like to define various testing levels the following way:

• unit: algorithmic complexity + no collaborators (public collaborators)
• integration: low-level interaction between collaborators (mocks for foreign collaborators)
• acceptance: black-box signals sent as the agent without internals leaking forward (exceptions include faking/mocking edge interfaces – adapters)

I believe components in ember, describe the JS/template that controls a chunk of template hierarchy. As such I do not believe today one can purely unit test this. It would have to be some adapted unit or more truthfully an integration mode of sorts.

We should explore a future where this is possible, I believe we can improve it, but pragmatically speaking the path of least resistance today (and short term) is likely more permissive component testing helpers.

This problem is shared by WebComponents and demonstrates that they aren't as isolated as people think. Their nested components are hidden, style and structure don't leak, but the implementation isn't actually encapsulated as at this phase, how to load/initialize and setup their descendants leaks forward.

a concrete example we can build a discussion around:

<my-cool-component>
  <!-- public and external -->
</my-cool-component> 

<!-- my-cool-component.html -->
<my-internal-component />
<input/>

<!-- my-internal-component.html -->
<my-other-internal-component />

When testing: my-cool-component we must introduce both my-internal-component and my-other-internal-component to the environment, or testing becomes impossible.

We do have the flexibility today to provide a mock my-internal-component that the my-component tester controls, but I believe the most common need we have is testing non-mock collaborators strung up in their natural template hierarchy .

[dgeb]

@ef4 Sorry I was not clear. I am not disagreeing, but seeking consensus on what we consider integration tests to be in ember.

From a pragmatic standpoint, I agree that the kind of integration test in this PR should be the default for testing a component. As @stefanpenner points out, there is no way to purely unit test components in ember today.

My point above is that there is room for a more general purpose layer of "integration" tests here. Integration tests that are focused on testing the interaction of multiple components together, multiple models together, or perhaps other aspects of an application instance. TestModuleForIntegration provides a template-driven sandbox for this kind of general integration test, albeit with an awkward name.

So really I'd like to come to an agreement on semantics to allow for space for general integration tests. I raised this here because this PR removes TestModuleForIntegration.

I agree with @stefanpenner's take on testing levels. So how do we translate those levels to these unit-ish integration tests of components? In which directory are these component test blueprints going to be placed? They seem to be the duck-billed platypus of testing.

[ef4]

Ah, I understand better now.

I guess we could still offer some more generic kind of integration test,
which would have pretty much the same implementation but a different name.

I'm hesitant though. More kinds of tests pushes more choice burden onto
users. Especially since we also still have acceptance tests.

In an Ember 2.0 world, everything is a component, so at every layer of your
app there is some component for which you can say moduleForComponent. I
think this necessarily covers all the multi-component interactions that
matter to your app, since they must occur within some enclosing component,
and that gives you a good place to put the integration test.
On Apr 9, 2015 5:19 PM, "Dan Gebhardt" notifications@github.com wrote:

@ef4 https://github.com/ef4 Sorry I was not clear. I am not
disagreeing, but seeking consensus on what we consider integration tests to
be in ember.

From a pragmatic standpoint, I agree that the kind of integration test in
this PR should be the default for testing a component. As @stefanpenner
https://github.com/stefanpenner points out, there is no way to purely unit
test components in ember today.

My point above is that there is room for a more general purpose layer of
"integration" tests here. Integration tests that are focused on testing the
interaction of multiple components together, multiple models together, or
perhaps other aspects of an application instance. TestModuleForIntegration
provides a template-driven sandbox for this kind of general integration
test, albeit with an awkward name.

So really I'd like to come to an agreement on semantics to allow for space
for general integration tests. I raised this here because this PR removes
TestModuleForIntegration.

I agree with @stefanpenner https://github.com/stefanpenner's take on
testing levels. So how do we translate those levels to these unit-ish
integration tests of components? In which directory are these component
test blueprints going to be placed? They seem to be the duck-billed
platypus of testing.

—
Reply to this email directly or view it on GitHub
#38 (comment)
.

[ef4]

I would like to help find consensus so we can land this feature.

It sounds like everyone supports the idea of TestModuleForComponent having an integration mode. The remaining discussion is around whether we should also have another kind of test that is more sandbox-like, without implying it's tied to a single component.

The two issues are separable. They got tied together because I eliminated TestModuleForIntegration -- which never managed to ship to users because we could never find a good UI for it. But to clarify: this PR does not foreclose the possibility of also offering more sandbox-like tests. In fact, I don't think such tests would even need to be implemented separately here in ember-test-helpers. TestModuleForComponent is a sufficiently general primitive that ember-qunit can use it to expose moduleForSnippet or ember-mocha could use it to expose describeScenario, etc. We are still free to design that API.

So I propose merging the uncontroversial feature (integration-mode for components), instead of waiting until we can design this other possible thing (sandbox tests). I suspect that design will not be trivial, which is why TestModuleForIntegration has been languishing.

[dgeb]

I am wary of changing the return value of isolatedContainer here, especially given the likelihood of the container/registry reform work proceeding very soon (emberjs/rfcs#46). In other words, I don't want developers who are using isolatedContainer to need to refactor twice in a short time. Maybe the question should be whether isolatedContainer is really used outside of ember-mocha and ember-qunit?

@ef4 @rwjblue thoughts?

[ef4]

I assumed that only ember-mocha and ember-qunit would be affected, but I could be wrong.

[ef4]

I can just implement isolatedRegistry instead. isolatedContainer can keep the same public API, but get reimplemented in terms of isolatedRegistry.

(While also maintaining compatibility for old Embers that don't have a registry.)

[dgeb]

Perfect! Thanks in advance @ef4 👍

[dgeb]

@ef4 sorry for stalling on this. I am fine with merging once we resolve the isolatedContainer issue.

[ef4]

No problem -- this is definitely an area where having a clear story is important.

[danfinlay]

Does this include scaffolding the needs relationships for model tests? I was referred here because I suggested that feature over at Ember-CLI.

[ef4]

Finally got around to making the last requested change, so that isolatedContainer maintains the same public API. This should be ready to go.

[ef4]

@FlySwatter I don't think you actually need this PR to get the behavior you want in a model test. You should already be able to pass integration: true to get a model test that doesn't need an explicit needs list, which is an appropriate way to integration test a model and its dependencies.

We do still need some clear documentation around that, and possibly some new blueprint generators.

[ef4]

I suspect the can helper is getting manually registered in an application
initializer. Initializers don't run for these tests, and we don't really
want them to, because booting a whole app is more expensive.

That's one reason it's easier to stick with the naming conventions.
Unfortunately helpers without a dash are not auto discovered.
On May 2, 2015 5:31 PM, "James Alexander Rosen" notifications@github.com
wrote:

When using ember-can https://github.com/minutebase/ember-can's {{if
(can "view thing")}} helper in a component's template along with this
branch (and integration: true), I get

Uncaught Error: Assertion Failed: A helper named 'can' could not be found

Yet if I pause right before the this.render call in my component test,
then this.registry.lookup('helper:can') is the right helper!

All the other lookups seem to be working. It's possible that the problem
lies in how that library defines the helper. Or it could be the way this
isolated resolver works.

—
Reply to this email directly or view it on GitHub
#38 (comment)
.

[jamesarosen]

@ef4 great catch on the can helper. I may register some things myself in the tests. It's possible that many of my component tests should really be integration tests.

[stefanpenner]

babel will likely start making it so that we can pre-compile easily via string templates: see: https://gist.github.com/stefanpenner/38a36298d04b29bbce5f

This would allow us to do this, without having to include the pre-compiler.

[dgeb]

Thanks for all your work (and patience) @ef4!

[rwjblue]

@stefanpenner - That gist looks nice, but still seems that it would use runtime compilation (not build time). Unless you are saying there are additional hooks added in babel.

(we could of course write a babel transform that does this for us though)

[stefanpenner]

@stefanpenner - That gist looks nice, but still seems that it would use runtime compilation (not build time). Unless you are saying there are additional hooks added in babel.

it would detect the string template handler, and pre-compile at build time. This allows valid language syntax to be used, but as an optimization (for both perf and dep size) it would be done at build time rather then runtime.

[ef4]

@jamesarosen there does seem to be something wrong with the action handling. I would have expected your example to work. I will investigate further.

[ef4]

I figured out the event handling problem. It's caused by doing render during the setup instead of during the test. If you move the render call into the test itself the problem goes away.

ember-qunit does a fairly wacky thing at the start of every test that breaks event handling for any view created before that point. So you can't render in setup/beforeEach.

[ef4]

There is an existing ember-qunit issue about this behavior. https://github.com/rwjblue/ember-qunit/issues/86

[technomage]

I am all for this feature. I have borrowed @ef4 version from his liquid-fire repo and it is significantly better than the current unit tests for components.

Not having initializers run in general, I understand, but there should be a clean way to spin up ember data in this environment. When components replace controllers, there needs to be a way to test the component/data interface without reimplementing Ember Data for each test.

[mgenev]

@technomage

Not having initializers run in general, I understand, but there should be a clean way to spin up ember data in this environment. When components replace controllers, there needs to be a way to test the component/data interface without reimplementing Ember Data for each test.

I agree so much!

[pangratz]

I played around with babel and created a babel-htmlbars-precompile plugin, which replaces a tagged template string with the precompiled HTMLBars template:

// babel-htmlbars-compiler/index.js
var compiler = require('./ember-template-compiler');

module.exports = function(babel) {
  var t = babel.types;

  return new babel.Transformer('babel-htmlbars-precompile', {
    TaggedTemplateExpression: function(node) {
      if (t.isIdentifier(node.tag, { name: "hbs" })) {
        var quasis = node.quasi.quasis;
        var template = quasis.map(function(quasi) {
          return quasi.value.cooked;
        }).join("");

        return "Ember.HTMLBars.template(" + compiler.precompile(template) + ")";
      }
    }
  });
};

which in turn can be used within Ember-CLI:

// Brocfile.js
var EmberApp = require('ember-cli/lib/broccoli/ember-app');

var app = new EmberApp({
  babel: {
    plugins: ['babel-htmlbars-precompile']
  }
});

module.exports = app.toTree();

and this allows to write for example a component test like this:

import { moduleForComponent, test } from 'ember-qunit';

moduleForComponent('my-component');

test('it renders', function(assert) {
  var component = this.subject({
    hello: "world",
    layout: hbs`huhu {{hello}}`
  });

  // Renders the component to the page
  assert.equal(this.$().text().trim(), "huhu world");
});

---

@stefanpenner is this something you had in mind with your gist https://gist.github.com/stefanpenner/38a36298d04b29bbce5f?

[rwjblue]

@pangratz - Perfect!

[pangratz]

Alright, I've published the inline precompile stuff as EmberCLI addon: ember-cli-htmlbars-inline-precompile.

[ballPointPenguin]

Reading threads like this reminds me why I love the Ember community. 😍

[Emerson]

Would love to see a few robust examples of this in action. I'm trying to do some pretty basic stuff and I'm not sure the best way of approaching the problem. As a simple example:

test('submit-button can be disabled', function(assert) {
  this.render(hbs `{{submit-button disabled=true}}`);

  assert.ok(this.$('button:disabled').length, 'It rendered a disabled submit button');
 // Great! This works.

  // Now I want to change the disabled property to false and assert that the button 
  // is no longer disabled. What's the best way of doing that?
});

I'm sure there is an easy answer to this question (just call render again?), but some more real world examples would be useful for those of us trying to migrate our tests over to this style.

Also interested in how we pass more complicated data into our components (arrays, objects, etc...).

Love the PR - thanks!

Update: rerendered the component and it works, but is that a new instance of the component? Or rerender of the original...

[rwjblue]

This should do the trick:

test('submit-button can be disabled', function(assert) {
  this.set('currentDisabledValue', true);

  this.render(hbs `{{submit-button disabled=currentDisabledValue}}`);

  assert.ok(this.$('button:disabled').length, 'It rendered a disabled submit button');

  this.set('currentDisabledValue', false);

  assert.equal(this.$('button:disabled').length, 0, 'It switched to disabled');
});

Links:

• https://github.com/yapplabs/ember-radio-button/blob/master/tests/unit/components/radio-button-test.js
• alisdair.mcdiarmid.org/2015/06/20/ember-component-integration-tests.html

[Emerson]

Ahh - makes good sense! Thanks @rwjblue