Use with RSpec (possibly via external gem / plugin)

[chriskilding]

We are looking to use Dredd to validate an API written in Rails. We use RSpec for our test suite, and want to hook Dredd + an API blueprint file into the test cycle. To make life easier for our Ruby programmers, we are looking for some form of Ruby-Dredd integration:

Idea 1

Build a basic RSpec plugin/gem which would npm install dredd and friends, then, at some point in the Rspec test cycle, do something like exec 'dredd blueprint.md http://localhost:3000' to invoke Dredd's command line runner.

Problems:

• RSpec has its own test runner, so we don't really want or need the Dredd runner.
• Need to obtain the actual host + port that the test Rails server is running on, it won't always be localhost or 3000.
• Running arbitrary npm install -g commands could mess up the user's system wide nodejs configuration.
• Would have to define before/after test behaviour in Dredd hook files, rather than Rspec before/after blocks, which makes things like DB access harder in a Rails project (and also not synced up with actions happening back in Ruby land like database_cleaner, a gem which is used to nuke the contents of the test database every so often during test suite execution)

Idea 2

Build an RSpec plugin/gem which would drag in Dredd + associated JS modules, then execute them with something like ExecJS. If we take this route, since RSpec provides its own test runner, we would want to call the lower level function in Dredd which executes each test, rather than using the Dredd runner. This would also allow programmers to define RSpec before and after hooks, e.g. for cleaning or inserting things into the database around each test, like they are used to.

Problems:

• ExecJS would deal with the execution of functions in the Dredd nodejs module, but doesn't handle obtaining the module itself.

Ideas?

[netmilk]

Hi,

my opinion is, that best solution is to install Node.JS and Dredd locally even if it could mess up with user's or CI system as you mention. If you don't wont to install it in global scope, just install it without -g option and run Dredd from ./node_modules/bin/dredd

Dredd is an additional tool in the stack for example same as JMeter or something else. In Rails I think it will be reasonable to create some kind of rake task wrapper for the Dredd CLI and make it part of your rake test:all task.

I understand the need to use Ruby/Rails native tooling e.g. FactoryGirl in before and after hooks, but unfortunately I don't see lot of possibilities how to integrate Node.js Dredd HTTP transaction runner tighter with Ruby RSpec. Using Exec.JS or therubyracer to call Node.JS code in Ruby it seems to be crazy for me.

For me it sounds more reasonable to create some API and some inter-process socket/pipe to enable possibility to write hooks in any language with their native tools. But I think it will be always little bit complicated.

[gonzalo-bulnes]

Hi!

For two weeks I have being validating little Sinatra applications against their blueprints using a Rake task as @netmilk suggested.

The blueprint:verify rake task that I use does depend on Dredd being available and the API being served. That's not ideal, particularly because those two steps must be handled manually.

However, after adding a few hints to quickly spot the missing Dredd errors and unresponsive API issues and having ensured that the API server is automatically reloaded when files change (see rerun for example), I must say that I found the result usable enough to add it to my default Rake task. (See the last line of the gist.)

@themasterchef I would also like Dredd to validate the blueprints accuracy without having to start a server (as RSpec, Rack::Test do). If you have any idea of how we could build that, I'd love to hear about it. The gem idea doesn't sound at all crazy to me : )

For now, here is a gist of my blueprint:verify rake task.

Best regards!

[netmilk]

@gonzalo-bulnes Thanks a lot for the example and more info on your approach! My goal is to make Dredd as convenient as possible in non-node.js environments, in Ruby and Rails especially. And this is perfect contribution.

I also saw some people using Dredd's --only argument from the rake task to run test cases from the blueprint one-by-one separately and doing setup and teardown between tests directly in the rake task in ruby.

About not running a server: It's imho not possible with use of Dredd. It's because Dredd works on the HTTP layer. When you are testing controllers e.g. in Rails, you are testing Ruby interface of controller classes, not an interaction on the HTTP layer.

[gonzalo-bulnes]

@netmilk About the HTTP layer: I think you're right and writing RSpec when meaning Capybara (even if Capybara does extend RSpec) is abusive and sometimes unclear. Well, Capybara does rely under the hood on browser engines and allows to describe the behaviour of the app when users intetact with it. What's interesting is the 'under the hood' thing: as a developer you only have to set the RSpec javascript: true tag to have the feature spec handled by a browser engine. I have no idea how that's done, but it may be an example to explore, and I believe that's what I have in mind when thinking about RSpec-Dredd integration (RSpec matchers and automatic database setup and teardown). I'll take a closer look at Capybara.

By the way @netmilk, thanks for Dredd, it's really awesome!

[gonzalo-bulnes]

Hi @netmilk,

I solved the inconvenience of having to start a server manually (looking at Capybara was a good thing to do). I would like to package this rake task as a gem, along with what may result of #134. Any suggestion for a gem name?

You said in a previous comment:

I also saw some people using Dredd's --only argument from the rake task to run test cases from the blueprint one-by-one separately and doing setup and teardown between tests directly in the rake task in ruby.

I don't see right now why I could want to run the blueprint test cases one by one separately, however, I'm sure there must a lot of good reasons to do that and I think these good reasons could be well served by appropriate RSpec matchers.

Along with the possibility of writing Dredd hooks in Ruby - I'm thinking aloud - that may result in the gem to provide some elements of a DSL for API-blueprint-driven acceptance testing in Ruby (in the same way Capybara extends the RSpec DSL with vocabulary for non API-blueprint-related acceptance testing).

Maybe I should open a separate issue to discuss this... what do you think?

[gonzalo-bulnes]

Hi @netmilk, @themasterchef

Here is an early version of Dredd::Rack. And here is the corresponding (minimal) example application.

Dredd::Rack does provide integration for Dredd into a Rack environment and is meant to make enjoyable to verify the API blueprints of any Rails, Sinatra, Padrino, Grape... application.

The enjoyable part consists in making unnecessary to start a local server when testing API blueprints. Dredd::Rack takes care of doing it and ensures that the app is served when required by Dredd and that the server is stopped once the verification is done. Verifying the API compliance with its blueprint does take two words: rake dredd, which makes realistic to use the API blueprints to practice BDD as soon as the blueprint is not excessively long, more on that later.

By the way, I named the default Dredd::Rack task :dredd, but creating a custom task named blueprint:verify is possible and (I hope) easy: the example app Rakefile features both tasks.

Concerning the Rake task by itself, it is pretty much the same of the gist I shared before.
The runner behind it is quite capable, and my idea is to extend both the code and the documentation to make easy to define custom Rake tasks with distinct Dredd options to cover as much scenarios as there are usages of Dredd. The runner should not limit the Dredd possibilities and that should allow to define tasks to run parts of the blueprints only, or in given circumstances, which should make easier to test-drive development with large API blueprints.
Until I complete the README, you can look at the Travis CI output as documentation, and at the code for details. Feedback would be very welcome!

Best regards!

[netmilk]

Hi @gonzalo-bulnes,

thanks a lot! This is absolutely mind-blowing! I really like that Dredd CLI wrapper class! I think it's worth extracting it to dedicated gem. On the other hand passing Dredd configuration to the rake task is not much straightforward.

Please have a look on my fork of yours example application. I've created a concept of possible approach for hooks worker in Ruby. Run rake blueprint:verify and look for Yay! Failed in ruby hook in its output. :) Dredd starts worker and connects to it in hooks and sends every transaction for execution in Ruby worker thru TCP socket.

So, you can write hooks in Ruby in doc/hooks/dredd-worker.rb for now . There's support only for before and after hooks, but it can be easily extended. Please, have a look at that and please let me know what you think and if it's useful for you. I'm also curious if it will work with FactoryGirl and company.

Disclaimer: It's mostly a throw-away code. It must be refactored into modules and tested. I was just curious if is anyone really willing to use it.

Any hints on usability of hooks are more than appreciated! :)

All the best!

[gonzalo-bulnes]

Hi @netmilk,

I glad you like the Dredd::Rack::Runner, I really enjoyed writting it : )
I preferred releasing earlier and ommitting the options passing mecanism in the first release, but I'm working on it, and I'll document it as soon as I can.

I'm trying to apply the same "release early, release often" idea to the possible division into several gems. I also thought about that, but for now I prefer to release first, get to something usable, stable, then extract as many gems as it is worth, one step at a time.

I'll try your fork, and try to write hooks, it's really great you're working on it! Many many thanks! I'll give you feedback.

Best regards!

[gonzalo-bulnes]

Hello!

Dredd::Rack v0.4.0 does allow to configure the runners when creating custom rake tasks. See this example. : )

@netmilk I started experimenting with hooks support from your fork and referred to it in #134.

[honzajavorek]

@gonzalo-bulnes @netmilk since Dredd now supports Ruby hooks and Dredd::Rack exists, do you think we can close this discussion now?

[gonzalo-bulnes]

Hi @honzajavorek, I think so.

[honzajavorek]

Thanks! 🎈