Skip to content

keep_source_order is using test run order, not source order #192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zofrex opened this issue Jan 19, 2015 · 9 comments · Fixed by #245
Closed

keep_source_order is using test run order, not source order #192

zofrex opened this issue Jan 19, 2015 · 9 comments · Fixed by #245

Comments

@zofrex
Copy link

zofrex commented Jan 19, 2015

I've set config.keep_source_order = true but my output is ordered by the order the tests run in, not the order they appear in the source code. Is this expected?

Edit: I have config.order = "random" in my RSpec configuration, that's probably important information.

@oestrich
Copy link
Contributor

Looking at the source code I don't know if it ever kept the examples in the source order if rspec was running them in a random order. The only thing it does is not sort by description, see here.

I think the way around this is already built in. Can you try rake docs:generate:ordered? It sets the order to default, which should hopefully be the source order.

@indiesquidge
Copy link

@oestrich I am running into trouble with rake docs:generate:ordered. It seems to load the doc/api/index.json data in the same order that rake docs:generate does, which is to say alphabetically rather than based on the source, which is what I expected (unless I'm misinterpreting ordered).

I was trying to dig around the rspec_api_documentation source code for some indication as to what this line does for me.

t.rspec_opts = ["--format RspecApiDocumentation::ApiFormatter", "--order default"]

What is --order default? Are there other ordering options? If so, how can I find them? How should I get the docs to match the order that the specs are written in? Any help would be greatly appreciated.

I know that most people create or comment on issues about stuff that is broken or that they don't like. But just know that I think this project is awesome, and I love the work you guys are doing on it. Keep it up! 😃

UPDATE:
I am finding that --order is actually something to do with RSpec itself? I see options for default and rand. default seems to be what I want (groups/examples are loaded in the order in which they are declared), but I can't get that to work with this library. Even if I change the option to --order rand, the docs don't seem to randomize either. I have removed the whole doc folder and repeated the steps, again with no luck seeing any changes. Seems suspicious when nothing changes at all. Kind of at a loss as to what I should be doing in order to get this ordered the way I would like.

UPDATE:
Ugh, I should read the issue I'm commenting on more carefully! Setting config.keep_source_order = true so that my helper filed looked as such

require 'rails_helper'
require 'rspec_api_documentation'
require 'rspec_api_documentation/dsl'

RspecApiDocumentation.configure do |config|
  config.format = :json
  config.curl_host = 'http://localhost:3000'
  config.keep_source_order = true
  config.api_name = "Rales Engine API V1"
end

I didn't realize that had to be set in order for rake docs:generate:ordered to work. I apologize for not seeing it sooner!

@oestrich
Copy link
Contributor

@indiesquidge Glad I could help! Got everything working?

@indiesquidge
Copy link

@oestrich well, now I'm curious about the difference in these rake tasks. When I have config.keep_source_order = true set, that seems to make the difference in the order, not the rake task; rake docs:generate and rake docs:generate:ordered appear to do the exact same thing.

Let's say, hypothetically, that I would like to have both tasks working, where rake docs:generate orders them alphabetically (seems to be the default), and rake docs:generate:ordered orders them based on the source. How would I go about getting that?

I suppose it all comes back to that one line I mentioned in my first comment and what that line is even doing for me/how I can manipulate it to do something different.

@oestrich
Copy link
Contributor

I think at one point we didn't order tests at all so the rspec --order mattered a lot. It probably happened around this line being changed

ordered_examples = configuration.keep_source_order ? examples : examples.sort_by(&:description)

@indiesquidge
Copy link

Ah, I see. So at this point, the --order is kind of irrelevant since everything now depends on config.keep_source_order, is that correct? So it's a bit harder if I want to have rake tasks that will change the order now then, as I would have to set config.keep_source_order in the task, yeah?

@indiesquidge
Copy link

I was thinking I could just configure it in the rake task, like so

desc 'Generate API request documentation from API specs'
RSpec::Core::RakeTask.new('docs:generate') do |t|
  RspecApiDocumentation.configure do |config|
    config.keep_source_order = false
  end
  t.pattern = 'spec/acceptance/**/*_spec.rb'
  t.rspec_opts = ["--format RspecApiDocumentation::ApiFormatter"]
end

and the respective ordered task as well

desc 'Generate API request documentation from API specs in source order'
RSpec::Core::RakeTask.new('docs:generate:ordered') do |t|
  RspecApiDocumentation.configure do |config|
    config.keep_source_order = true
  end
  t.pattern = 'spec/acceptance/**/*_spec.rb'
  t.rspec_opts = ["--format RspecApiDocumentation::ApiFormatter"]
end

But this doesn't seem to work. Its back to having config.keep_source_order set to false even if I set it to true in the rake task.

@davidstosik
Copy link
Contributor

I've been facing the same problem. Multiple times, I thought I'd solved it after changing something in the way I generate the doc, but nothing works. Note that, as @zofrex, I have config.order = "random" in my RSpec configuration, and commenting the line fixes the problem (documentation is sorted as the source).

  • setting config.keep_source_order = true does not work
  • using rake docs:generate:ordered does not work (although the output shows that --option default was appended to the rspec command
  • doing both above together didn't fix the problem either

Now checking RSpec documentation (rspec --help), the problem is clear: the --order options changed: while old versions of RSpec will use --order default to run tests in the source order, more "recent" (3 and above?) versions use --order defined. (See this commit on rspec-core repository.)

I also confirmed that no backwards compatibility is assured by running bin/rspec -fd --order default on my local, which ran my tests in a random order (and also output the random seed at the end).

davidstosik added a commit to davidstosik/rspec_api_documentation that referenced this issue Nov 18, 2015
As I didn't know for sure what exact version the change happened (major > 2?),
I checked inside RSpec::Core::Ordering::Registry.
If we're comfortable with making the switch on version 2 to 3 of RSpec, I can
change the condition to check `Gem.latest_spec_for('rspec-core').version.segments[0]`.

This should fix zipmark#192.
@davidstosik
Copy link
Contributor

If I may, I'd like to detail what I understood, so that it's not lost.

  • keep_source_order = true actually just means "Do not sort by description. Instead, the order in which tests were ran will be used (ie. if tests run in random order, result will be random too)." I'd suggest a rename, such as sort_by_description, with a default to true, for backwards compatibility.
  • The difference between rake docs:generate and rake docs:generate:ordered is that the latter explicitly tries to tell rspec not to randomize the tests (--order default in old RSpec versions, but should be --order defined in recent versions, see PR Pass rspec the right --order option (default or defined). #245 ).
  • For some people, rake docs:generate will be sorted, for others it won't. It depends on their project's setup, and whether rspec was set to run tests randomly or not (my spec_helper.rb file contains config.order = :random, so I need to use rake docs:generate:sorted if I want to get a sorted documentation).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants