Mark pending blocks as failed if they succeed. #1267
Conversation
|
Spec failure is due to That is going to require an |
|
Pushed up a commit that addresses the issue, though build won't be green until rspec/rspec-mocks#544 is merged - chicken and egg problem. It would be possible with a few extra PRs to merge this in a way that keeps everything green but I don't think it's worth it. I'm not sure whether I like this solution overall :/ |
|
Just to give my feedback as a user: I actually never knew that when you used pending inside a test it had different behaviour than when you made the whole test (it) pending. Surely it's documented somewhere, but it's the sortof documentation I would look up when I wanted to know how something works, but in this case, pending already worked for me, so I never bothered to read the docs on it. So I think that improving the consistency is a clear improvement. I tend to use pending for a bunch of different scenarios. Without body, when I just write out some tests I have to write later. I make an existing test pending when it causes trouble, and isn't worthwhile to fix now (ie. an acceptance test which fails intermittently, and is testing it isn't very critical right now), this is basically a 'fixme'. Personally, I never used the pending case supported here, and must admit to not totally getting the usecase of this. Is this to do red-green-refactor without getting an actual red? Last thought is that I don't really like 'xit'. It feels really hackish. Otoh, maybe my regular use is rather hackish, so that is okay. |
|
@markijbema -- thanks for engaging with us on this! It's always great to get feedback from rspec users about changes we're making.
I've used this form of
@xaviershay -- I still owe you a full review of this. Still planning on doing that, but wanted to respond to @markijbema's feedback for now. |
Writing |
| define_method(name) do |*all_args, &block| | ||
| desc, *args = *all_args | ||
| options = Metadata.build_hash_from(args) | ||
| options.update(:pending => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block | ||
| options.update(:skip => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block |
There was a problem hiding this comment.
As we now have essentially two behaviours, "Pending" and "Skip" perhaps we should name these as such?
|
This LGTM apart from the fact that I wonder wether we should split |
|
@myronmarston I'm only using xit for situations where I shouldn't commit it. Sometimes I don't have the time to fix it properly though, and I do. This typically involves an integration test which fails because it is written poorly, and fails even though conceptually the functionality works. Sometimes the logic is unfortunately so complex that I decide to postpone this post my pull-request, and fix it in a seperate one (for instance, one of our helpers we use in capybara tests has to be rewritten because there is a critical flaw in it; rewriting it in my own pull-request also feels wrong, because it is totally unrelated). I like the suggestion of 'skip' though. I think that gives a nice semantic difference between skip and pending. Also, I really like that the rspec team dares to drop backwards compatibility to improve the product. Keep up the great work :) |
| @@ -77,6 +77,22 @@ Feature: pending examples | |||
| And the output should contain "Expected pending 'something else getting finished' to fail. No Error was raised." | |||
| And the output should contain "pending_with_passing_block_spec.rb:3" | |||
|
|
|||
| Scenario: pending any arbitrary reason, with a top-level block that passes | |||
There was a problem hiding this comment.
I'm not sure what "a top-level block" means in this context. I tend to think of "top-level" as being in the context of main but that's not the case here. How about "Using pending to define an example that is currently passing"?
There was a problem hiding this comment.
I wasn't happy with this description either.
|
Thanks for all the comments! Pretty sure I know how to address most of them, working on it today. |
|
See new commits, I think they address everything. I just rebased this and @myronmarston's new |
| # | ||
| # @see RSpec::Core::Pending#pending | ||
| def pending(*all_args, &block) | ||
| desc, *args = *all_args |
There was a problem hiding this comment.
Why not just define this as:
def pending(desc, *args, &block)
# ...
end...rather than pending(*all_args) and then split it on the first line?
There was a problem hiding this comment.
needs a default of nil, but yeah that works.
|
Build failures are the same expected RSpec mock errors. |
| @@ -888,26 +888,55 @@ def define_and_run_group(define_outer_example = false) | |||
| end | |||
| end | |||
|
|
|||
| %w[pending xit xspecify xexample].each do |method_name| | |||
| describe "::pending" do | |||
There was a problem hiding this comment.
I've mentioned this elsewhere, but I'm not a fan of using :: for message sends. I know you're being consistent, but my preference is to change all the places we work in to use the more standard . for message sends. Mind updating it?
|
See last 4 commits. The implementation of new |
|
Turns out changing the behaviour of I think all feedback has been addressed now. There is the open issue of whether we want to provide a different interface to formatters, since that won't be a backwards incompatible change I don't think it needs to be included here. |
|
|
|
urgh actually hold off, still a few more outstanding issues. |
|
This is still a bit of mess when interacting with mocks, but for the most part you just get specs marked as pending. See 41ab9a7#diff-3bac19fc9abdbdd90aaec18fabf14a95R211 for my current understanding of the problem. |
|
We could also just ditch this entire "speculatively run pending blocks" feature. It's not that useful and certainly complicates things a lot. |
I think I can do this. I'll take a stab at it later tonight (I'm only online for a couple minutes right now).
I'm personally a fan of this feature and it's never been a maintenance burden before. It also seems like the kind of thing that would be hard to achieve in an extension gem using public APIs. I can understand your frustration, though...I'll try to take a look at things later tonight to see where things stand. |
|
I'm mostly worried about shipping an inconsistent feature. Although it's inconsistent now... Actually, if we remove the pending-with-a-block-in-an-example feature, I think the problem goes away. You can still use |
|
Semantically, saying "this part of the spec is broken now, and should work in the future, but the rest of the spec still works fine and is valuable" seems unlikely and confusing to me. Instead, marking it pending halfway through (without a block) seems a lot clearer and easy to reason about. Even in the case of an acceptance spec, executing a pending block half-way through the spec is problematic since it puts you in an unknown state. To put it another way, I think the high-level functionality that used to be provide by pending-with-a-block ("let me know when this passes") is useful, but the old API for it is no good. By keeping the functionality but with the new API, and removing the old API, we get a much better product. |
|
Ah, I misunderstood what you were talking about. I thought you were talking about the feature as a whole, and not simply the "pending block from within an example" aspect (which was the only API to the feature in 2.x). Now that I get what you're saying, I think I agree: it doesn't seem needed or useful, and the new API seems much better. Actually, there is one case where it might still be useful: for code that is being written against both the RSpec 2.x API and RSpec 3.x API. Pending block within an example is the only API for this in RSpec 2.x, and if we remove that, there is no common API for both versions people can use. That's probably not a big deal though: I can't think of any rspec extension gems that use this (as it's generally something in a specific end-user spec, not in an extension gem). It might affect Sequel, though, as I know it's suite is meant to work on RSpec 1.x, 2.x and 3.x (see jeremyevans/sequel@101cc3a). I don't think that's a strong enough argument for it, though....so my vote is to axe it. |
|
Noticed that you have to do some effort to generate those fixtures locally. Is this something which needs to be done a lot? If so, you could consider leveraging Travis to generate them for you. |
Initially I was, but I changed :)
If we remove it completely, then |
|
@markijbema This would be the second time I've had to do it. They change pretty rarely. I really just need to get my dev environment set up properly... |
|
As I said earlier, my only strong opinion is that there should be a forward-compatible transition from RSpec 2.x, that:
Cutting the pending block feature seems to remove this possibility, or have I misunderstood something? |
|
@grddev you can either switch to |
|
@grddev I understand what you want, and I agree that it is preferable, but there isn't really a way if we want to call this feature 'pending' in 3, right? Is there a way rspec can differentiate between a pending with block which was meant to be run, and fail if succeed, and a block which was meant not to be run? The only solution I see which would be a little closer is allow something like: which shows that it should be run in 2.99, and have the run param be optional in 3. Would it be an idea to make the pending inside the block deprecated from version 3 onward? I don't think it has much more value above this syntax, right? I think that would solve the transition, since then you could transition in 2.99, and transition in 3 again. It's not perfect though. I don't see how you could solve this with one version though. For a gem I maintain we did some sequential upgrades to work around problems like this (reappropriating existing methods/names). You could also do it with 2.98 and 2.99 of course, but it feels a bit like a kludge. |
|
HTML fixtures need re-gen ... am hoping to be able to rebase this on top of #1306 to solve that problem. |
Might be worth warning and/or raising an error if a block is given to |
* Execute pending examples and mark as failed if they succeed. * Removed pending with a block. * Introduce `skip` method and metadata for old pending behaviour. This is a backwards-incompatible change. Implements #1208.
|
I'm done making changes to this, please give it a final once over.
We're already providing a warning in 2.99, I don't think it's worth the code here. In the common case you'll get flagged anyway, since the block won't execute and the example will pass, causing a failure. |
| an example is skipped using xexample | ||
| # Temporarily skipped with xexample | ||
| # ./temporarily_skipped_spec.rb:8 | ||
| """ |
There was a problem hiding this comment.
There's a 5th way: tagging an example with :skip => "reason why". Not a merge blocker.
|
Thanks for working so hard on this, @xaviershay. I left a few more comments, but any of them that are worth addressing can be addressed in a separate PR. I'm 100% OK with this being merged as is. |
|
Thanks! |
|
Merging. Will investigate potential bugs around the |
Mark pending blocks as failed if they succeed.
DO NOT MERGE: This needs final review.
See #1208 for background.
@grddev @myronmarston @JonRowe @markijbema