rspec / rspec-core

Added @wip scenario to expose this: https://github.com/rspec/rspec-core/commit/234d7632c71c39d1a31429a6483200812395b877

Added pending example as well: https://github.com/rspec/rspec-core/commit/b78129b34ae5f99b7873bc67667808b4e332b786

This applies to after(:all) as well - maybe these shouldn't be treated the same as failures, but should have their own messaging other than bubbling up an error? Or maybe bubbling up an error in before/after(:all) is perfectly fine feedback.

Giving this some more thought, I think an error in before/after all is perfectly fine feedback. If anybody is paying attention and thinks otherwise, make your case a comment in this issue, please.

david, to answer your questions from the issue logged in rspec-rails[1].

What version of rspec are you using? We're already up to beta.14 and you're citing beta.8. I'm definitely not seeing the error being swallowed. What I see is that the error bubbles up and execution stops entirely:

$ bundle exec rspec --version
2.0.0.beta.12
$ bundle show rspec
/home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-2.0.0.beta.12

I don't see the error being swallowed when running spec (or rspec) directly
either: truncating that part of the output when I opened the issue was sloppy on
my part.

However, the stack trace can be swallowed (and I think I have seen it swallowed
by autotest) because it prints to STDERR but the other output prints to STDOUT.
The problem is that the STDOUT output that lists the numbers of examples and
failures is exactly the same as in success, and indeed is even green when rspec
is run with -c.

One alternative would be to capture and report it in the same way as before(:each), but in before(:each) an error halts execution of the example - if it's in a before(:all), should we report all the examples in that group as failing? Or just bail out of the group on the first failure?

Thoughts?

Treating all examples in the group as failing seems like the right thing to do.
Personally I think not being able to set up an example counts as a failure, but
I can understand a view that says it's a different kind of error condition.

BTW I updated to 2.0.0.beta.15 and get the same behaviour (which is of course,
not surprising).

$ bundle exec rspec spec/models/zombie_spec.rb
Finished in 0.00036 seconds
1 example, 0 failures
/home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/backward_compatibility.rb:26:in `const_missing': uninitialized constant UndefinedConstant (NameError)
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-expectations-2.0.0.beta.15/lib/rspec/expectations/backward_compatibility.rb:6:in `const_missing'
    from /home/davidjh/devel/.sandbox/rspec-issue-before-all-badness/spec/models/zombie_spec.rb:6
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/hooks.rb:60:in `instance_eval'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/hooks.rb:60:in `run_hook!'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/example_group.rb:148:in `eval_before_alls'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/example_group.rb:173:in `run'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/command_line.rb:41:in `run_examples'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/command_line.rb:41:in `each'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/command_line.rb:41:in `inject'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/command_line.rb:41:in `run_examples'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/command_line.rb:24:in `run'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/formatters/base_formatter.rb:37:in `report'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/formatters/base_formatter.rb:152:in `sync_output'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/formatters/base_formatter.rb:34:in `report'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/command_line.rb:21:in `run'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/runner.rb:46:in `run_in_process'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/runner.rb:37:in `run'
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/gems/rspec-core-2.0.0.beta.15/lib/rspec/core/runner.rb:22
    from /home/davidjh/.rvm/gems/ree-1.8.7-2010.01/bin/rspec:19

[1] https://github.com/rspec/rspec-rails/issues/issue/110

I am either not able to figure out how, or simply don't have permission to,
reopen this issue, but in case it's not clear from my previous comment, I
consider this still an issue, at least in version 2.0.0.beta.15.

Again (sorry for being repetitive) the stack trace is printed to stderr, this is
fine. But what is printed to stdout is in green and strongly implies that
the examples pass, when they in fact do not. Worse, if stderr is swallowed for
whatever reason, the output will appear exactly as if the examples all pass.

On a (semi-)related note, thanks for rspec. It is much appreciated :-)

I confirm on this one.
I think considering whole group a failure is a good idea.

Does this relate to the fact that if you use this in an example group:

before(:each) { pending }

You get this error?

 uncaught throw :pending_declared_in_example

This is really inconvenient, as I do this all the time to selectively put examples in the background to deal with later :-(

@ashleymoran - unrelated.

FYI - to make a group pending, just say so at the group level:

describe "something", :pending => true do

That makes all of the examples in the group pending.

HTH,
David

off-topic: wow, another trick I never knew.

Thanks - I didn't know that. Just checked and it works fine.

Is pending in before blocks deprecated, or is that still a bug? The error message is not very helpful, and looks like it's unanticipated behaviour.

@markiz - I like these! Using focus: true with the appropriate RSpec config is really useful too, in case you've not seen it (on David's blog).

@ashleymoran - deprecated suggests it was supported before :) Please open a separate issue and we can discuss there.

Ha :) I've opened #77 to discuss pending in before.

is this bug the same as:

describe 'it' do
  
  it 'should' do
  end
  after(:all) do
    raise 'bad'
  end
end

=>

Finished in 0.019 seconds
1 example, 0 failures
D:/installs/ruby-1.9.2-rc1-i386-mingw32/ruby-1.9.2-rc1-i386-mingw32/lib/ruby/gems/1.9.1/gems/rspec-core-2.0.0.beta.17/lib/rspec/core/hooks.rb:42:in `rescue in run_in': undefined method
`set_exception' for nil:NilClass (NoMethodError)
        from D:/installs/ruby-1.9.2-rc1-i386-mingw32/ruby-1.9.2-rc1-i386-mingw32/lib/ruby/gems/1.9.1/gems/rspec-core-2.0.0.beta.17/lib/rspec/core/hooks.rb:35:in `run_in'

I've been slamming my head against this as well.

When using the textmate bundle everything shows in html mode. And of course the formatter doesn't work with real exceptions, so I'm left seeing useless message:

Exception encountered: # backtrace: /Users/kevin/.rvm/gems/ree-1.8.7-2010.02/gems/dm-validations-1.0.0/lib/dm-validations.rb:141:in name' /Users/kevin/.rvm/gems/ree-1.8.7-2010.02/gems/dm-validations-1.0.0/lib/dm-validations.rb:141:insend' /Users/kevin/.rvm/gems/ree-1.8.7-

The exception message doesn't escape the < and > so its not even shown and the backtrace with no newlines is painful.

It'd be nice if it was at least rendered nicely as a part of the formatter.

Alternatively, perhaps there should be an implicit first test that any before(:all) blocks don't raise an exception.

I think this is the same issue (or please correct me - if let is a different beast I'll move this). Anything in a class definition causes RSpec to blow up spectacularly:

module DomainLib
  describe Representation do
    let(:representation_class) {
      Class.new do
        extend Representation
        properties :a, :b, :c
      end
    }
    let(:representation) {
      representation_class.new(a: 1, b: 2)
    }
    it "exposes the property attributes" do
      representation.a.should eq 1
      representation.b.should eq 2
      representation.c.should be_nil
    end
  end  
end

You don't get a runner output, just the following:

/Users/ashleymoran/.rvm/rubies/ruby-1.9.2-rc2/bin/ruby -S rspec /Users/ashleymoran/Dropbox/PatchSpace/Development/work-source/twisty_lists/twisty_lists/spec/domain/checklist_template_spec.rb
/Users/ashleymoran/Dropbox/PatchSpace/Development/work-source/twisty_lists/twisty_lists/app/domain/checklist_template.rb:10:in `<class:Representation>': undefined method `properties' for ChecklistTemplate::Representation:Class (NoMethodError)
    from /Users/ashleymoran/Dropbox/PatchSpace/Development/work-source/twisty_lists/twisty_lists/app/domain/checklist_template.rb:8:in `<class:ChecklistTemplate>

@ashleymoran - seems unrelated. Separate issue please, and please include the def for Representation.

I haven't written that yet :) Will file when I've got the code working...

@ashleymoran - aha! So your issue is that the error is not being captured, yes? If so, then it's the same issue :)

Yep - that's the one! Did I just save myself a ticket? =)

FYI - I'm keen to get this resolved before a release candidate of rspec-2. I've just been very busy wrapping up The RSpec Book and this issue relates to some deeper architectural questions so I don't want to just hack together a fix.

@ashleymoran - your self and everyone else's selves.

Cool cool. I agree, if it's a fundamental design issue, it's definitely one that needs resolving properly before an RC. I always assume (being used to RSpec 1.3) that if the output doesn't get formatted that I've made a syntax, so it puts my brain in the wrong type of bug-hunting mode. Lemme know if you want me test anything.

Started work on this. Introduced a Reporter which delegates to n formatters. It also keeps track of all the counts, so formatters don't have to.

Don't forget you can do:

it "raises an error", :full_backtrace => true do ....

Thanks justinko, didn't know about that!

rspec-2 filtering is quite flexible, but you have to type things into an RSpec.configure block to use them. It would be nice to access them directly from the command line, and make it similar to the API for cucumber (from the command line, not internally).

Possibilities include:

• explicit hash key

describe "something" do
  it "does something", :tags => [:foo, :bar] do
    ...

This would run with either of these (supporting the @ prefix to ease confusion when switching between rspec and cucumber commands):

$ rspec spec -t foo
$ rspec spec -t bar
$ rspec spec -t @foo
$ rspec spec -t @bar

• keys and/or key/value pairs

describe "something" do
  it "does something", :focus => true do
    ...
  it "does something slow", :slow => true do
$ rspec spec -t focus # runs the first one
$ rspec spec -t focus:true # runs the first one
$ rspec spec -t ~@slow # runs all but the second one
$ rspec spec -t ~@slow:true # runs all but the second one

• profiles

RSpec.configure do |c|
  c.profile :wip => "--tags @wip"
end
describe "something" do
  it "does something", :wip => true do
    ...
$ rspec spec --profile wip

I'm interested in hearing thoughts about these ideas and any new ideas as well.

I like the explicit hash key (:tags => [:foo, :bar]) as it maps most closely to what people are already used to from Cucumber.

The second idea seems a little more complex to me, and I don't see what the extra complexity buys us.

The idea of using "~" on the command line to negate the selection would work for the explicit hash key variant as well:

$ rspec spec -t ~@foo

Not sure if there are any shells out there though that treat "~" as a glob-like special character and automatically expand it before passing it off to the rspec process. Might be worth checking that out...

What about making the bag of parameters that's passed to the #it or #describe block be able to be either an array (of symbols) or a hash, or a mixture of both.

That way the keys can be treated as tags implicitly, but you can also give value to tags (something cucumber doesn't let you do).

e.g.

describe "something" do
  it "does something", :focus, :ruby => 1.9 do
  ...

This is nice and terse - IMO the => true or :tags => bit is just noise really.

Or would that conflict with other existing behaviour for these arguments?

Another thought is that you can use the '@' character in the tags when defining them in the describe blocks, I think that might make things easier to understand. You could either use strings or symbols I reckon:

:@focus, '@ruby' => 1.9

Dunno, maybe that doesn't look so nice. At least it's obvious they're tags though.

@wincent - the reason to support the key/val pair is because it's the same structure we use in RSpec.configure for filtering:

RSpec.configure do |c|
  c.filter_run :focus => true
end

This way we'd use the same markup (for lack of a better term) in the spec files and be able to access it from the config block or the command line.

The ~@tagname format is what cucumber uses, which is the main reason to support it. If a shell doesn't allow for that it wouldn't allow it for cucumber either. Not that that makes it good, but it does make it consistent across the two :)

@mattwynne - rspec-1 supports arbitrary # of args, but that dates back to pre-nesting days so you could say describe SomeClass, "#some_method". I'd be a bit nervous to change the way that works now (one more point of confusion in the upgrade process), but it's not out of the question.

Not sure that :@focus is obvious as a tag to anyone other than cucumber users :)

Thanks to you both for the feedback. Any others?

Ruby version?

ruby 1.8.6 (2010-02-04 patchlevel 398) [i386-mingw32]
rspec version is 1.3.0

The problem also exists with:
ruby 1.8.7 (2010-01-10 patchlevel 249) [i386-mingw32]

But everything is working with:
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-mingw32]

So it seems that this problem exists only with Ruby 1.8

i think that this is the problem I had when I updated to beta13. It does not happen in beta12

Is this fixed with beta 14?

Seems so, thanks a lot :D

David, I think I might have a solution for this:

When "it_should_behave_like" (or the alias) is called, it sets the following metadata on the ExampleGroup:

Name of the shared example group.
File path and line number of "it_should_behave_like" method call.

When an example fails, RSpec will check to see if that example exists in the shared example group. If it does, it will append to the output the file path and line number (stored in the ExampleGroup metadata).

What do you think?

Oh nice I see that metadata is already there :-)

Patch: https://github.com/justinko/rspec-core/commit/a62a575f

Cannot test this -- linecache 0.43 won't install with jruby 1.5.1

doesn't jruby come with linecache already installed or built in?

I can see showing the classname, but value do you get from the failure/error distinction?

The failure/error distinction comes a bit out of the first. I was figuring that you don't want to show ExpectationNotMetError, because it's obvious in it's description, you just want to provide the clean message like we're seeing now.

I'd think it would give a bit cleaner output, that's all. It's just nice to have.

Would you please post a separate issue for the E/F distinction. Though related, these are two separate requests.

Sure: https://github.com/rspec/rspec-core/issues/issue/93

Please post the initialize method in Activity and the full backtrace.

Thx

I'll work on getting that to you. Activity is an ActiveRecord model (Rails.)

I did find this patch https://gist.github.com/401267 and applying it fixes MOST, but not all of the failed specs.

Here is the full backtrace from a failed spec:

ArgumentError in 'Activity '
wrong number of arguments (1 for 0)
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/subject.rb:43:in `block in implicit_subject'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/subject.rb:77:in `instance_eval'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/subject.rb:77:in `subject'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/subject.rb:91:in `should'
spec/models/activity_spec.rb:18:in `block (2 levels) in <top (required)>'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_methods.rb:40:in `instance_eval'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_methods.rb:40:in `block in execute'
/Users/pete/.rvm/rubies/ruby-1.9.2-rc2/lib/ruby/1.9.1/timeout.rb:44:in `timeout'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_methods.rb:37:in `execute'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_group_methods.rb:214:in `block in run_examples'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_group_methods.rb:212:in `each'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_group_methods.rb:212:in `run_examples'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_group_methods.rb:103:in `run'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/runner/example_group_runner.rb:23:in `block in run'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/runner/example_group_runner.rb:22:in `each'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/runner/example_group_runner.rb:22:in `run'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/runner/options.rb:152:in `run_examples'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/runner/command_line.rb:9:in `run'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/bin/spec:5:in `<top (required)>'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/bin/spec:19:in `load'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/bin/spec:19:in `<main>'

I'm also getting this at the top of the output:

No description supplied for example declared on spec/models/activity_spec.rb:18:in `block in <top (required)>'

A similar error occurs in pending specs, such as:

it "should do something"

The backtrace in this case is:

ArgumentError in 'Activity should do something'
wrong number of arguments (1 for 0)
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_group_methods.rb:80:in `block in pending_implementation'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_methods.rb:40:in `instance_eval'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_methods.rb:40:in `block in execute'
/Users/pete/.rvm/rubies/ruby-1.9.2-rc2/lib/ruby/1.9.1/timeout.rb:44:in `timeout'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_methods.rb:37:in `execute'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_group_methods.rb:214:in `block in run_examples'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_group_methods.rb:212:in `each'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_group_methods.rb:212:in `run_examples'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/example/example_group_methods.rb:103:in `run'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/runner/example_group_runner.rb:23:in `block in run'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/runner/example_group_runner.rb:22:in `each'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/runner/example_group_runner.rb:22:in `run'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/runner/options.rb:152:in `run_examples'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/lib/spec/runner/command_line.rb:9:in `run'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/gems/rspec-1.3.0/bin/spec:5:in `<top (required)>'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/bin/spec:19:in `load'
/Users/pete/.rvm/gems/ruby-1.9.2-rc2/bin/spec:19:in `<main>'

This is similar to the issue pointed out at https://rspec.lighthouseapp.com/projects/5645/tickets/971

I'm not sure if this is related, but I'm seeing

Failure/Error: its(:legacy_password) { should be_nil }
implicit argument passing of super from method defined by define_method() is not supported. Specify all arguments explicitly.

I'm unable to reproduce this problem. What versions of ruby and rspec are you running, and what OS?

ruby 1.9.2dev (2009-07-18 trunk 24186) [i386-darwin10.4.0]
rspec 2.0.0.beta.16

The regex is looking for FAILED whereas my output says Failure

This commit fixes it for the base text formatter. Instead of copying output text into the spec (which I think was outdated) it gets the text from the formatter itself
https://github.com/snappycode/rspec-core/commit/9344bcff9c658cc5b5cda78c85e8953644009463