eigenclass logo
MAIN  Index  Search  Changes  PageRank  Login

rcov 0.8.1: compatibility with Ruby 1.8.6-p11[01], intentional testing with RSpec, etc.

Version 0.8.1 of the rcov code coverage tool for Ruby addresses the problems experienced by ruby 1.8.6-p11[01] (and in particular Leopard) users, and includes a few new features.

Update [2007-11-20] REXML workaround incorrectly applied to 1.8.6 < p110, fixed in 0.8.1.1.

Update [2007-11-22] Further bugs found in REXML (1.8.6-p11[01]), addressed in 0.8.1.2. rcov has been tested on 1.8.5 and 1.8.6-p110.

If you are new to rcov, take a look at this sample report. In addition to indicating which code has been covered by your tests, rcov allows you to navigate through your code easily. rcov records where each method was called from and can generate fully cross-referenced reports, letting you inspect the control flow. This is most useful when you're trying to understand the overall organization of third-party code or you're refactoring.

cross-refs-teaser.png

"Spec-only" mode

You can indicate that only code executed inside a spec must be considered covered (this is similar to --test-unit-only).

Here's a minimal example. Consider this target code: 

# bowling.rb
class Bowling
  def hit(pins)
  end

  def score
    0
  end

  def num_throws
    1
  end
end

and the following spec: 

require 'rubygems'
require 'spec'
require 'bowling'

describe Bowling do
  before(:each) do
    @bowling = Bowling.new
  end

  it "should score 0 for gutter game" do
    20.times { @bowling.hit(0) }
    @bowling.score.should == 0
  end
end

Bowling.new.num_throws

If you invoke rcov with the --spec-only option, you'll obtain:

$ rcov --text-coverage --no-color --spec-only bowling_spec.rb 
.

Finished in 0.016369 seconds

1 example, 0 failures
 ================================================================================
bowling.rb
================================================================================
!! # bowling.rb
!! class Bowling
     def hit(pins)
     end
   
     def score
       0
     end
!! 
!!   def num_throws
!!     1
!!   end
!! end
   
================================================================================
bowling_spec.rb
================================================================================
!! require 'spec'
!! require 'bowling'
   
   describe Bowling do
     before(:each) do
       @bowling = Bowling.new
     end
   
     it "should score 0 for gutter game" do
       20.times { @bowling.hit(0) }
       @bowling.score.should == 0
     end
   end
!! 
!! Bowling.new.num_throws

Note how the num_throws method is not covered: it wasn't executed inside the spec. This allows you to differentiate deliberate from "accidental" code coverage.

Download

You can get it from rcov: code coverage for Ruby, or install it via RubyGems with

 gem install rcov

(if you get an older version/a 404 error, just wait for a while until the gem propagates to rubyforge's mirrors)

Change summary (since 0.8.0)

Features

  • you can use an existent data file to generate coverage reports without having to execute your code again
  • --spec-only: only consider covered code executed inside a spec (tested with RSpec 1.0.5, 1.0.8)
  • --charset can be used to specify the charset in the Content-Type declaration

Bugfixes

  • workaround for bugs in Safari, IE (self-closing anchors break colorization)
  • workaround for bugs in REXML shipped with 1.8.6-p11[01]
  • rethrow exceptions generated by traced scripts
  • compatibility with Ruby < 1.8.5
  • (0.8.1.1) REXML workaround incorrectly applied to 1.8.6 < p110
  • (0.8.1.1) --spec-only should work with RSpec trunk and detect when it cannot work

Bugfix releases

0.8.1.1

  • REXML workaround incorrectly applied to 1.8.6 < p110
  • --spec-only should work with RSpec trunk and detect when it cannot work

0.8.1.2

  • REXML from 1.8.6-p11[01] is more broken than anticipated; more invasive workarounds. Tested on 1.8.6-p110 and 1.8.5. Note that code using REXML to generate XML will behave differently under rcov in 1.8.6-p11[01] (to begin with, it won't crash in REXML::Document#write).

Thanks to

Lee Marlow

  • patch allowing to run rcov against a data file with no input code

Kurt Stephens

  • patch to rethrow any exceptions generated by the traced scripts after report generation; notably SystemExit, allowing to use the exit code from test runners under rake.

Brian Candler

  • found compatibility issues with the REXML lib included in ruby-1.8.6-p110 and provided a workaround

Mat Schaffer

  • reported missing line colorization on Safari and probably other browsers, owing to self-closing <a> not being handled properly despite being valid XHTML 1.0.

Sam Granieri

  • tested workaround for REXML bug

Kosmas Schütz, Daniel Berger, François Beausoleil, Bil Kleb

  • provided information about the ruby-1.8.6-p11[01] REXML problems

Jeremy Hinegardner

  • more info about REXML's bugs in 1.8.6-p11[01]

Latest release - Chad Humphries (2007-11-19 (Mon) 14:39:00)

In rspec trunk class Spec::DSL::Example has changed to class Spec::Example::ExampleGroup.

Also, I get odd Rexml::Formatter errors with this release (vs the last release)

mfp 2007-11-19 (Mon) 15:16:57

Ouch, I wanted to believe that that class wouldn't change soon :-| I will have to check the RSpec version.

What ruby version are you using? Sam Granieri tested it on 1.8.6-p110; I have tried it with 1.8.5. Also, could you paste the errors you're getting? Thanks.

mfp 2007-11-19 (Mon) 15:19:01

I've realized the workaround might fail with 1.8.6 < p110; is that what you're using?

mfp 2007-11-20 (Tue) 03:14:30

Fixed the REXML workaround, changed --spec-only to use Spec::Example::ExampleGroup if there's no Spec::DSL::Example. Thanks.

rcov 0.8.1 + Ruby 1.8.6 p36 on 10.5.1 = troubles - Ruben (2007-11-19 (Mon) 17:56:44)

Using Leopard 10.5.1 with ruby 1.8.6 (2007-06-07 patchlevel 36) [universal-darwin9.0] raises:

 uninitialized constant REXML::Formatters

when running rake spec:rcov.

Editing report.rb and commenting out the 10.5.1 fix solves the problem for me.

mfp 2007-11-20 (Tue) 03:11:59

Yup, the workaround only works for 1.8.6 >= p110. Fixed in 0.8.1.1.

I still seem to be having problems - Greg Campbell (2007-11-20 (Tue) 17:55:11)

using:

ruby 1.8.6 (2007-09-23 patchlevel 110) [x86_64-linux]

rcov 0.8.1.1 2007-11-20

I get the following error (I hope it isn't a problem to post the whole stack trace). Any help which can be provided would be appreciated.

/usr/local/lib/ruby/1.8/rexml/document.rb:186:in `write': undefined local variable or method `transitive' for <UNDEFINED> ... </>:REXML::Document (NameError)
from (eval):93:in `pretty'
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/lib/rcov/report.rb:761:in `format_overview'
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/lib/rcov/report.rb:796:in `create_index'
from (eval):104:in `create'
from (eval):80:in `tracking_additions'
from (eval):103:in `create'
from (eval):372:in `x_'
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/lib/rcov/report.rb:796:in `create_index'
... 22 levels...
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/lib/rcov.rb:640:in `each'
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/lib/rcov.rb:640:in `dump_coverage_info'
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/bin/rcov:421
from /usr/local/lib/ruby/1.8/test/unit.rb:278

Kurt Schrader 2007-11-20 (Tue) 19:29:14

You need to go into /usr/local/lib/ruby/1.8/rexml/document.rb and change `transitive' to 'trans' on line 186 in order to get it to work.

See http://www.germane-software.com/projects/rexml/ticket/115

mfp 2007-11-21 (Wed) 02:30:10

This is exactly what the workaround in lib/rcov/report.rb is meant to solve; do you see any reason why the following wouldn't work? (This is executed after require 'rexml/document'.) 

# Try to fix bug in the REXML shipped with Ruby 1.8.6
# This affects Mac OSX 10.5.1 users and motivates endless bug reports.
if RUBY_VERSION == "1.8.6" && defined? REXML::Formatters::Transitive &&
   RUBY_RELEASE_DATE < "2007-11-04"
    class REXML::Document
        def write( output=$stdout, indent=-1, trans=false, ie_hack=false )
            if xml_decl.encoding != "UTF-8" && !output.kind_of?(Output)
                output = Output.new( output, xml_decl.encoding )
            end
            formatter = if indent > -1
                if trans
                    REXML::Formatters::Transitive.new( indent, ie_hack )
                else
                    REXML::Formatters::Pretty.new( indent, ie_hack )
                end
            else
                REXML::Formatters::Default.new( ie_hack )
            end
            formatter.write( self, output )
        end
    end
end

Jeremy Hinegardner 2007-11-21 (Wed) 21:53:50

It appears that rexml/formatters/transitive.rb has not been required by the time the lib/rcov/report.rb monkey patches REXML::Document. I did a couple of traversals through the rexml codebase and I was unable to find where, between rcov and rexml that rexml/formatters/transitive.rb was ever required. I wasn't able to work through the nested evals at that point.

I put this at the top of lib/rcov/report.rb

 require 'rexml/formatters/transitive'

I also applied these patches

After that, running rcov through the rspec rake task worked correctly.

mfp 2007-11-22 (Thr) 04:15:46

Thanks, I'm now requiring it explicitly and patching REXML::Formatters::Transitive at runtime. rcov 0.8.1.2 works for me on 1.8.5 and 1.8.6-p110.

Jeremy Hinegardner 2007-11-22 (Thr) 11:34:12

Yup, I agree, all is working good for me. I backed out the rspec and rexml patches linked previously, updated to rcov 0.8.1.2 and all is well. Thanks!

rcov + Safari 3 - Mark Rowe (mrowe@apple.com) (2007-11-24 (Sat) 01:40:04)

To clarify a little on the Safari issue here: it's not a bug in Safari.

Safari's (really the underlying WebKit engine) handling of the code is correct given HTML 5’s error handling rules. The underlying issue here is that rcov is generating XHTML code such as <a name="line1"/> but is being served as text/html to web browsers. If this XHTML code is served to WebKit with the correct MIME type (application/xml+xhtml) then WebKit will interpret it as XML and render it as intended. Since it is not served with the correct MIME type, browsers correctly treats it as invalid HTML and applies error handling rules to attempt to make sense of it. Currently all browsers do this in slightly different fashions but the upcoming HTML 5 specification attempts to clarify and standardise this behaviour.

The correct fix for this problem is to not generate XHTML and claim it is text/html. Either serve it as application/xml+xhtml or generate valid HTML. Ian Hickson, the driving force behind the HTML 5 specification process, has a great write-up on this issue at http://www.hixie.ch/advocacy/xhtml. They key relevant point to this rcov issue is under the subheading "Documents sent as text/html are handled as tag soup [1] by most browsers".

Please get in touch with me if you have any questions about this.

mfp 2007-11-26 (Mon) 12:32:12

Thank for the clarification. The reference you provided was very helpful, in particular this:

You can't trigger from the DOCTYPE since the W3C might introduce new XHTML DOCTYPEs in future, so you don't know which DOCTYPEs to look for. (Not to mention that DOCTYPEs are optional for well-formed XHTML documents, DOCTYPE parsing is hard, DOCTYPEs may be hidden in comments, and DOCTYPE sniffing has been called harmful by many leading figures at the W3C and elsewhere.)

I will consider switching to HTML 4.01, and document the need to serve contents as application/xhtml+xml.

Rails + RSpec - stiff (2007-11-28 (Wed) 01:00:50)

I'm trying to enable "--spec-only" in my Rails application rcov.opts, but when I add it and issue "rake spec:rcov", rcov starts complaining that it didn't find any files to analyze. Any ideas why is this the case?

By the way, do you think the cross referencing mechanism could be extraced from rcov and used as better ctags/rtags (working in both sides) ie. in Emacs/Vim/TextMate? I was looking for a way to add some sort of "find usage" for Ruby to Emacs for a long time...

C language extension doesn't build using Ruby 1.9 - M. Edward (Ed) Borasky (2007-12-23 (Sun) 23:21:34)

I'm trying to install rcov-0.8.1.2 with a freshly-built Ruby 1.9 from Subversion. It seems to be working except for the part where it builds the C extension:

$ ruby setup.rb 
---> bin
<--- bin
---> lib
---> lib/rcov
<--- lib/rcov
<--- lib
---> ext
---> ext/rcovrt
/home/znmeb/cougar/ruby-1.9-testing/install/bin/ruby  
/home/znmeb/cougar/ruby-1.9-testing/rcov-0.8.1.2/ext/rcovrt/extconf.rb
creating Makefile
<--- ext/rcovrt
<--- ext
---> bin
updating shebang: rcov
<--- bin
---> lib
---> lib/rcov
<--- lib/rcov
<--- lib
---> ext
---> ext/rcovrt
make
gcc -I. -I/home/znmeb/cougar/ruby-1.9-testing/install/include/ruby-1.9/x86_64-linux -I/home/znmeb/cougar/ruby-1.9-testing/install/include/ruby-1.9 -I/home/znmeb/cougar/ruby-1.9-testing/rcov-0.8.1.2/ext/rcovrt   -fPIC -march=athlon64 -O3   -o callsite.o -c callsite.c
callsite.c:2:17: error: env.h: No such file or directory
callsite.c:3:18: error: node.h: No such file or directory
callsite.c:4:16: error: st.h: No such file or directory
callsite.c: In function 'callsite_custom_backtrace':
callsite.c:82: error: 'ruby_frame' undeclared (first use in this function)
callsite.c:82: error: (Each undeclared identifier is reported only once
callsite.c:82: error: for each function it appears in.)
callsite.c:84: error: 'NODE' undeclared (first use in this function)
callsite.c:84: error: 'n' undeclared (first use in this function)
callsite.c:89: error: dereferencing pointer to incomplete type
callsite.c:90: error: dereferencing pointer to incomplete type
callsite.c:92: error: dereferencing pointer to incomplete type
callsite.c:92: error: dereferencing pointer to incomplete type
callsite.c:93: error: dereferencing pointer to incomplete type
callsite.c:93: error: dereferencing pointer to incomplete type
callsite.c:94: error: dereferencing pointer to incomplete type
callsite.c:96: error: dereferencing pointer to incomplete type
callsite.c:96: error: dereferencing pointer to incomplete type
callsite.c:101: error: dereferencing pointer to incomplete type
callsite.c: At top level:
callsite.c:121: error: expected ')' before 'event'
callsite.c: In function 'cov_install_callsite_hook':
callsite.c:161: error: 'coverage_event_callsite_hook' undeclared (first use in this function)
callsite.c:162: error: too few arguments to function 'rb_add_event_hook'
callsite.c: In function 'cov_remove_callsite_hook':
callsite.c:176: error: 'coverage_event_callsite_hook' undeclared (first use in this function)
make: *** [callsite.o] Error 1
setup.rb:655:in `raise': system("make") failed (RuntimeError)
from setup.rb:655:in `command'
from setup.rb:664:in `make'
from setup.rb:1258:in `setup_dir_ext'
from setup.rb:1532:in `block in traverse'
from setup.rb:1549:in `dive_into'
from setup.rb:1530:in `traverse'
from setup.rb:1534:in `block (2 levels) in traverse'
from setup.rb:1533:in `each'
from setup.rb:1533:in `block in traverse'
from setup.rb:1549:in `dive_into'
from setup.rb:1530:in `traverse'
from setup.rb:1524:in `block in exec_task_traverse'
from setup.rb:1519:in `each'
from setup.rb:1519:in `exec_task_traverse'
from setup.rb:1246:in `exec_setup'
from setup.rb:996:in `exec_setup'
from setup.rb:813:in `invoke'
from setup.rb:773:in `invoke'
from setup.rb:1578:in `<main>'

Is there some path or environment variable I need to set?

znmeb at cesmail dot net

Segfaults with RSpec (even 1.1) - Venkat (2008-01-01 (Tue) 10:46:30)

Thank you very much for RCov. We find it valuable to check for missing tests.

We have a suite of RSpec examples with a number over a 2000 spanning across 300 spec files. For sometime I have stopped running RCov due to segfaults. When I read your blog about 0.8.0 release with segfault fix, we have tried it again but we continue to get segfaults.

Here is what we get:

../vendor/plugins/rspec/lib/spec/example/example_group_methods.rb:142: [BUG] Segmentation fault ruby 1.8.5 (2006-08-25) [i486-linux]

Aborted (core dumped) rake aborted!

Running RCov 0.8.1.2 with RSpec 1.1.1 in Ubuntu Feisty.

The line here the segfault is reported, the code simply doing a array.flatten.compact.

I would appreciate your help in fixing it. I wish I have some knowledge about these codes like rcov to be able to contribute a patch but sadly I don't.

The --spec-only option also doesn't work and says no matching files found. I was guessing that by turning of coverage behaviour from code outside of specs, I could workaround the problem but sinc --spec-only doesn't work I am out of luck. Can you please provide a example of how to do this?

Thank you, Venkat.

problems with --spec-only on newest rspec - hrvoje (2008-01-24 (Thr) 08:13:47)

I used your example:

rcov --text-coverage --no-color --spec-only bowling_spec.rb 

No file to analyze was found. All the files loaded by rcov matched one of the
following expressions, and were thus ignored:
[/\A\/usr\/lib/,
/\btc_[^.]*.rb/,
/_test\.rb\z/,
/\btest\//,
/\bvendor\//,
  
\A\/usr\/lib\/ruby\/gems\/1\.8\/gems\/rcov\-0\.8\.1\.2\.0\/lib\/rcov\/report\.rb\z/]
You can solve this by doing one or more of the following:
* rename the files not to be ignored so they don't match the above regexps
* use --include-file to give a list of patterns for files not to be ignored
* use --exclude-only to give the new list of regexps to match against
* structure your code as follows:
     test/test_*.rb  for the test cases
     lib/**/*.rb     for the target source code whose coverage you want
 making sure that the test/test_*.rb files are loading from lib/, e.g. by 
 using the -Ilib command-line argument, adding  
   $:.unshift File.join(File.dirname(__FILE__), "..", "lib")
 to test/test_*.rb, or running rcov via a Rakefile (read the RDoc
 documentation or README.rake in the source distribution).
/usr/lib/ruby/gems/1.8/gems/rcov-0.8.1.2.0/bin/rcov:525:in `instance_method':  undefined method `run' for class `Spec::Example::ExampleGroup' (NameError)
       from /usr/lib/ruby/gems/1.8/gems/rcov-0.8.1.2.0/bin/rcov:525
       from /usr/lib/ruby/gems/1.8/gems/rcov-0.8.1.2.0/bin/rcov:533:in  `instance_eval'
       from /usr/lib/ruby/gems/1.8/gems/rcov-0.8.1.2.0/bin/rcov:533
       from /usr/bin/rcov:19:in `load'
       from /usr/bin/rcov:19

My info:

$ ruby --version
ruby 1.8.6 (2007-09-24 patchlevel 111) [i686-linux]
$ gem list rspec 
*** LOCAL GEMS ***
rspec (1.1.1, 1.0.8)
$ gem list rcov
*** LOCAL GEMS ***
rcov (0.8.1.2.0, 0.8.0.2)
$ gem --version
1.0.1

Default gem path not excluded on Leopard - Nick Sieger (2008-02-06 (Wed) 10:32:39)

I'm on Leopard, where the gem path is not in the standard list of exclude paths.

Does this code seem reasonable to add right before the definition of RCov::Formatter::DEFAULT_OPTS?

   if defined?(Gem)
       Gem.path.each do |p|
           ignore_files << /\A#{Regexp.escape(p)}/
       end
   end

This has the effect of excluding all gems from the coverage report, which is desirable for me.

sub! call on Nil object - Ryan Bell (2008-03-17 (Mon) 20:35:14)

I think this block of code in rcov/report.rb might have a bug:


 _get_callsites(ref_blocks, filename, lineno, linetext, "<<") do |ref| # "
   ref.file.sub!(%r!^./!, '')
   "#{mangle_filename(ref.file||'C code')}:#{ref.line} " +
     "in #{ref.klass}##{ref.mid}"
 end

Should the sub! call look like this?:

   ref.file.sub!(%r!^./!, '') if ref.file

Bus Error - Florian Weber (2008-04-07 (Mon) 16:00:57)

I keep getting random bus errors that seem to happen in different parts of Rails:

Started .................../Users/florian/Projects/unlike/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:74: [BUG] Bus Error ruby 1.8.6 (2007-09-24) [universal-darwin9.0]

The tests pass just fine.

ruby 1.8.6 (2007-09-24 patchlevel 111) [universal-darwin9.0] rcov 0.8.1.2 2007-11-22 Rails revision 9176

Thanks!

Last modified:2007/11/19 12:50:12
Keyword(s):[blog] [ruby] [frontpage] [rcov] [release] [0.8.1]
References:[rcov: code coverage for Ruby]

Incremental search

feed-icon-28x28.png RSS 1.0 - RSS 2.0

Source code

Recent
Popular
Generated by Hiki 0.8.7 (2007-06-24).
Powered by Ruby 1.8.3 (2005-10-25).
Founded by Mauricio Fernandez.