rcov 0.8.1: compatibility with Ruby 1.8.6-p11[01], intentional testing with RSpec, etc.

Version 0.8.1 of the rcov code coverage tool for Ruby addresses the problems experienced by ruby 1.8.6-p11[01] (and in particular Leopard) users, and includes a few new features.

Update [2007-11-20] REXML workaround incorrectly applied to 1.8.6 < p110, fixed in 0.8.1.1.

Update [2007-11-22] Further bugs found in REXML (1.8.6-p11[01]), addressed in 0.8.1.2. rcov has been tested on 1.8.5 and 1.8.6-p110.

If you are new to rcov, take a look at this sample report. In addition to indicating which code has been covered by your tests, rcov allows you to navigate through your code easily. rcov records where each method was called from and can generate fully cross-referenced reports, letting you inspect the control flow. This is most useful when you’re trying to understand the overall organization of third-party code or you’re refactoring.

  “Spec-only” mode

You can indicate that only code executed inside a spec must be considered covered (this is similar to –test-unit-only).

Here’s a minimal example. Consider this target code:

# bowling.rb
class Bowling
  def hit(pins)
  end
  def score
    0
  end
  def num_throws
    1
  end
end

and the following spec:

require 'rubygems'
require 'spec'
require 'bowling'
describe Bowling do
  before(:each) do
    @bowling = Bowling.new
  end
  it "should score 0 for gutter game" do
    20.times { @bowling.hit(0) }
    @bowling.score.should == 0
  end
end
Bowling.new.num_throws

If you invoke rcov with the –spec-only option, you’ll obtain:

$ rcov --text-coverage --no-color --spec-only bowling_spec.rb 
.
Finished in 0.016369 seconds
1 example, 0 failures
 ================================================================================
bowling.rb
================================================================================
!! # bowling.rb
!! class Bowling
     def hit(pins)
     end
     def score
       0
     end
!! 
!!   def num_throws
!!     1
!!   end
!! end
================================================================================
bowling_spec.rb
================================================================================
!! require 'spec'
!! require 'bowling'
   describe Bowling do
     before(:each) do
       @bowling = Bowling.new
     end
     it "should score 0 for gutter game" do
       20.times { @bowling.hit(0) }
       @bowling.score.should == 0
     end
   end
!! 
!! Bowling.new.num_throws

Note how the num_throws method is not covered: it wasn’t executed inside the spec. This allows you to differentiate deliberate from “accidental” code coverage.

  Download

You can get it from rcov: code coverage for Ruby, or install it via RubyGems with

 gem install rcov

(if you get an older version/a 404 error, just wait for a while until the gem propagates to rubyforge’s mirrors)

  Change summary (since 0.8.0)

Features

• you can use an existent data file to generate coverage reports without having to execute your code again
• –spec-only: only consider covered code executed inside a spec (tested with RSpec 1.0.5, 1.0.8)
• –charset can be used to specify the charset in the Content-Type declaration

Bugfixes

• workaround for bugs in Safari, IE (self-closing anchors break colorization)
• workaround for bugs in REXML shipped with 1.8.6-p11[01]
• rethrow exceptions generated by traced scripts
• compatibility with Ruby < 1.8.5
• (0.8.1.1) REXML workaround incorrectly applied to 1.8.6 < p110
• (0.8.1.1) –spec-only should work with RSpec trunk and detect when it cannot work

  Bugfix releases

  0.8.1.1

• REXML workaround incorrectly applied to 1.8.6 < p110
• –spec-only should work with RSpec trunk and detect when it cannot work

0.8.1.2

• REXML from 1.8.6-p11[01] is more broken than anticipated; more invasive workarounds. Tested on 1.8.6-p110 and 1.8.5. Note that code using REXML to generate XML will behave differently under rcov in 1.8.6-p11[01] (to begin with, it won’t crash in REXML::Document#write).

  Thanks to

Lee Marlow

• patch allowing to run rcov against a data file with no input code

Kurt Stephens

• patch to rethrow any exceptions generated by the traced scripts after report generation; notably SystemExit, allowing to use the exit code from test runners under rake.

Brian Candler

• found compatibility issues with the REXML lib included in ruby-1.8.6-p110 and provided a workaround

Mat Schaffer

• reported missing line colorization on Safari and probably other browsers, owing to self-closing <a> not being handled properly despite being valid XHTML 1.0.

Sam Granieri

• tested workaround for REXML bug

Kosmas Schütz, Daniel Berger, François Beausoleil, Bil Kleb

• provided information about the ruby-1.8.6-p11[01] REXML problems

Jeremy Hinegardner

• more info about REXML’s bugs in 1.8.6-p11[01]

  Latest release – Chad Humphries (2007-11-19 (Mon) 14:39:00)

In rspec trunk class Spec::DSL::Example has changed to class Spec::Example::ExampleGroup.

Also, I get odd Rexml::Formatter errors with this release (vs the last release)

mfp 2007-11-19 (Mon) 15:16:57

Ouch, I wanted to believe that that class wouldn’t change soon 😐 I will have to check the RSpec version.

What ruby version are you using? Sam Granieri tested it on 1.8.6-p110; I have tried it with 1.8.5. Also, could you paste the errors you’re getting? Thanks.

mfp 2007-11-19 (Mon) 15:19:01

I’ve realized the workaround might fail with 1.8.6 < p110; is that what you’re using?

mfp 2007-11-20 (Tue) 03:14:30

Fixed the REXML workaround, changed –spec-only to use Spec::Example::ExampleGroup if there’s no Spec::DSL::Example. Thanks.

  rcov 0.8.1 + Ruby 1.8.6 p36 on 10.5.1 = troubles – Ruben (2007-11-19 (Mon) 17:56:44)

Using Leopard 10.5.1 with ruby 1.8.6 (2007-06-07 patchlevel 36) [universal-darwin9.0] raises:

 uninitialized constant REXML::Formatters

when running rake spec:rcov.

Editing report.rb and commenting out the 10.5.1 fix solves the problem for me.

mfp 2007-11-20 (Tue) 03:11:59

Yup, the workaround only works for 1.8.6 >= p110. Fixed in 0.8.1.1.

  I still seem to be having problems – Greg Campbell (2007-11-20 (Tue) 17:55:11)

using:

ruby 1.8.6 (2007-09-23 patchlevel 110) [x86_64-linux]

rcov 0.8.1.1 2007-11-20

I get the following error (I hope it isn’t a problem to post the whole stack trace). Any help which can be provided would be appreciated.

/usr/local/lib/ruby/1.8/rexml/document.rb:186:in `write': undefined local variable or method `transitive' for <UNDEFINED> ... </>:REXML::Document (NameError)
from (eval):93:in `pretty'
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/lib/rcov/report.rb:761:in `format_overview'
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/lib/rcov/report.rb:796:in `create_index'
from (eval):104:in `create'
from (eval):80:in `tracking_additions'
from (eval):103:in `create'
from (eval):372:in `x_'
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/lib/rcov/report.rb:796:in `create_index'
... 22 levels...
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/lib/rcov.rb:640:in `each'
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/lib/rcov.rb:640:in `dump_coverage_info'
from /usr/local/lib/ruby/gems/1.8/gems/rcov-0.8.1.1.0/bin/rcov:421
from /usr/local/lib/ruby/1.8/test/unit.rb:278

Kurt Schrader 2007-11-20 (Tue) 19:29:14

You need to go into /usr/local/lib/ruby/1.8/rexml/document.rb and change `transitive’ to ‘trans’ on line 186 in order to get it to work.

See http://www.germane-software.com/projects/rexml/ticket/115

mfp 2007-11-21 (Wed) 02:30:10

This is exactly what the workaround in lib/rcov/report.rb is meant to solve; do you see any reason why the following wouldn’t work? (This is executed after require ‘rexml/document’.)

# Try to fix bug in the REXML shipped with Ruby 1.8.6
# This affects Mac OSX 10.5.1 users and motivates endless bug reports.
if RUBY_VERSION == "1.8.6" && defined? REXML::Formatters::Transitive &&
   RUBY_RELEASE_DATE < "2007-11-04"
    class REXML::Document
        def write( output=$stdout, indent=-1, trans=false, ie_hack=false )
            if xml_decl.encoding != "UTF-8" && !output.kind_of?(Output)
                output = Output.new( output, xml_decl.encoding )
            end
            formatter = if indent > -1
                if trans
                    REXML::Formatters::Transitive.new( indent, ie_hack )
                else
                    REXML::Formatters::Pretty.new( indent, ie_hack )
                end
            else
                REXML::Formatters::Default.new( ie_hack )
            end
            formatter.write( self, output )
        end
    end
end

Jeremy Hinegardner 2007-11-21 (Wed) 21:53:50

It appears that rexml/formatters/transitive.rb has not been required by the time the lib/rcov/report.rb monkey patches REXML::Document. I did a couple of traversals through the rexml codebase and I was unable to find where, between rcov and rexml that rexml/formatters/transitive.rb was ever required. I wasn’t able to work through the nested evals at that point.

I put this at the top of lib/rcov/report.rb

 require 'rexml/formatters/transitive' 

I also applied these patches

After that, running rcov through the rspec rake task worked correctly.

mfp 2007-11-22 (Thr) 04:15:46

Thanks, I’m now requiring it explicitly and patching REXML::Formatters::Transitive at runtime. rcov 0.8.1.2 works for me on 1.8.5 and 1.8.6-p110.

Jeremy Hinegardner 2007-11-22 (Thr) 11:34:12

Yup, I agree, all is working good for me. I backed out the rspec and rexml patches linked previously, updated to rcov 0.8.1.2 and all is well. Thanks!

  rcov + Safari 3 – Mark Rowe ([email protected]) (2007-11-24 (Sat) 01:40:04)

To clarify a little on the Safari issue here: it’s not a bug in Safari.

Safari’s (really the underlying WebKit engine) handling of the code is correct given HTML 5’s error handling rules. The underlying issue here is that rcov is generating XHTML code such as <a name=”line1″/> but is being served as text/html to web browsers. If this XHTML code is served to WebKit with the correct MIME type (application/xml+xhtml) then WebKit will interpret it as XML and render it as intended. Since it is not served with the correct MIME type, browsers correctly treats it as invalid HTML and applies error handling rules to attempt to make sense of it. Currently all browsers do this in slightly different fashions but the upcoming HTML 5 specification attempts to clarify and standardise this behaviour.

The correct fix for this problem is to not generate XHTML and claim it is text/html. Either serve it as application/xml+xhtml or generate valid HTML. Ian Hickson, the driving force behind the HTML 5 specification process, has a great write-up on this issue at http://www.hixie.ch/advocacy/xhtml. They key relevant point to this rcov issue is under the subheading “Documents sent as text/html are handled as tag soup [1] by most browsers”.

Please get in touch with me if you have any questions about this.

mfp 2007-11-26 (Mon) 12:32:12

Thank for the clarification. The reference you provided was very helpful, in particular this:

You can’t trigger from the DOCTYPE since the W3C might introduce new XHTML DOCTYPEs in future, so you don’t know which DOCTYPEs to look for. (Not to mention that DOCTYPEs are optional for well-formed XHTML documents, DOCTYPE parsing is hard, DOCTYPEs may be hidden in comments, and DOCTYPE sniffing has been called harmful by many leading figures at the W3C and elsewhere.)

I will consider switching to HTML 4.01, and document the need to serve contents as application/xhtml+xml.

  Rails + RSpec – stiff (2007-11-28 (Wed) 01:00:50)

I’m trying to enable “–spec-only” in my Rails application rcov.opts, but when I add it and issue “rake spec:rcov”, rcov starts complaining that it didn’t find any files to analyze. Any ideas why is this the case?

By the way, do you think the cross referencing mechanism could be extraced from rcov and used as better ctags/rtags (working in both sides) ie. in Emacs/Vim/TextMate? I was looking for a way to add some sort of “find usage” for Ruby to Emacs for a long time…