rcov: code coverage for Ruby

Overview

rcov is a code coverage tool for Ruby. It is commonly used for viewing overall test coverage of target code. It features:

• fast execution: 20-300 times faster than previous tools
• multiple analysis modes: standard, bogo-profile, "intentional testing", dependency analysis...
• detection of uncovered code introduced since the last run ("differential code coverage")
• fairly accurate coverage information through code linkage inference using simple heuristics
• cross-referenced XHTML and several kinds of text reports
• support for easy automation with Rake and Rant
• colorblind-friendliness

rcov can be useful for testing, optimizing and understanding your Ruby programs.

• Overview
• Downloading and installing
• Binary .gem package for win32 users
• Installing
• Using RubyGems
• Installing from the tarball
• How do I use it? What does it look like?
• Summary
• Detailed code coverage reports
• Cross-referenced code coverage reports
• Decorated text output
• Text report
• Execution count textual output
• "Differential code coverage" (uncovered code detection)
• FAQs
• '"Code coverage" what??': an explanation of C0, C1 & C2 coverage analysis

Downloading and installing

The last version is rcov-0.8.1.2 See the release notes for more details. You can also use that page to report problems.

Previous releases are available at http://eigenclass.org/static/rcov. Older release info (including problem reports and solutions):

• rcov 0.8.0 (Ruby code coverage): new output modes, fix for RSpec woes, superior emacs integration
• rcov 0.7.0: code coverage and callsite/defsite data aggregation across multiple runs
• rcov 0.6.0: "differential code coverage", full (faster) cross-referenced reports, vim integration
• rcov 0.5.0: cross-referenced reports (code coverage and callsite info), RubyGems, Rant support, improved heuristics
• rcov 0.4.0: more accurate code coverage, scriptability, unrotten internals
• rcov 0.3.0: happier Rails (code coverage analysis)
• rcov 0.2.0: speedy, prettier, more convenient code coverage

Binary .gem package for win32 users

A RubyGems package for win32 is available. It was cross-compiled using mingw, and should work with the ruby-mswin32 and ruby-mingw32 builds, as well as recent One Click distributions, which are based on the former.

Installing

Using RubyGems

As usual,

gem install rcov

should do the trick. You'll be given two options: choose the mswin32 binary if you're using the One Click Installer, ruby-mswin32 or ruby-mingw32.

Installing from the tarball

The full install (which includes the rcovrt extension that makes rcov over 100 times faster) requires a working build environment and Ruby >=1.8.3. Just run

 ruby setup.rb

to get a full install.

If you cannot compile extensions for some reason, or you have an older Ruby (<1.8.3), try

 ruby setup.rb all --without-ext

or copy bin/rcov to the desired destination directory. Note that rcov will run much slower if used without the extension.

If you're using ruby-mswin32 or a recent One Click Installer for win32 distro, you can get the pre-compiled rcovrt.so. Copy it to the appropriate extension dir (something like c:\ruby\lib\ruby\site_ruby\1.8\i386-mswin32\).

How do I use it? What does it look like?

You can take a look the sample reports: normal code coverage report cross-referenced one.

rcov is similar to the testrb program shipped with Ruby: just use rcov to run your program (instead of ruby), and a number of XHTML files with the code coverage information will be generated, e.g.

rcov test/*.rb

will execute all the .rb files under test/ and generate the code coverage report under coverage/. Note that rcov deliberately ignores "uninteresting" files: the tests themselves, standard & site_ruby libraries installed on your system, etc. You can get the list of regular expressions rcov matches against with

 rcov --help

rcov can also operate in "bogo-profiling mode" and output the relevant information in alternative formats (for the time being, plain text with execution count annotations and colored text). Use

rcov -h

for more information.

Summary

The code coverage summary will look like

Detailed code coverage reports

Per-file coverage information will be shown as follows:

Cross-referenced code coverage reports

rcov will indicate where methods are being called from as follows:

Decorated text output

Text report

+-----------------------------------------------------+-------+-------+--------+
|                  File                               | Lines |  LOC  |  COV   |
+-----------------------------------------------------+-------+-------+--------+
|app/controllers/application.rb                       |    39 |    29 |  31.0% |
|app/helpers/application_helper.rb                    |   147 |   119 |  23.5% |
|app/models/aggregations/tada.rb                      |    75 |    45 |  31.1% |
|app/models/aggregations/upcoming.rb                  |    78 |    48 |  29.2% |
|app/models/article.rb                                |   109 |    78 |  67.9% |
|app/models/category.rb                               |    30 |    23 |  60.9% |
|app/models/sidebar.rb                                |    36 |    27 |  40.7% |
|components/plugins/sidebars/archives_controller.rb   |    35 |    27 |  29.6% |
|components/plugins/sidebars/category_controller.rb   |    20 |    16 |  50.0% |
|components/plugins/sidebars/delicious_controller.rb  |    20 |    16 |  50.0% |
|components/plugins/sidebars/flickr_controller.rb     |    20 |    16 |  50.0% |
|components/plugins/sidebars/fortythree_controller.rb |    20 |    16 |  50.0% |
|...s/plugins/sidebars/fortythreeplaces_controller.rb |    20 |    16 |  50.0% |
|components/plugins/sidebars/static_controller.rb     |    27 |    24 |  29.2% |
|components/plugins/sidebars/tada_controller.rb       |    20 |    16 |  50.0% |
|components/plugins/sidebars/technorati_controller.rb |    20 |    16 |  50.0% |
|components/plugins/sidebars/upcoming_controller.rb   |    20 |    16 |  50.0% |
|components/plugins/sidebars/xml_controller.rb        |    16 |    13 |  53.8% |
|components/sidebars/sidebar_controller.rb            |   110 |    80 |  32.5% |
|lib/html_engine.rb                                   |    29 |    23 |  78.3% |
|lib/login_system.rb                                  |    85 |    34 |  23.5% |
|lib/migrator.rb                                      |    28 |    22 |  40.9% |
|lib/renderfix.rb                                     |    32 |    25 |  16.0% |
|lib/xmlrpc_fix.rb                                    |    13 |    12 |  25.0% |
+-----------------------------------------------------+-------+-------+--------+
|Total                                                |  1754 |  1251 |  60.1% |
+-----------------------------------------------------+-------+-------+--------+
60.1%   42 file(s)   1754 Lines   1251 LOC

(note that only files with coverage under 80% were shown in the above report, that's why you don't see 42 files there)

Execution count textual output

$ rcov --no-html --text-counts b.rb 
================================================================================
./b.rb
================================================================================
                                                                      |      2
a, b, c = (1..3).to_a                                                 |      2
10.times do                                                           |      1
  a += 1                                                              |     10
  20.times do |i|                                                     |     10
    b += i                                                            |    200
    b.times do                                                        |    200
      c += (j = (b-a).abs) > 0 ? j : 0                                | 738800
    end                                                               |      0
  end                                                                 |      0
end                                                                   |      0

"Differential code coverage" (uncovered code detection)

rcov can detect when you've added code that is not covered by your unit tests:

   $ rcov --text-coverage-diff --no-color test/*.rb
   Started
   .......................................
   Finished in 1.163085 seconds.
   
   39 tests, 415 assertions, 0 failures, 0 errors
   
================================================================================
   !!!!! Uncovered code introduced in lib/rcov.rb
       
   ### lib/rcov.rb:207
        
        def precompute_coverage(comments_run_by_default = true)
          changed = false
          lastidx = lines.size - 1
          if (!is_code?(lastidx) || /^__END__$/ =~ @lines[-1]) && !@coverage[lastidx]
   !!       # mark the last block of comments
   !!       @coverage[lastidx] ||= :inferred
   !!       (lastidx-1).downto(0) do |i|
   !!         break if is_code?(i)
   !!         @coverage[i] ||= :inferred
   !!       end
   !!     end
          (0...lines.size).each do |i|
            next if @coverage[i]
            line = @lines[i]

FAQs

You can see the list of FAQs and add your own in this page.

'"Code coverage" what??': an explanation of C0, C1 & C2 coverage analysis

Code coverage shouldn't be abused (in few words, C0 coverage guarantees nothing) but it's still useful for testing: it will at least tell you when your tests need more work, and most importantly where.

C0 coverage analysis (as done by rcov) tells you which lines of code have been executed: you typically use it to find the areas of your program that have not been sufficiently tested, i.e. those that were not run by your test cases. C1 coverage refers to the different branches of conditional statements, for instance,

if somecondition; dofoo; else dobar end

gets 100% C0 coverage (you did "execute that line") , but only 50% C1 coverage if only one branch was followed: iow. in order to get 100% C1 coverage you'd have to change your tests so that somecondition is true in a run and false in another.

C2 (path) coverage corresponds to the amount of execution paths through your code that actually happened:

if foo == 1
  do1()
else
  do2()
end

if bar
  do3()
else
  do4()
end

The possible execution paths are:

• do1 do3
• do1 do4
• do2 do3
• do2 do4

In order to get a C2 coverage of 100%, you'd have to ensure that all those combinations (cartesian product of {foo == 1, foo != 1} and {bar, !bar}) happen.

It's possible to get very close to 100% C0 and C1 coverage. While a C0-1 coverage of 100% doesn't guarantee that your program is correct, if your coverage is much lower it certainly means that your tests aren't good enough. As for C2 coverage, it's usually not possible to make it very high: you can keep it as an indirect measure of quality, but the gain/cost ratio is lower than for C0-1 coverage, I guess.