eigenclass logo
MAIN  Index  Search  Changes  PageRank  Login

xmp redux: expanding test assertions for profit

Update

update.png A new, improved version of the xmp filter is available.


_why just blogged about my xmp code annotator for Ruby and shot the following idea

You know what would be wild? To use this as a way of writing test cases. You write the code that needs to be tested and annotate it with the asserts.

So here is a rough implementation... the enhanced^3 xmp filter will now turn

 require 'test/unit'
 class Foo
   class StillTooSmallWheresYourManhoodDude < Exception; end
   attr_reader :b
   def initialize; @a = [] end
   def foo(x); @a << x end
   def bar; @a.join(" ") end
   def baz; @a.size end
   def foobar; raise StillTooSmallWheresYourManhoodDude unless @a.size > 6; "OK" end
 end

 class TestFoo < Test::Unit::TestCase
   def setup; @f = Foo.new end
   def test_basic_foo
     @f.foo 1
     @f.bar # =>
     @f.baz # =>
     @f.b # =>
   end

   def test_more_foo
     @f.foobar # =>
     10.times{|i| @f.foo i }
     @f.bar # =>
     @f.foobar # =>
   end
 end

into

 require 'test/unit'
 class Foo
   class StillTooSmallWheresYourManhoodDude < Exception; end
   attr_reader :b
   def initialize; @a = [] end
   def foo(x); @a << x end
   def bar; @a.join(" ") end
   def baz; @a.size end
   def foobar; raise StillTooSmallWheresYourManhoodDude unless @a.size > 6; "OK" end
 end

 class TestFoo < Test::Unit::TestCase
   def setup; @f = Foo.new end
   def test_basic_foo
     @f.foo 1
     assert_equal("1", @f.bar)
     assert_equal(1, @f.baz)
     assert_nil(@f.b)
   end

   def test_more_foo
     assert_raise(Foo::StillTooSmallWheresYourManhoodDude){@f.foobar}
     10.times{|i| @f.foo i }
     assert_equal("0 1 2 3 4 5 6 7 8 9", @f.bar)
     assert_equal("OK", @f.foobar)
   end
 end

When to use it

The xmp filter in -unittest mode is but a "smart macro" and it will be of some use only when adding tests to (mostly) functional code. I can imagine it being used if:

  • the code you want to test is already written
  • that code seems to work mostly OK

In other words, it's useless if you're going TDD. xmp3 can't anticipate your expectations about the code.

On the other hand, if these two conditions hold, the xmp filter allows you to save some time (not much) because you won't have to type the assert_equals, assert_raise, etc. manually. That's the full scope of its utility: you expand the assertions once as you're writing the tests and verify that the generated assertions are OK. If your existing implementation is mostly working, they will more often than not be just fine, and you'll save some precious keypresses. That's all.

If you change your implementation later and introduce a regression, the tests will catch it; they will not change automagically to match the new implementation. That would be pointless because they would then be guaranteed to pass, as pointed out by fatgeekuk and Eden in the comments.


I can picture a variant that generates TeSLa or Desire code. And so much more will become possible if I transcend the mere textual representation and ...

Usage

Just pipe your code into xmp3.rb as in

 xmp3.rb < mycode.rb

for normal annotation, and

 xmp3.rb -unittest < mycode.rb

to expand assertions as shown above. You can also specify which interpreter to use:

 xmp3.rb -unittest ruby19 < mycode.rb

in which case you can just use

 xmp3.rb -unittest ruby19 mycode.rb

since the script will read from ARGF.

You can find a few vim mappings in the original xmp page.

Download

xmp3.rb

#!/usr/bin/env ruby
require 'open3'

MARKER = "!XMP#{Time.new.to_i}_#{rand(1000000)}!"
XMP_RE = Regexp.new("^" + Regexp.escape(MARKER) + '\[([0-9]+)\] (=>|~>) (.*)')
VAR = "_xmp_#{Time.new.to_i}_#{rand(1000000)}"
WARNING_RE = /-:([0-9]+): warning: (.*)/

if ARGV[0] == "-unittest"
  mode = :unittest
  ARGV.shift
else
  mode = :normal
end

interpreter = ARGV.shift || "ruby"
code = ARGF.read

idx = 0
newcode = code.gsub(/^(.*) # =>.*/) do |l|
  expr = $1
  (/^\s*#/ =~ l) ? l :
  basic_eval = %!((#{VAR} = (#{expr}); $stderr.puts("#{MARKER}[#{idx+=1}] => " + #{VAR}.inspect) || #{VAR}))!
  case mode
  when :unittest
    %|begin; #{basic_eval}; rescue Exception; $stderr.puts("#{MARKER}[#{idx}] ~> " + $!.class.to_s); end|
  else
    basic_eval
  end
end
stdin, stdout, stderr = Open3::popen3(interpreter, "-w")
stdin.puts newcode
stdin.close
output = stderr.readlines

results = Hash.new{|h,k| h[k] = []}
exceptions = Hash.new{|h,k| h[k] = []}
output.grep(XMP_RE).each do |line|
  result_id, op, result = XMP_RE.match(line).captures
  case op
  when "=>"
    results[result_id.to_i] << result
  when "~>"
    exceptions[result_id.to_i] << result
  end
end

assertion = lambda do |expression, index|
  if !(wanted = results[index]).empty? || !exceptions[index]
    wanted = wanted + ["fillme"]
    case wanted
    when %w[nil fillme]
      "assert_nil(#{expression})"
    else
      "assert_equal(#{wanted[0]}, #{expression})"
    end
  else
    "assert_raise(#{exceptions[index][0]}){#{expression}}"
  end
end

idx = 0
annotated = code.gsub(/^(.*) # =>.*/) do |l|
  expr = $1
  next l if /^\s*#/ =~ l
  case mode
  when :unittest
    indent =  /^\s*/.match(l)[0]
    "#{indent}#{assertion[expr.strip, idx+=1]}"
  else
    "#{expr} # => " + (results[idx+=1] || []).join(", ")
  end
end.gsub(/ # !>.*/, '').gsub(/# (>>|~>)[^\n]*\n/m, "");

warnings = {}
output.join.grep(WARNING_RE).map do |x|
  md = WARNING_RE.match(x)
  warnings[md[1].to_i] = md[2]
end
idx = 0
annotated = annotated.map do |line|
  w = warnings[idx+=1]
  w ? (line.chomp + " # !> #{w}") : line
end
puts annotated
output.reject!{|x| /^-:[0-9]+: warning/.match(x)}
if exception = /^-:[0-9]+:.*/m.match(output.join)
  puts exception[0].map{|line| "# ~> " + line }
end
if mode == :normal && (s = stdout.read) != ""
  puts "# >> #{s.inspect}" 
end


I think I get it now - Eden (2005-11-23 (Wed) 18:15:53)

This could be thought of as a test generator for code that you have high confidence in.

So: 1) Write code and perhaps manually test a few cases. 2) Run xmp3 to generate the cases you want to test against for the future. 3) Review xmp3 generated cases and perhaps add more. After step 3, you have a full set of test cases with less manual typing required. This provides you with a battery of tests to test your code against as it grows and changes. Future implementation changes will not require modification of the test code if the expected functionality doesn't change.

Excellent. Thank you for taking the time to answer my questions.

Actually, I just realized that xmp3 can also serve as a sanity check the first time you run through it. If the asserts don't match your expectations, then you know your code needs more work.


mfp 2005-11-25 (Fri) 07:32:19

Your explanation is better than mine :)

Small explanation - mfp (2005-11-23 (Wed) 16:21:39)

fatgeekuk, Eden: I've tried to explain the scope of xmp3 and how it could be useful. You probably didn't get it because it's much more humble than you thought :-)

Another person who doesn't get it - Eden (2005-11-23 (Wed) 11:59:20)

I am still new to Ruby, but I also do not understand the point.

My understanding of unit testing is that I sit down and write test cases that verify that a function or class works the way I expect. I feed it known values and compare the result to the answer I know is correct. I do not need to know the implementation to test.

This code seems to look at the implementation and generate a test that confirms the implementation works as implemented not as expected. But unless the underlying libraries are broken, testing the implementation against itself will always be "true". I think this is what fatgeekuk meant by system in the loop.

Example: def add_two(a,b); a+b+1 end

My understanding of your code is that the system would generate: assert_equal(3, add_two(1,1)) Which is of course true, but is not the expected result in the programmers mind (add_two(1,1) should equal 2). So the test case verified very little for us.

Is this a correct interpretation of the situation? Perhaps my basic understanding of Ruby is keeping me from getting the deeper usefulness of this.

BTW - This is not an attack on the xmp code. I think it is amazing and your post has expanded my knowledge of Ruby. Even if it turns out not to be useful for testing, it is valuable for many other uses.


mfp 2005-11-23 (Wed) 16:31:42

In your example, the filter would generate assert_equal(3, add_two(1,1)) which I would correct right away; that's the end of the story, since the filter won't touch that assertion again: it will stay the way I left it. So what's the point then? It turns out that we sometimes add tests to code sections that seem to work OK right now: in that case, the generated assertion will be often be correct, and we'll have saved (a few) keypresses. It's as simple (and humble) as that. Just a way to save some typing in some situations.


mfp 2005-11-23 (Wed) 16:35:08

Oh, and BTW xmp3.rb is ugly (you can compare it to some vimscript :-), so don't try to draw much Rubyness out of that, but if you can still manage to get something good out of it, great for you :-)))

Am I missing the point? - fatgeekuk (2005-11-23 (Wed) 09:55:14)

I thought the whole idea of unit tests is that some human who thinks he/she understands the requirements for the system generates some sanity checks to validate that the code is producing what a human would expect. In this system the "loop" is open and is bridged by a human mind.

Using an automated process to produce assertions from the code under test will find nothing unless the generated assertions as examined for validity.

Surely, any test case assertions generated by an automated process from the code will pass.

Or am I missing something obvious?

There's is no loop, no automated process - mfp (2005-11-23 (Wed) 11:11:53)

There's no "automated process" as such: you perform the expansion only once (typically inside the very text editor you're using the write the unit tests) and then review the results. That's why I said it is just an "intelligent macro", nothing ground-shaking. Also, it is only useful if you have written the code you want to test before; there's no gain if you're doing TDD.

For me, the primary usage of the xmp filter was and remains annotating the examples in my ruby-talk postings... I'm also beginning to use it for the sort of things most people use irb for (like verifying that some core method works the way I think, or that some snippet does what I want).