eigenclass logo
MAIN  Index  Search  Changes  PageRank  Login

Firebrigade: automated, sandboxed testing of RubyGems packages by other developers

Firebrigade tests every gem ever made on every platform under the sun and summarizes the results, enabling developers to see how their packages perform on systems they don't have access to.

This is very good news; it's something many of us had been waiting for*1. If you haven't yet (chances are you haven't since there are only 5 registered users and I'm the first to have submitted a build report), go see firebrigade, take a look at the reports for your software, and submit some.

This is what firebrigade says about FastRI 0.3.0: fastri-report.png

firebrigade allows end-users to test packages by running their tests in a Tinderbox (a sandboxed RubyGems installation directory) and submit the results. So far, only one out of four gems "builds" correctly...

As a developer, firebrigade allows you to see how well your software works on other platforms. For instance, thanks to firebrigade I know that rcodetools' tests need some work.

Tinderbox isn't yet able to test gems with multiple installation candidates, so there are no build results for rcov, but I modified Tinderbox locally to allow it to operate on an existent sandbox created manually*2, and found an error in the Rakefile that breaks the automated, sandboxed testing.

Submitting build results

You can use tinderbox_gem_run to test packages automatically, but I wanted to give Firebrigade::API a try, and used it to submit a report for FastRI 0.3.0 under i686-linux/Ruby 1.8.5 2006-08-25:

#>> require 'rubygems'
=> true
#>> require 'firebrigade'
=> true
#>> require 'firebrigade/api'
=> true
#>> fb = Firebrigade::API.new("firebrigade.seattlerb.org:80", "mfp", "MYPASSWORD")
=> #<Firebrigade::API:0xa7642f74 @username="mfp", @password="MYPASSWORD", 
     @url=#<URI::HTTP:0xfd3b21706 URL:http://firebrigade.seattlerb.org/api/REST/>
#>> target = fb.add_target(RUBY_VERSION, RUBY_RELEASE_DATE, RUBY_PLATFORM)
=> #<struct Firebrigade::API::Target id=7, version="1.8.5", release_date="2006-08-25", 
     platform="i686-linux", username="mfp">
#>> me = fb.get_owner("mfp")
Firebrigade::API::NotFound: No such owner exists
        from /home/batsman/usr/lib/ruby/gems/1.8/gems/firebrigade_api-1.0.0/lib/firebrigade/api.rb:136:in `check_error'
        from /home/batsman/usr/lib/ruby/gems/1.8/gems/rc-rest-2.1.0/lib/rc_rest.rb:108:in `get'
        from /home/batsman/usr/lib/ruby/gems/1.8/gems/firebrigade_api-1.0.0/lib/firebrigade/api.rb:154:in `get_owner'
        from (irb):9
#>> me = fb.add_owner("mfp")
=> #<struct Firebrigade::API::Owner id=1944, name="mfp">
#>> fastri = fb.add_project("fastri", me.id)
=> #<struct Firebrigade::API::Project id=4662, name="fastri", owner_id=1944>
#>> fastri_version = fb.add_version("0.3.0.1", fastri.id)
=> #<struct Firebrigade::API::Version id=4708, name="0.3.0.1", project_id=4662>
#>> build = fb.add_build(fastri_version.id, target.id, true, 4.980325, <<EOF)
### installing fastri-0.3.0.1 + dependencies
### fastri-0.3.0.1
### testing fastri-0.3.0.1
!!! HAS Rakefile, DOES NOT DEPEND ON RAKE!  NEEDS s.add_dependency 'rake'
### rake installed, even though you claim not to need it
### /home/batsman/usr/bin/ruby -S rake test
/home/batsman/usr/bin/ruby -Ilib "/home/batsman/usr//lib/ruby/site_ruby/1.8/rake/rake_test_loader.rb"
"test/test_ri_index.rb" "test/test_full_text_index.rb" "test/test_functional_ri_service.rb"
"test/test_full_text_indexer.rb" "test/test_name_descriptor.rb" "test/test_util.rb" 
"test/test_integration_full_text_index.rb" 
(in /home/batsman/mess/2007/05/tinderbox/sandbox/gems/fastri-0.3.0.1)
Loaded suite /home/batsman/usr//lib/ruby/site_ruby/1.8/rake/rake_test_loader
Started
.................................................................
Finished in 1.56997 seconds.

65 tests, 317 assertions, 0 failures, 0 errors
EOF
=> #<struct Firebrigade::API::Build id=3350, successful=true, duration=4.980325, 
     created_on=Wed Jan 31 16:06:57 UTC 2007, version_id=4708, target_id=7>


Great tool!! - Blackshakc (2007-01-31 (Mercredi) 10:45:43)

It's a very good idea to automatize informations about installtion of gem like this. After your post, i'd quickly go to install and run firebrigad tools!!

thanks for the information.


Woah - Danno (2007-01-31 (Wed) 12:58:04)

This is like, way awesome. Total Community Quality Intelligence.

My only question is this: I've never heard of this type of tool before, where did they get the idea?

mfp 2007-01-31 (Wed) 13:23:25

Well, the idea is not exactly new, but this doesn't make Firebrigade any less cool though :)

See for instance http://testers.cpan.org/

Eric Hodel 2007-01-31 (Wed) 23:25:37

The idea is far from new, Mozilla has a tinderbox, FreeBSD has a tinderbox (and pointyhat cluster), ...

So I built one for Ruby.


Contact details. - hgs (2007-02-01 (Thr) 04:13:39)

There seems to be no links on the firebrigade site to enable contact with the people behind it. (Or the links are sufficiently hidden for me to miss them!) Any chance you know who these people are and could let them know?

mfp 2007-02-02 (Fri) 02:35:36

The most visible contact person is Eric Hodel. There's no way to contact the people running the sandboxes, and Eric said on the ML it'll probably remain that way (private emails).


contact details. - hgs (2007-02-02 (Fri) 06:30:20)

Thanks. That's sufficient information for now.


Last modified:2007/01/31 09:57:52
Keyword(s):[blog] [ruby] [frontpage] [firebrigade] [test] [rubygems]
References:

*1 I worked on making something like this possible a couple years ago, with little success, but it's great to see it happen at last

*2 just a kludge involving ad-hoc changes in gem_runner.rb