[Home] [Blog] [Contact] - [Talks] [Workshop] [Bio] [Customers]
twitter linkedin youtube github rss

Patrick Debois

Vagrant Testing, Testing, One Two

Now that we have Vagrant up and running with our favorite Config Management, let’s see how we can integrate testing into our workflow.

Given our awesome project from my ‘Using Vagrant as a Team’ post we have the following components:

[DIR] awesome-vagrant (2)
	- [DIR] awesome-frontend
	- [DIR] awesome-datastore
	- [DIR] awesome-data
	- [DIR] awesome-chefrepo (1a)
	- [DIR] awesome-puppetrepo (1b)

What do we test?

As awesome-{frontend,datastore,data} are considered traditional software components, they would include the usual unit and integration tests from themselves. You can find ample information on the web for your favorite software component.

Cucumber and friends

Testing your configuration management is not that common yet, let’s explore our options there:

Most of the current tools are inspired by ‘cucumber’ a ‘behavior driven development’ tool. Lindsay Holmwood his great presentation at devopsdays 2009 on ‘cucumber-nagios inspired a lot of the authors to use it.

A good book on Cucumber is the rspec book and here is a great slideshare presentation on ‘Writing software not code with cucumber’ and some caveats in You’re cuking it wrong.

Alternatively there is another framework called Babushka that sets out with it’s own testing DSL. I find it refreshing to see another approach being build upon.

Puppet testing options

puppet you have ‘cucumber-puppet’ written by Nikolay Sturm a testing framework for your manifests.

Chef testing options

As chef did not implement the noop-mode, I guess it took some time to have an equivalent.

Integration testing

For our project we took another route: Instead of testing our chef recipes as standalone piece, we would test the whole of our deployed stack: the provisioned/configured system + all application and data deployed. You have to see this as complementary to your recipe/manifest tests:

  1. Testing all components together allows you to test the interaction/integration,
  2. where as if you only test the recipes itself, it would not test integration stuff like (sessions no being generated). But the advantage is that you have a better idea where things are failing when in type 1 tests.

This is very similar to the complementary fact of unit tests and bdd tests: test inside out, and outside in.

Installing cucumber

cucumber is a rubygem: this means that we now require not only the ‘vagrant’ gem needs to be installed cucumber and cuken too. Note we will include only cucumber-nagios steps and not the cuken part as they still conflict in their ssh steps.

To avoid that we need to communicate the exact version to every team member or any subsequent gem we need, we set out to create a ‘Gemfile’ that can be used by bundler. Our Gemfile would look like this

source 'http://rubygems.org'
gem 'vagrant', '0.7.2'
gem 'cuken'
gem 'cucumber'
gem 'cucumber-nagios'

I tried to include cuken (that has the chef steps) work from the latest gitrepo:

gem 'cuken', :git => "git://github.com/hedgehog/cuken.git"
gem 'ssh-forever', :git => "git://github.com/mattwynne/ssh-forever.git"

But it complains on ssh-forever not being there because that version was yanked . So no chef steps yet….

Update: 31/03/2011: It should work, and was probably a temporary fluke in my gemset

Now let’s continue the installation of our gems using bundler.

We use a global gemset with rvm to install the bundler gem for all subsequent projects. And install run bundler on our awesome-vagrant gemset

$ rvm gemset use @global
$ gem install bundler
$ bundle install
$ rvm gemset use awesome-vagrant

So now instead of doing ‘gem install’, you do:

$ bundle install

And it will install all the versions you specified in Gemspec the awesome-vagrant gemset . We add it to our git repo of the awesome-vagrant so people can add things if they need to.

You should now be able to run the cucumber command:

$ cucumber

Setting up our feature structure

In contract to using cucumber with other frameworks such as rails, we have do some work to get it working. We need to create a feature directory similar to below.

[DIR]awesome-vagrant
	- Vagrantfile
	- Gemspec
	- awesome-{frontend,datastore,date,chefrepo} git repos
	- features
		- steps
			(steps go here)
		- support
			env.rb
		- (features go here)

In env.rb you can put all the necessary requires for libraries you want to include :

require 'bundler'
begin
  Bundler.setup(:default, :development)
rescue Bundler::BundlerError => e
  $stderr.puts e.message
  $stderr.puts "Run `bundle install` to install missing gems"
  exit e.status_code
end

$LOAD_PATH.unshift(File.dirname(__FILE__) + '/../../lib')

# Disabling cuken until it gets less conflicting with other parts
# require 'cuken/ssh'
# require 'cuken/cmd'
# require 'cuken/file'
# require 'cuken/chef'

# We don't include all nagios steps only the http , but there are of-course more
# require 'cucumber/nagios/steps'
# Disable the following line if you want to use the extended ssh_steps
require 'cucumber/nagios/steps/ssh_steps'
require 'cucumber/nagios/steps/http_steps'
require 'cucumber/nagios/steps/http_header_steps'

require 'rspec/expectations'

# We use mechanize as this doesn't require us to be a rack application
require 'mechanize'
require 'webrat'

World(Webrat)
World do
  Webrat::Session.new(Webrat::MechanizeAdapter.new)
end

Using SSH to run commands

Our first feature using cucumber ssh steps

Let’s write our first feature that checks our apache. Based on the example described on the cucumber nagios blogpost

Feature: Executing commands
  In order to test a running system
  As an administrator
  I want to verify the apache behavior

Scenario: Checking if apache is running
    When I ssh to "localhost" with the following credentials: 
     | username | password  |
     | vagrant  | vagrant | 
    And I run "ps -ef |grep http|grep -v grep" 
    Then I should see "http" in the output

Now run (assuming you have apache of course)

$ cucumber

The problem with the standard cucumber-nagios steps is that it assumes to be on port 22 and vagrant has mapped our port. See the ssh_steps code for details.

Our enhanced version of the ssh steps

We decided to extend the ssh steps to add a few more rinkles to it.

  • Download our extended ssh steps file and put it into the steps directory we created earlier as filename ‘ssh_extended_steps.rb’. It extends the ssh_steps to be able specify the ssh_port, and capture stderr, stdout and the exit-code too.
  • And do the same for ‘vagrant_steps.rb’: this will make your ssh steps vagrant aware

Note: To avoid conflict with the cucumber-nagios be sure to disable the “cucumber/nagios/steps/ssh_steps” in your ’env.rb’

Feature: Executing commands
  In order to test a running system
  As an administrator
  I want to verify the apache behavior

	@apache2
	Scenario: Checking if apache is running through vagrant	
	Given I have a vagrant project in "."	
	When I ssh to vagrantbox "default" with the following credentials: 
	| username | password|
	| vagrant  | vagrant | 
	And I run "ps -ef |grep apache2|grep -v grep" 
	Then I should see "apache2" in the output
	And it should have exitcode 0
	And I should see "apache2" on stdout
	And there should be no output on stderr

The step Given I have a vagrant project, loads the vagrant environment

Given /^I have a vagrant project in "([^\"]*)"$/ do |path|
  @vagrant_env=Vagrant::Environment.new(:cwd => path)
  @vagrant_env.load!
end

And the step When I ssh to vagrantbox calculates the port it need to ssh too

unless @vagrant_env.multivm?
  port=@vagrant_env.primary_vm.ssh.port
else
  port=@vagrant_env.vms[boxname.to_sym].ssh.port
end

On a side note, you might notice the @apache2 these are tags in cucumber that you can use to specify only certain tasks. This will only run the features with tag apache

$ cucumber -tags @apache

And this is how you the step When I do a vagrant provision is implemented

And /^I do a vagrant provision$/ do 
  Vagrant::CLI.start(["provision"], :env => @vagrant_env)
end

Running component unit tests from within the machine

You can use the same mechanism to run your components tests inside the machine itself. You can your application tests mounted inside the VM and run the tests from there. We use it complementary to our ‘vagrant project’ tests. The advantage of the vagrant tests is that it does an actual network connect without working through loopback and allows you to orchestrate the VM you need to login into in a multivm setup.

Feature: Executing commands
  In order to test a running system
  As an administrator
  I want to verify the apache behavior

	@unittests
	Scenario: Checking if componentX unittests ok	
	Given I have a vagrant project in "."	
	When I ssh to vagrantbox "default" with the following credentials: 
	| username | password|
	| vagrant  | vagrant | 
	And I run "cd /opt/awesome-frontend; rails_env=test rake" 
	And it should have exitcode 0

Testing HTTP access to a vagrant box

Besides running commands on the box, we wanted to be able to check HTTP things. The two main webtesting gems in Ruby/Rails land are either webrat or the newcomer on the block Capybara . Both implement different ‘browser’ types to check your content: they have adaptors for real browsers (firefox, chrome, safari) through selenium or alike. We needed only simple http testing no DOM checking. The usual suspect is ‘rack/test’ but as we don’t have a rack application that failed miserably. We found that webrat has another option through mechanize. The gem comes installed when you install cucumber_nagios. Also the webrat websteps are implemented in http_steps of cucumber_nagios.

Update 31/03/2011: if using capybara there are two frameworks that look an alternative to leave webrat

A feature would like this

Scenario: Surf to apache
Given I go to "http://localhost:9000" 
Then I should see "It works"

Similar to our ssh problem, you see that we have to specify our port to the mapped port of vagrant. And this would also fail for virtual hosts as it would not send the correct ‘Host’ attribute to the server.

Our enhanced vagrant version adds the Give I go vagrant ‘url’ syntax

@vagrant
Scenario: Surf to apache via vagrant
Given I have a vagrant project in "."
Given I go to vagrant "http://www.sample.com" 
Then I should see "It works"

Given /^I go to vagrant "([^\"]*)"$/ do |url|
    virtual_visit(url)
end

The following snippet implements that virtual_visit:

  • it assumes @vagrant_env is loaded
  • and the correct the Host: headers accordingly to make the site virtual aware
  • it maps the url port to the port in the guest machine
  • the function is added to the webrat module so it is accessible in your steps
module Webrat #:nodoc:
    class Session #:nodoc:
		def virtual_visit(url, data=nil, options = {})
			# Options = Headers in regular visit
			uri = URI.parse(url)

			# We default to the same port
			port=uri.port

			# Now we translate url port to vagrant port
			# These mappings of ports are global and not per machine
			if @vagrant_env.nil?
			throw "No vagrant environment got loaded"
			end
			@vagrant_env.config.vm.forwarded_ports.each do |name,mapping|
			if mapping[:guestport]==uri.port
			port=mapping[:hostport]
			end
			end

			# Override the hostname to the Headers 
			header=options
			headers=options.merge({ 'Host' => uri.host+":"+port.to_s})

			# For the extended get method we need to wrap it
			# Traditional get method works 
			# => with an URL as first arg
			# => and second  = parameters (methods I guess)
			# But given some other arguments the get command behaves differently
			# See http://mechanize.rubyforge.org/mechanize/Mechanize.html for the source
			# https://github.com/brynary/webrat/blob/master/lib/webrat/adapters/mechanize.rb
			# https://github.com/brynary/webrat/blob/master/lib/webrat/core/session.rb

			# def get(options, parameters = [], referer = nil)
			@response = get({ 
			:headers => headers,
			:url => "#{uri.scheme}://localhost:#{port}#{uri.path}?#{uri.query}", 
			:verb => :get}, nil,options['Referer'])
		end
	end
end

Now we can use the standard URL and behind the scenes the URL is translated to the correct http request.

Final note:

This is pretty much work in progress, I hope to both contribute to the cuken project for the vagrant and ssh steps to make them uniformly available. Also while writing this blogpost it occurred to me that we need a vagrant-cucumber plugin that will generate the feature structure and integrate cucumber as a subcommand.

Also I’m aware that these are bad examples of BDD, as they don’t express Business talk unless your customer is a Sysadmin :)

I’ve cut off this blogpost here, I did promise you the integration in Jenkins in a CI, so that’s the next blogpost.

Hope to hear from you if you found this useful.