Puppet unit testing like a pro

A big thanks to Atlassian for allowing me to post this series!!

In our previous blogpost on Puppet Versioning, we described the most basic check to see if a puppet manifest was valid. We used the parseonly function to see if it would compile.

Until know this means we have only have if the compiler is happy, not that it performs the function it needs to do. In 2009 after the first devopsdays I wrote a collection of Test Driven Infrastructure Links . This was obviously inspired by Lindsay Holmwood’s talk on cucumber-nagios.

On the Opscode chef front, Stephen Nelson-Smith wrote a great book Test-driven Infrastructure with Chef on how to do this. Also see the cuken project where re-usable cucumber steps are grouped.

Because we are using Puppet here at Atlassian, I was out to understand the current state of puppet testing. A lot can already be found at http://puppetlabs.com/blog/testing-modules-in-the-puppet-forge/

Note that I’ve purposely named this blog ‘Puppet unit testing’, as the tests I’m describing now, don’t run against an actual system. Therefore it’s hard to test the actual behavior.

In his post on Thoughts on testing puppet manifests he explains that the idea of writing tests is NOT about duplicating the code, and he identified the most common problems he was facing are:

• catalog does not compile: syntax errors, missing template files, ..
• catalog does compile, but cannot be applied: unreachable or non-existent resources, missing file resources in repo
• catalog does applies, but is faulty: faulty files, due to empty manifests variables or wrong values, missing dependencies (wrong order …), files are installed without ensuring a directory …

An important advice is:

Resource specifications can be useful for documentation purposes or refactorings. However, there is a risk of reimplementing your Puppet manifest, so be wary.

$ cd puppet-mymodule
$ gem install cucumber-puppet

Write features per module, this is the structure we are aiming at:

module
  +-- manifests
  +-- lib
  +-- features
       +-- support
       |     +-- hooks.rb
       |     +-- world.rb
       +-- catalog
       +-- feature..

Generate a cucumber-puppet world:

$ cucumber-puppet-gen world
Generating with world generator:
     [ADDED]  features/support/hooks.rb
     [ADDED]  features/support/world.rb
     [ADDED]  features/steps

# Adjust the paths to your modules and manifests
$ cat features/support/hooks.rb
Before do
  # adjust local configuration like this
  # @puppetcfg['confdir']  = File.join(File.dirname(__FILE__), '..', '..')
  # @puppetcfg['manifest'] = File.join(@puppetcfg['confdir'], 'manifests', 'site.pp')
  # @puppetcfg['modulepath']  = "/srv/puppet/modules:/srv/puppet/site-modules"

  # adjust facts like this
  @facts['architecture'] = "i386"
end

# Nothing exciting here
$ cat features/support/world.rb

require 'cucumber-puppet/puppet'
require 'cucumber-puppet/steps'

World do
  CucumberPuppet.new
end

Generating a policy feature:

$ cucumber-puppet-gen policy
Generating with policy generator:
     [ADDED]  features/catalog

# Notice the <hostname>.example.com.yaml
# These files contain the facts to test your catalog against
# 
$ cat features/catalog/policy.feature 
Feature: General policy for all catalogs
  In order to ensure applicability of a host's catalog
  As a manifest developer
  I want all catalogs to obey some general rules

  Scenario Outline: Compile and verify catalog
    Given a node specified by "features/yaml/<hostname>.example.com.yaml"
    When I compile its catalog
    Then compilation should succeed
    And all resource dependencies should resolve

    Examples:
      | hostname  |
      | localhost |

To do an actual run:

$ cucumber-puppet features/catalog/policy.feature 
Feature: General policy for all catalogs
  In order to ensure applicability of a host's catalog
  As a manifest developer
  I want all catalogs to obey some general rules

  Scenario Outline: Compile and verify catalog                            # features/catalog/policy.feature:6
    Given a node specified by "features/yaml/<hostname>.example.com.yaml" # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:1
    When I compile its catalog                                            # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:14
    Then compilation should succeed                                       # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:48
    And all resource dependencies should resolve                          # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:28

    Examples: 
      | hostname  |
      | localhost |
      Cannot find node facts features/yaml/localhost.example.com.yaml. (RuntimeError)
      features/catalog/policy.feature:7:in `Given a node specified by "features/yaml/<hostname>.example.com.yaml"'

Failing Scenarios:
cucumber features/catalog/policy.feature:6 # Scenario: Compile and verify catalog

1 scenario (1 failed)
4 steps (1 failed, 3 skipped)
0m0.006s

List of commands:

Generators for cucumber-puppet

Available generators
    feature                          Generate a cucumber feature
    policy                           Generate a catalog policy
    testcase                         Generate a test case for the test suite
    testsuite                        Generate a test suite for puppet features
    world                            Generate cucumber step and support files

General options:
    -p, --pretend                    Run, but do not make any changes.
    -f, --force                      Overwrite files that already exist.
    -s, --skip                       Skip files that already exist.
    -d, --delete                     Delete files that have previously been generated with this generator.
        --no-color                   Don't colorize the output
    -h, --help                       Show this message
        --debug                      Do not catch errors

He has also added support for testing exported-resources.

• I found it not very clear from the documentation, I better understood the basics while watching a good video tutorial on cucumber-puppet was given by Tom Sulston.
• General information on cucumber can be found in the Pragmatic Programmers’s Cucumber book
• Or at the the cucumber main site http://cukes.info/

And for a more practical explanation, see how Oliver Hookins describes the way Nokia uses cucumber-puppet

Scenario: Proxy host and port have sensible defaults
  Given a node of class "mymodule::myapp"
  And we have loaded "test" settings
  And we have unset the fact "proxy_host"
  And we have unset the fact "proxy_port"
  When I compile the catalog
  Then there should be a file "/etc/myapp/config.properties"
  And the file should contain "proxy.port=-1"
  And the file should contain /proxy\.host=$/

----

Then /^the file should contain "(.*)"$/ do |text|
  fail "File parameter 'content' was not specified" if @resource["content"].nil?
  fail "Text content [#{text}] was not found" unless @resource["content"].include?(text)
end

Then /^the file should contain \/([^\"].*)\/$/ do |regex|
  fail "File parameter 'content' was not specified" if @resource["content"].nil?
  fail "Text regex [/#{regex}/] did not match" unless @resource["content"] =~ /#{regex}/
end

module
  +-- manifests
  +-- lib
  +-- spec
       +-- spec_helper.rb
       +-- classes
       |     +-- <class_name>_spec.rb
       +-- defines
       |     +-- <define_name>_spec.rb
       +-- functions
             +-- <function_name>_spec.rb

require 'rspec-puppet'

RSpec.configure do |c|
   c.module_path = File.expand_path(File.join(File.dirname(__FILE__), '..', '..'))
   c.manifest_dir = File.expand_path(File.join(File.dirname(__FILE__), '..', '..','..','manifests'))
end


desc "Run specs check on puppet manifests"
RSpec::Core::RakeTask.new(:spec) do |t|
   t.pattern = './demo-puppet/modules/**/*_spec.rb' # don't need this, it's default
   t.verbose = true
   t.rspec_opts = "--format documentation --color"
    # Put spec opts in a file named .rspec in root
  end

require "#{File.join(File.dirname(__FILE__),'..','spec_helper')}"

describe 'apache', :type => :class do
  let (:title { 'basic' })
  let(:params) { { } }
  let(:facts) { {:operatingsystem => 'Debian', :kernel => 'Linux'} }

  it { should contain_package('httpd').with_ensure('installed') }
end

require 'puppet-lint'

desc "Run lint check on puppet manifests"
task :lint do
linter =  PuppetLint.new
  Dir.glob('./demo-puppet/modules//**/*.pp').each do |puppet_file|
    puts "Evaluating #{puppet_file}"
    linter.file = puppet_file
    linter.run
  end
  fail if linter.errors?
 end

$ rake lint