Extending Puppet

rcrowley.org/talks/puppet-camp-eu-2011/

Hi, I’m Richard Crowley

Equal opportunity technology hater.
DevStructure’s operator and UNIX hacker.

Part 1

Custom types

(Warmup.)

How I’m running Puppet

[main]
    confdir = /home/rcrowley/work/extending-puppet
    logdir = /var/log/puppet
    rundir = /var/run/puppet
    ssldir = $vardir/ssl
    vardir = /var/lib/puppet

    pluginsync = true

How I’m running Puppet

# Master
puppet master --confdir=$HOME/work/extending-puppet

# Agent
puppet agent --confdir=$HOME/work/extending-puppet

How I’m running Puppet

github.com/rcrowley/extending-puppet

Plugins in modules

Not just “plugins.”
Any Puppet source tree.
Monkey patches, too.

Custom types

Puppet::Type.newtype :foo do

  # TODO Properties and parameters.

end

Properties

newproperty :foo do
  desc "Foo."
  newvalue :bar
  newvalue :baz
end

newproperty :quux do
  desc "Quux."
end

Properties

Things written to disk.
provider.foo called to read.
provider.foo= called to write.

Properties

No explicit, documented
guarantees on order.
Hard to work with in procedural code.

Using properties effectively

Use only one property per type.
Usually, this is the ensure property.

`ensure` property

exists?
create
destroy

`ensure` property

    newvalue(:present) do
      if @resource.provider and
         @resource.provider.respond_to?(:create)
        @resource.provider.create
      else
        @resource.create
      end
      nil # return nil so the event is autogenerated
    end

    newvalue(:absent) do
      if @resource.provider and
         @resource.provider.respond_to?(:destroy)
        @resource.provider.destroy
      else
        @resource.destroy
      end
      nil # return nil so the event is autogenerated
    end

Parameters

newparam :foo do
  desc "Foo."
  newvalue :bar
  newvalue :baz
end

newparam :quux do
  desc "Quux."
end

Parameters

Configure the resource.
No associated actions.
Use as many as you need.

Types versus providers

A package is a type.
apt is a provider of packages.
Types parse and normalize arguments.
Providers make it so.

GitHub authorized keys

`lib/puppet/type/github.rb`

require 'puppet/type'

Puppet::Type.newtype :github do
  @doc = "Send a public key to GitHub."

  newparam :path, :namevar => true do
    desc "Private key pathname."
  end

  newparam :username do
    desc "GitHub username."
  end

  newparam :token do
    desc "GitHub API token."
  end

  ensurable do
    defaultvalues
    defaultto :present
  end

end

Example resource

github { "/root/.ssh/github":
    ensure   => present,
    token    => "0123456789abcdef0123456789abcdef",
    username => "rcrowley",
}

`lib/puppet/provider/github/https.rb`

require 'base64'
require 'json'
require 'net/http'
require 'net/https'
require 'openssl'

Puppet::Type.type(:github).provide :https do
  @doc = "Send a public key to GitHub."

  defaultfor :operatingsystem => :ubuntu

  # TODO exists?, create, destroy

  # TODO private methods

end

`lib/puppet/provider/github/https.rb`

  def exists?
    return false unless File.exists?("#{@resource[:path]}")
    return false unless File.exists?("#{@resource[:path]}.pub")
    !github_id.nil?
  end

  def create
    system "ssh-keygen -q -f '#{@resource[:path]
      }' -b 2048 -N '' -C ''"
    POST("/api/v2/json/user/key/add",
      :title => File.basename("#{@resource[:path]}"),
      :key => File.read("#{@resource[:path]}.pub"))
  end

  def destroy
    if id = github_id
      POST("/api/v2/json/user/key/remove", :id => id)
    end
    File.unlink "#{@resource[:path]}"
    File.unlink "#{@resource[:path]}.pub"
  end

`lib/puppet/provider/github/https.rb`

private

  def github_id
    public_key = File.read("#{@resource[:path]}.pub").strip
    JSON.parse(
      GET("/api/v2/json/user/keys").body,
      :symbolize_names => true
    )[:public_keys].find { |gh| gh[:key] == public_key }[:id]
  rescue NoMethodError
    nil
  end

`lib/puppet/provider/github/https.rb`

private

  def GET(path)
    request = Net::HTTP::Get.new(path)
    authorize(request)
    connection.request(request)
  end

  def POST(path, options={})
    request = Net::HTTP::Post.new(path)
    authorize(request)
    request.set_form_data(options)
    connection.request(request)
  end

`lib/puppet/provider/github/https.rb`

private

  def connection
    return @connection if defined?(@connection)
    @connection = Net::HTTP.new("github.com", 443)
    @connection.use_ssl = true
    @connection.verify_mode = OpenSSL::SSL::VERIFY_NONE
    @connection
  end

  def authorize(request)
    request["Authorization"] = "Basic #{Base64.encode64(
      "#{@resource[:username]}/token:#{@resource[:token]}"
    ).gsub("\n", "")}"
  end

github.com/rcrowley/extending-puppet

Providers for
existing types

`puppet-pip`

package provider for Python’s
pip package management tool.
Included in Puppet 2.7 release candiate.

Install `puppet-pip` on 2.6

gem install puppet-pip

export \
    RUBYLIB=$GEM_HOME/puppet-pip-1.0.0/lib

`lib/puppet/provider/package/pip.rb`

require 'puppet/provider/package'
require 'xmlrpc/client'

Puppet::Type.type(:package).provide :pip,
  :parent => ::Puppet::Provider::Package do

  desc "Python packages via `pip`."

  has_feature :installable, :uninstallable,
    :upgradeable, :versionable

  # TODO self.parse, self.instances,
  # TODO query, latest, install, uninstall, update,
  # TODO lazy_pip

end

`lib/puppet/provider/package/pip.rb`

  def self.parse(line)
    if line.chomp =~ /^([^=]+)==([^=]+)$/
      {:ensure => $2, :name => $1, :provider => name}
    else
      nil
    end
  end

  def self.instances
    packages = []
    pip_cmd = which('pip') or return []
    execpipe "#{pip_cmd} freeze" do |process|
      process.collect do |line|
        next unless options = parse(line)
        packages << new(options)
      end
    end
    packages
  end

`lib/puppet/provider/package/pip.rb`

  def query
    self.class.instances.each do |provider_pip|
      if @resource[:name] == provider_pip.name
        return provider_pip.properties
      end
    end
    return nil
  end

  def latest
    client = XMLRPC::Client.new2("http://pypi.python.org/pypi")
    client.http_header_extra = {"Content-Type" => "text/xml"}
    result = client.call("package_releases", @resource[:name])
    result.first
  end

`lib/puppet/provider/package/pip.rb`

  def install
    args = %w{install -q}
    if @resource[:source]
      args << "-e"
      if String === @resource[:ensure]
        args << "#{@resource[:source]}@#{@resource[:ensure]}" +
          "#egg=#{@resource[:name]}"
      else
        args << "#{@resource[:source]}#egg=#{@resource[:name]}"
      end
    else
      case @resource[:ensure]
      when String
        args << "#{@resource[:name]}==#{@resource[:ensure]}"
      when :latest
        args << "--upgrade" << @resource[:name]
      else
        args << @resource[:name]
      end
    end
    lazy_pip *args
  end

`lib/puppet/provider/package/pip.rb`

  def uninstall
    lazy_pip "uninstall", "-y", "-q", @resource[:name]
  end

`lib/puppet/provider/package/pip.rb`

  def update
    install
  end

`lib/puppet/provider/package/pip.rb`

  private
  def lazy_pip(*args)
    pip *args
  rescue NoMethodError => e
    if pathname = which('pip')
      self.class.commands :pip => pathname
      pip *args
    else
      raise e
    end
  end

Example resource

package { "django":
    ensure   => "1.3",
    provider => pip,
}

Part 2

Custom functions

(Because the Puppet master
needs more things to do.)

GitHub authorized keys

Redux

Don’t share GitHub API token with agents.

Example resource

file { "/root/.ssh/github":
    content => github(
        "my sweet public key",
        "rcrowley",
        "0123456789abcdef0123456789abcdef"
    ),
    ensure => file,
    group => "root",
    mode => 0600,
    owner => "root",
}

`lib/puppet/parser/ functions/github.rb`

require 'base64'
require 'net/https'
require 'sshkey'

Puppet::Parser::Functions.newfunction :github,
  :type => :rvalue do |args|

  # TODO Check GitHub for an existing public key.

  key = SSHKey.generate
  request = Net::HTTP::Post.new("/api/v2/json/user/key/add")
  request["Authorization"] = "Basic #{Base64.encode64(
    "#{args[1]}/token:#{args[2]}").gsub("\n", "")}"
  request.set_form_data(
    :title => args[0], :key => key.ssh_public_key)

  connection = Net::HTTP.new("github.com", 443)
  connection.use_ssl = true
  connection.request(request)

  key.rsa_private_key
end

github.com/rcrowley/extending-puppet

Part 3

WTF is an indirector?

(It’s not even in the dictionary.)

Node terminus

$ puppet --genconfig | grep node_terminus
    # node_terminus = plain
$

From `lib/puppet/ indirector/node/plain.rb`

require 'puppet/node'
require 'puppet/indirector/plain'

class Puppet::Node::Plain < Puppet::Indirector::Plain
  # ...

  # Just return an empty node.
  def find(request)
    node = super
    node.fact_merge
    node
  end
end

From `lib/puppet/indirector/plain.rb`

require 'puppet/indirector/terminus'

# An empty terminus type, meant to just return empty objects.
class Puppet::Indirector::Plain < Puppet::Indirector::Terminus
  # Just return nothing.
  def find(request)
    indirection.model.new(request.key)
  end
end

How did we get here?

  def find(request)
    puts caller
    indirection.model.new(request.key)
  end

How did we get here?

lib/puppet/indirector/node/plain.rb:15:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
lib/puppet/indirector/catalog/compiler.rb:91:in `find_node'
lib/puppet/indirector/catalog/compiler.rb:119:in `node_from_request'
lib/puppet/indirector/catalog/compiler.rb:33:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
lib/puppet/network/http/handler.rb:106:in `do_find'
lib/puppet/network/http/handler.rb:68:in `send'
lib/puppet/network/http/handler.rb:68:in `process'
lib/puppet/network/http/webrick/rest.rb:24:in `service'
# ...

`Puppet::Indirector:: Indirection#find`

Picks and caches the terminus class.
Dispatches calls to it.

lib/puppet/indirector/node/plain.rb:15:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
lib/puppet/indirector/catalog/compiler.rb:91:in `find_node'
lib/puppet/indirector/catalog/compiler.rb:119:in `node_from_request'
lib/puppet/indirector/catalog/compiler.rb:33:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
lib/puppet/network/http/handler.rb:106:in `do_find'
lib/puppet/network/http/handler.rb:68:in `send'
lib/puppet/network/http/handler.rb:68:in `process'
lib/puppet/network/http/webrick/rest.rb:24:in `service'
# ...

What’s the node object for?

  def find(request)
    extract_facts_from_request(request)

    node = node_from_request(request)

    if catalog = compile(node)
      return catalog
    else
      # This shouldn't actually happen; we should either return
      # a config or raise an exception.
      return nil
    end
  end

A node to compile

  def find(request)
    indirection.model.new(request.key)
  end

indirection.model returns Puppet::Node.
A Puppet::Node has @parameters and @classes.
The node terminus should populate @parameters and @classes.

Catalog compilation

From `lib/puppet/parser/compiler.rb`

  def compile
    # Set the client's parameters into the top scope.
    set_node_parameters
    create_settings_scope

    evaluate_main

    evaluate_ast_node

    evaluate_node_classes

    evaluate_generators

    finish

    fail_on_unevaluated

    @catalog
  end

Catalog compilation

set_node_parameters and evaluate_node_classes handle
@parameters and @classes
set by the node terminus.
evaluate_ast_node handles
node definitions from Puppet code.

`@parameters` and `@classes`

Sound suspiciously like the makings
of the external node classifier.

Setup external node classifier in `puppet.conf`

external_nodes = /usr/local/bin/classifier
node_terminus = exec

`/usr/local/bin/classifier`

#!/bin/sh
cat <<EOF
---
classes: []
parameters:
  foo: bar
EOF

From `lib/puppet/ indirector/node/exec.rb`

  def find(request)
    output = super or return nil

    # Translate the output to ruby.
    result = translate(request.key, output)

    create_node(request.key, result)
  end

From `lib/puppet/ indirector/exec.rb`

  def find(request)
    # Run the command.
    unless output = query(request.key)
      return nil
    end

    # Translate the output to ruby.
    output
  end

How did we get here?

  def find(request)
    puts caller
    # Run the command.
    unless output = query(request.key)
      return nil
    end

    # Translate the output to ruby.
    output
  end

How did we get here?

lib/puppet/indirector/node/exec.rb:17:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
lib/puppet/indirector/catalog/compiler.rb:91:in `find_node'
lib/puppet/indirector/catalog/compiler.rb:119:in `node_from_request'
lib/puppet/indirector/catalog/compiler.rb:33:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
lib/puppet/network/http/handler.rb:106:in `do_find'
lib/puppet/network/http/handler.rb:68:in `send'
lib/puppet/network/http/handler.rb:68:in `process'
lib/puppet/network/http/webrick/rest.rb:24:in `service'
# ...

Node terminus indirected

As expected, everything’s the same except where the indirector looked for the node.
Where’s that foo parameter injected by the external node classifier?

lib/puppet/indirector/node/exec.rb:17:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
lib/puppet/indirector/catalog/compiler.rb:91:in `find_node'
lib/puppet/indirector/catalog/compiler.rb:119:in `node_from_request'
lib/puppet/indirector/catalog/compiler.rb:33:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
lib/puppet/network/http/handler.rb:106:in `do_find'
lib/puppet/network/http/handler.rb:68:in `send'
lib/puppet/network/http/handler.rb:68:in `process'
lib/puppet/network/http/webrick/rest.rb:24:in `service'
# ...

From `lib/puppet/ indirector/catalog/compiler.rb`

  def find_node(name)
    begin
      return nil unless node =
        Puppet::Node.indirection.find(name)
    rescue => detail
      puts detail.backtrace if Puppet[:trace]
      raise Puppet::Error,
        "Failed when searching for node #{name}: #{detail}"
    end


    # Add any external data to the node.
    add_node_data(node)

    node
  end

What’s in a `node`?

  def find_node(name)
    begin
      return nil unless node =
        Puppet::Node.indirection.find(name)
    rescue => detail
      puts detail.backtrace if Puppet[:trace]
      raise Puppet::Error,
        "Failed when searching for node #{name}: #{detail}"
    end


    # Add any external data to the node.
    add_node_data(node)

    puts node.inspect
    node
  end

What’s in a `node`?

#<Puppet::Node:0xb70fa674 @name="devstructure.hsd1.ca.comcast.net.",
@classes={}, @expiration=Sat Jan 08 22:47:33 +0000 2011,
@parameters={"network_eth1"=>"33.33.33.0", "netmask"=>"255.255.255.0",
"kernel"=>"Linux", "processorcount"=>"1", "swapfree"=>"998.67 MB",
"physicalprocessorcount"=>"0", "uniqueid"=>"007f0101",
"lsbmajdistrelease"=>"10", "fqdn"=>"devstructure.hsd1.ca.comcast.net.",
"operatingsystemrelease"=>"10.10", "virtual"=>"physical",
"ipaddress"=>"10.0.2.15", "memorysize"=>"346.15 MB",
"is_virtual"=>"false", :_timestamp=>Sat Jan 08 22:17:32 +0000 2011,
"clientversion"=>"2.6.4", "hardwaremodel"=>"i686",
"kernelrelease"=>"2.6.35-22-generic-pae",
"rubysitedir"=>"/usr/local/lib/site_ruby/1.8", "ps"=>"ps -ef",
"macaddress_eth0"=>"08:00:27:8e:9e:65", "domain"=>"hsd1.ca.comcast.net.",
"netmask_eth0"=>"255.255.255.0", "serverip"=>"10.0.2.15",
"servername"=>"devstructure.hsd1.ca.comcast.net.", "timezone"=>"UTC",
"uptime_days"=>"9", "macaddress_eth1"=>"08:00:27:c3:78:5a", "id"=>"root",
"netmask_eth1"=>"255.255.255.0", "processor0"=>"Intel(R) Core(TM)2 Duo CPU
T8300  @ 2.40GHz", "hardwareisa"=>"unknown", "lsbdistrelease"=>"10.10",
"selinux"=>"false", "manufacturer"=>"innotek GmbH", "interfaces"=>"eth0,eth1",
"memoryfree"=>"225.10 MB", "uptime_hours"=>"216", "foo"=>"bar",
"lsbdistdescription"=>"Ubuntu 10.10", "lsbdistcodename"=>"maverick", "kernelversion"=>"2.6.35", "path"=>"/home/vagrant/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/gems/1.8/bin", "hostname"=>"devstructure", "uptime"=>"9 days", "puppetversion"=>"2.6.4", "environment"=>"production", "serialnumber"=>"0", "macaddress"=>"08:00:27:8e:9e:65", "facterversion"=>"1.5.8", "ipaddress_eth0"=>"10.0.2.15", "kernelmajversion"=>"2.6", "swapsize"=>"1011.00 MB", "sshrsakey"=>"AAAAB3NzaC1yc2EAAAADAQABAAABAQDrJ0U+iSED6LNmnC0bHfpLKaDvWh0UXgQzdElhtY1xOS065TRMJIrb6LPaAxJH3iWXLP57rM8Vldurx+pOJWVz6ZmsGSz/EOugg6rC6K2cPAS8V5nRq6SUKksb/eBR0LbM0ygMoVEKi0ogBIkQuZNeVqdUtIcT5P7DBrasPkxzkBqxqAgYxzMeR82vYSLBEwhpeyrVUzpOHLh9mVxfCQoZRdEpnckGmMxYSOW7OBzf46CBhPrXAB5ZX3aZB+iDxNhMVvMYZFircDb/ZKZ4ph6qHFVWvBtd54N9yXf+XSLZPfox4zIRR8BRoUz8rx7ccZDZ6SWVdMCZ6jO3MuxEIeFJ", "operatingsystem"=>"Ubuntu", "serverversion"=>"2.6.4", "network_eth0"=>"10.0.2.0", "lsbdistid"=>"Ubuntu", "architecture"=>"i386", "uptime_seconds"=>"779012", "sshdsakey"=>"AAAAB3NzaC1kc3MAAACBAJNDwltq7JCbq7ql1a3g2IxLWwhJNLxk0jYj/oLLc7LvpiN1kcfuNoK2bZooUCnGkcyZs+6U3jbz2idISOncq+hfFjrMv0qIGKHXWANh1qLZuhRjBHwRlkD6ll5gw4ioTohmt3VfUbE4hJfK0z3wtQn/SLLoz4MSNnR09OIZOCgzAAAAFQDLW+bfrDl+WdsE4ixDCRr4vEW1lwAAAIBt+Bz62PhQLZSWpLaHCOOaFeoHwv9IPvQ/ors0zDOxDEoK5GHOY3BcCO8s4rHr17aQKmMP5ztNwzUBl+OlkSlQ7kAEMBRvehFbhK+SOcjvqIt6i2i9/3nn9ba0AZ8bQ+1T8Z+A/6lRmWtaeUqsZNRJitfzez7eWRfvd+X/GK1eaAAAAIAqyACyAElrdGd4YfwV/YPGjiUpIjvzqiBCogO2qvMj6/ohzdN7wBmIdIUmYQ1uYsQ6avd3GL5S2I/xJ1uGosIh6xlKa0Dk4SqOq1LGebdCpv1aUKIwecQlkxrQE6Da9Q4zVgHrtlglOW7A8uq8gfl/VQdwU1sow36scLSE8PCn/g==", "rubyversion"=>"1.8.7", "ipaddress_eth1"=>"33.33.33.33", "productname"=>"VirtualBox", "clientcert"=>"devstructure.hsd1.ca.comcast.net."}, @time=Sat Jan 08 22:17:33 +0000 2011, @environment="production">

foo is indeed bar but where’d the rest of the facts come from?

From `lib/puppet/ indirector/node/exec.rb`

  def create_node(name, result)
    node = Puppet::Node.new(name)
    set = false
    [:parameters, :classes, :environment].each do |param|
      if value = result[param]
        node.send(param.to_s + "=", value)
        set = true
      end
    end

    node.fact_merge
    node
  end

What’s in a `node`?

  def create_node(name, result)
    node = Puppet::Node.new(name)
    set = false
    [:parameters, :classes, :environment].each do |param|
      if value = result[param]
        node.send(param.to_s + "=", value)
        set = true
      end
    end

    puts node.inspect
    node.fact_merge
    node
  end

What’s in a `node`?

#<Puppet::Node:0xb71eb704
@name="devstructure.hsd1.ca.comcast.net.",
@classes={}, @parameters={"foo"=>"bar"},
@time=Sat Jan 08 22:23:12 +0000 2011>

From `lib/puppet/node.pp`

  def fact_merge
      if facts = Puppet::Node::Facts.indirection.find(name)
        merge(facts.values)
      end
  rescue => detail
      error = Puppet::Error.new(
        "Could not retrieve facts for #{name}: #{detail}")
      error.set_backtrace(detail.backtrace)
      raise error
  end

Facts terminus

$ puppet --genconfig | grep facts_terminus
    # facts_terminus = facter
$

From `lib/puppet/ indirector/facts/facter.rb`

  def find(request)
    result = Puppet::Node::Facts.new(request.key, Facter.to_hash)

    result.add_local_facts
    result.stringify
    result.downcase_if_necessary

    result
  end

What are our options?

$ ls -l lib/puppet/indirector/facts/
total 24
-rw-r--r-- 1 rcrowley rcrowley 1067 2010-12-22 01:10 active_record.rb
-rw-r--r-- 1 rcrowley rcrowley  674 2010-12-22 01:10 couch.rb
-rw-r--r-- 1 rcrowley rcrowley 2296 2010-12-22 01:10 facter.rb
-rw-r--r-- 1 rcrowley rcrowley  358 2010-12-22 01:10 memory.rb
-rw-r--r-- 1 rcrowley rcrowley  262 2010-12-22 01:10 rest.rb
-rw-r--r-- 1 rcrowley rcrowley  235 2010-12-22 01:10 yaml.rb
$

See the pattern?

lib/puppet/indirector/node/plain.rb:15:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
lib/puppet/indirector/catalog/compiler.rb:91:in `find_node'
lib/puppet/indirector/catalog/compiler.rb:119:in `node_from_request'
lib/puppet/indirector/catalog/compiler.rb:33:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
# ...

lib/puppet/indirector/node/exec.rb:17:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
lib/puppet/indirector/catalog/compiler.rb:91:in `find_node'
lib/puppet/indirector/catalog/compiler.rb:119:in `node_from_request'
lib/puppet/indirector/catalog/compiler.rb:33:in `find'
lib/puppet/indirector/indirection.rb:193:in `find'
# ...

The catalog compiler is itself an indirection we can change!

Catalog terminus

$ puppet --genconfig | grep catalog_terminus
    # catalog_terminus = compiler
$

What are our options?

$ ls -l lib/puppet/indictor/catalog/
total 24
-rw-r--r-- 1 rcrowley rcrowley 1198 2010-12-22 01:11 active_record.rb
-rw-r--r-- 1 rcrowley rcrowley 4933 2011-01-08 22:18 compiler.rb
-rw-r--r-- 1 rcrowley rcrowley  140 2010-12-22 01:10 queue.rb
-rw-r--r-- 1 rcrowley rcrowley  189 2010-12-22 01:10 rest.rb
-rw-r--r-- 1 rcrowley rcrowley  534 2010-12-22 01:10 yaml.rb
$

Blazing our own trail

DNS TXT record
node terminus

`lib/puppet/indirector/node/dns.rb`

require 'puppet/node'
require 'puppet/indirector/plain'
require 'resolv'

class Puppet::Node::Dns < Puppet::Indirector::Plain

  def find(request)
    node = super
    begin
      resolver = Resolv::DNS.new
      resource = resolver.getresource(
        request.key, Resolv::DNS::Resource::IN::TXT)
      node.classes += resource.data.split
    rescue Resolv::ResolvError
    end
    node.fact_merge
    node
  end

end

Example record

foo.example.com. IN TXT foo bar baz quux
#   certname   #        #   @classes   #

Hierarchical
facts terminus

`lib/puppet/indirector/facts/hier.rb`

require 'puppet/indirector/facts/facter'

class HierValue < Hash
  attr_accessor :top
  def initialize(top=nil)
    @top = top
  end
  def to_s
    @top
  end
end

class Puppet::Node::Facts::Hier < Puppet::Node::Facts::Facter
  def destroy(facts)
  end
  def save(facts)
  end
end

`lib/puppet/indirector/facts/hier.rb`

class Puppet::Node::Facts::Hier < Puppet::Node::Facts::Facter
  def find(request)
    hier = {}
    super.values.reject do |key, value|
      Symbol === key
    end.each do |key, value|
      value = value.split(",") if value.index(",")
      h = hier
      if key.index("_") and keys = key.split("_")
        while 1 < keys.length and key = keys.shift
          h = HierValue === h[key] ?
            h[key] : h[key] = HierValue.new(h[key])
        end
        key = keys.shift
      end
      if HierValue === h[key]
        h[key].top = value
      else
        h[key] = value
      end
    end
    Puppet::Node::Facts.new(request.key, hier)
  end
end

Example

{"kernel"=>"Linux",
 "netmask"=>{"eth0"=>"255.255.255.0", "eth1"=>"255.255.255.0"},
 "ipaddress"=>{"eth0"=>"10.0.2.15", "eth1"=>"33.33.33.33"},
 "kernelrelease"=>"2.6.35-22-generic-pae",
 "ps"=>"ps -ef",
 "network"=>{"eth0"=>"10.0.2.0", "eth1"=>"33.33.33.0"},
 "interfaces"=>["eth0", "eth1"],
 "kernelversion"=>"2.6.35",
 "puppetversion"=>"2.6.4",
 "hostname"=>"devstructure",
 "uptime"=>{"seconds"=>"858930", "days"=>"9", "hours"=>"238"}}

facts["ipaddress"] # => "10.0.2.15"
facts["ipaddress"]["eth1"] # => "33.33.33.33"

Caching catalog terminus

`lib/puppet/indirector/ catalog/caching_compiler.rb`

require 'puppet/indirector/catalog/compiler'

class Puppet::Resource::Catalog::CachingCompiler <
  Puppet::Resource::Catalog::Compiler

  @commit, @cache = `git rev-parse HEAD`.chomp, {}

  def self.cache
    commit = `git rev-parse HEAD`.chomp
    @commit, @cache = commit, {} if @commit != commit
    @cache
  end

  def find(request)
    self.class.cache[request.key] ||= super
  end

end

Limitations

In-process cache is not
shared between masters.
Invalidation brings a
thundering herd when HEAD moves.

github.com/rcrowley/extending-puppet

One more thing:
Puppet faces

New in 2.7.
Formerly a module called puppet-interfaces.

Puppet faces

require 'puppet/face'

Puppet::Face.define(:configurer, '0.0.1') do
  action(:synchronize) do
    when_invoked do |certname, options|
      facts = Puppet::Face[:facts, '0.0.1'].find(certname)
      catalog = Puppet::Face[:catalog, '0.0.1'].
        download(certname, facts)
      report = Puppet::Face[:catalog, '0.0.1'].apply(catalog)
      report
    end
  end
end

Puppet faces

github.com/puppetlabs/puppet/tree/next/lib/puppet/face

Thank you

richard@devstructure.com or @rcrowley
P.S. use DevStructure.

space, →	next slide
←	previous slide
d	debug mode
## <ret>	go to slide #
c	table of contents (vi)
f	toggle footer
r	reload slides
z	toggle help (this)

Extending Puppet

Hi, I’m Richard Crowley

Part 1

Custom types

(Warmup.)

How I’m running Puppet

How I’m running Puppet

How I’m running Puppet

Plugins in modules

Custom types

Properties

Properties

Properties

Using properties effectively

ensure property

ensure property

Parameters

Parameters

Types versus providers

GitHub authorized keys

lib/puppet/type/github.rb

Example resource

lib/puppet/provider/github/https.rb

lib/puppet/provider/github/https.rb

lib/puppet/provider/github/https.rb

lib/puppet/provider/github/https.rb

lib/puppet/provider/github/https.rb

Providers forexisting types

puppet-pip

Install puppet-pip on 2.6

lib/puppet/provider/package/pip.rb

lib/puppet/provider/package/pip.rb

lib/puppet/provider/package/pip.rb

lib/puppet/provider/package/pip.rb

lib/puppet/provider/package/pip.rb

lib/puppet/provider/package/pip.rb

lib/puppet/provider/package/pip.rb

Example resource

Part 2

Custom functions

(Because the Puppet masterneeds more things to do.)

GitHub authorized keys

Redux

Example resource

lib/puppet/parser/ functions/github.rb

Part 3

WTF is an indirector?

(It’s not even in the dictionary.)

Node terminus

From lib/puppet/ indirector/node/plain.rb

From lib/puppet/indirector/plain.rb

How did we get here?

How did we get here?

Puppet::Indirector:: Indirection#find

What’s the node object for?

A node to compile

Catalog compilation

From lib/puppet/parser/compiler.rb

Catalog compilation

@parameters and @classes

Setup external node classifier in puppet.conf

/usr/local/bin/classifier

From lib/puppet/ indirector/node/exec.rb

From lib/puppet/ indirector/exec.rb

How did we get here?

How did we get here?

Node terminus indirected

From lib/puppet/ indirector/catalog/compiler.rb

What’s in a node?

What’s in a node?

From lib/puppet/ indirector/node/exec.rb

What’s in a node?

What’s in a node?

From lib/puppet/node.pp

Facts terminus

From lib/puppet/ indirector/facts/facter.rb

What are our options?

See the pattern?

Catalog terminus

What are our options?

`ensure` property

`ensure` property

`lib/puppet/type/github.rb`

`lib/puppet/provider/github/https.rb`

`lib/puppet/provider/github/https.rb`

`lib/puppet/provider/github/https.rb`

`lib/puppet/provider/github/https.rb`

`lib/puppet/provider/github/https.rb`

Providers for
existing types

`puppet-pip`

Install `puppet-pip` on 2.6

`lib/puppet/provider/package/pip.rb`

`lib/puppet/provider/package/pip.rb`

`lib/puppet/provider/package/pip.rb`

`lib/puppet/provider/package/pip.rb`

`lib/puppet/provider/package/pip.rb`

`lib/puppet/provider/package/pip.rb`

`lib/puppet/provider/package/pip.rb`

(Because the Puppet master
needs more things to do.)

`lib/puppet/parser/ functions/github.rb`

From `lib/puppet/ indirector/node/plain.rb`

From `lib/puppet/indirector/plain.rb`

`Puppet::Indirector:: Indirection#find`

From `lib/puppet/parser/compiler.rb`

`@parameters` and `@classes`

Setup external node classifier in `puppet.conf`

`/usr/local/bin/classifier`

From `lib/puppet/ indirector/node/exec.rb`

From `lib/puppet/ indirector/exec.rb`

From `lib/puppet/ indirector/catalog/compiler.rb`

What’s in a `node`?

What’s in a `node`?

From `lib/puppet/ indirector/node/exec.rb`

What’s in a `node`?

What’s in a `node`?

From `lib/puppet/node.pp`

From `lib/puppet/ indirector/facts/facter.rb`

DNS TXT record
node terminus

`lib/puppet/indirector/node/dns.rb`

Hierarchical
facts terminus

`lib/puppet/indirector/facts/hier.rb`

`lib/puppet/indirector/facts/hier.rb`

`lib/puppet/indirector/ catalog/caching_compiler.rb`

One more thing:
Puppet faces