Richard Crowley’s bloghttp://rcrowley.org/index.xml2011-12-31T07:53:25ZRichard CrowleySquare
http://rcrowley.org/2012/01/03/square.html2011-12-31T07:53:25Z2011-12-31T07:53:25ZRichard Crowley
<article>
<header>
<time datetime="2012-01-03" pubdate>2012-01-03</time>
<h1>Square</h1>
</header>
<p>Today is my first day at Square. I’m joining the infrastructure team there to help automate whole datacenters, build APIs to real hardware, smooth out deployment, and probably much more along the way. I’m very excited to return to working on scalability and performance problems on a pretty highly-trafficked API. (When you spend all your time writing configuration management software, you tend not to manage any servers.)</p>
<p>Blueprint will remain an open-source configuration management tool that I maintain, newly decoupled from any business interests. In fact, I just merged the S3-backed Blueprint Server into <a href="https://github.com/devstructure/blueprint/commits/master"><code>master</code></a>.</p>
<p>It’s been about two years since Matt and I set out to build heavily automated development environments and stumbled into a new configuration management workflow. I can’t understate how much I’ve learned about both engineering and startups in that time. For now, though, I want to spend my time engineering and not selling or searching for consulting work.</p>
<p>It’s hip to be (a) square.</p>
</article>
Debian packaging for busy people — Debra and Freight
http://rcrowley.org/articles/packaging.html2011-12-31T07:53:25Z2011-12-31T07:53:25ZRichard Crowley
<article>
<header>
<time datetime="2011-01-28" pubdate>2011-01-28</time>
<h1>Debian packaging for busy people — Debra and Freight</h1>
</header>
<p>I know the stereotypes. Developers suck at packaging. We’re typecast as thinking <code>./configure && make && make install</code> is good enough or that piping <code>wget</code> into <code>sh</code> is acceptable. But there are plenty of <code>Makefile</code>s without a working <code>uninstall</code> target and there are plenty of malicious shell scripts waiting to be piped to your shell. Good packaging is a thing of beauty.</p>
<p>Of course, even the beautiful things get old. There are parts of the Debian packaging system that I don’t believe have much use anymore so the tools and workflows I’m about to advocate do not and probably will not support them.</p>
<h2><code>conffiles</code> control file</h2>
<p><code>conffiles</code> instructs the package manager to treat the files listed within specially, taking great care not to squash deviations from the packaged version. The extra care is welcome from the central Debian archive and never needed elsewhere. Sometimes I know enough about my target systems to package configuration files just like other files. All other times, I use Puppet to intelligently manage the file across my entire installation.</p>
<h2>Source packages</h2>
<p>The concept make sense for the central Debian archive because of the great distance between your average open-source project and the associated Debian maintainer. See <a href="http://www.lucas-nussbaum.net/blog/?p=617">Giving up on Ruby packaging</a> to get a sense of the problem. However, in the year 2011 when version control is not heretic but required, why endorse another way to ship source code around? Source packages have been superseded by GitHub. The argument gets even stronger when I’m both author and maintainer. Rather than having binary packages, source packages, source package source, and source source (still here?), everything is reduced to a Git repository and <code>make deb</code>.</p>
<h2>Debra builds Debian packages</h2>
<p>Debra was <a href="http://rcrowley.org/articles/debra.html">released months ago</a> so I’ll treat this as a brief review. Debra is how I build packages. It ends up calling <code>dpkg-deb</code>(1) just like you’d expect but handles the heavy lifting for you. Three commands:</p>
<pre>debra create foo control
# Put files in foo/
debra build foo foo_$VERSION_$ARCH.deb
debra destroy foo</pre>
<p><a href="http://rcrowley.github.com/debra/">Debra</a></p>
<h2>Freight builds Debian archives</h2>
<p>Freight replaces <code>reprepro</code>(1) with two shell scripts. <code>freight-add</code>(1) manages a staging area, typically in <code>/var/lib/freight</code> where it maps repositories to sets of package filenames. Packages may appear in more than one repository to, for example, distribute the same package to several Debian/Ubuntu releases. <code>freight-cache</code>(1) manages the web server’s document root, typically <code>/var/cache/freight</code>. There it creates the <code>Release</code> and <code>Packages</code> files, deals with GPG signing, and managed the various available architectures.</p>
<p>Freight uses hard links aggressively to drop old packages from the cache and use as little storage as possible. (I’m mildly obsessed with hard links.)</p>
<pre>freight add foo_0.0.0-1_all.deb apt/lucid apt/maverick
freight cache apt/lucid apt/maverick</pre>
<p>That’s all it takes to publish a Debian archive. Just BYO web server.</p>
<p><a href="http://rcrowley.github.com/freight/">Freight</a></p>
<h2>Pack it up</h2>
<p>You don’t have to go through what I’ve heard described as a “two year hazing process” to become a Debian developer and understand The Right Way to make packages. Think through the benefits they provide (if you need help, read <a href="http://morethanseven.net/2011/01/16/Why-developers-should-care-about-system-packages.html">Why Developers Should Care About System Packages</a>) then get that <code>make deb</code> target ready.</p>
</article>
Operable Ruby
http://rcrowley.org/articles/operable-ruby.html2011-12-31T07:53:25Z2011-12-31T07:53:25ZRichard Crowley
<article>
<header>
<time datetime="2011-03-28" pubdate>2011-03-28</time>
<h1>Operable Ruby</h1>
</header>
<p>Ruby today is a moving target. The reference implementation is releasing rapidly on several branches and at least three alternatives to MRI are in wide use. The RubyGems package manager is releasing nearly every week and the dizzying array of variously compatible gems are releasing whenever they feel like it. Exciting times, to be sure, but also an operator’s nightmare. At every level of this dependency tree, robust packaging brings order to the chaos. Given that, it surprises me to read folks recommending <a href="http://rvm.beginrescueend.com/">RVM</a> for production use.</p>
<p>An operable production environment relies on packages to account for most files on the system. Think of it as a paper trail: <code>dpkg-query -S <em>pathname</em></code> will tell you the package that owns a particular file. With the mapping of package names to sets of pathnames, packages become uninstallable. Naturally, anything that can be uninstalled entirely can be upgraded. Do you trust that shell script to measure up?</p>
<p>Hopefully we’ve settled that packages are not optional in production. By my philosophy, development systems should faithfully mimic production systems in every affordable way, so I extend the policy to my development virtual machines, too. Reliable development environments are just as important to me as reliable production environments.</p>
<p>There’s been an interesting <a href="https://groups.google.com/forum/#!topic/devops-toolchain/bslnhqrTT7M">discussion</a> on the <a href="https://groups.google.com/group/devops-toolchain">devops-toolchain mailing list</a> this week about how best to package the Ruby environment. Unsurprisingly, RVM came up as well as several better options. I’d like to offer executable proof that building packages from arbitrary Ruby versions is easy.</p>
<p>The remainder of this article will walk through building up a shell script that uses <a href="http://rcrowley.github.com/debra/"><code>debra</code>(1)</a> to package arbitrary versions of Ruby for Debian-based Linux distributions.</p>
<p>We’re building a system package here: the prefix is <code>/usr</code> and the files should be owned by <code>root</code>. It isn’t <code>debra</code>’s business who owns the files in the package, so we will run the entire build process as root to ensure this is true.</p>
<p>First things first, the version numbers we’re using:</p>
<pre>VERSION="1.9.2"
PATCH="180"
DPKG_BUILD_ARCH="$(dpkg --print-architecture)"</pre>
<p>Note well that <code>set -e</code> should really appear in every single shell script you write so they will exit on failure.</p>
<pre>set -e</pre>
<p>Next, use <a href="http://rcrowley.github.com/debra/debra-create.1.html"><code>debra-create</code>(1)</a> to start building up a Debian package in a temporary directory. The first argument to all <code>debra</code> commands is the directory being used for the build. This block uses one of my favorite shell scripting idioms: using <code>mktemp</code>(1) to create a temporary directory and <code>trap</code> to clean up on <code>EXIT</code> (using <a href="http://rcrowley.github.com/debra/debra-destroy.1.html"><code>debra-destroy</code>(1)</a> in this case).</p>
<pre>DESTDIR="$(mktemp -d)"
debra create "$DESTDIR"
trap "debra destroy \"$DESTDIR\"" EXIT</pre>
<p>The extensions we’re going to enable in the Ruby build require a few development libraries:</p>
<pre>apt-get -y install libssl-dev libreadline5-dev zlib1g-dev</pre>
<p>Now for the meat of the Ruby build. Using <a href="http://rcrowley.github.com/debra/debra-sourceinstall.1.html"><code>debra-sourceinstall</code>(1)</a> makes this very easy. It downloads, unpacks, and builds from the tarball given by the second argument. <code>-b</code> provides a “bootstrapping” command that is run immediately after unpacking to setup the extensions we want to enable. <code>-p</code> relays its argument as <code>--prefix</code> to the <code>./configure</code> program and <code>-f</code> provides other arbitrary arguments to <code>./configure</code>. After configuring, <a href="http://rcrowley.github.com/debra/sourceinstall.1.html"><code>sourceinstall</code>(1)</a> runs <code>make</code> and <code>make install</code> to install the package in <code>DESTDIR</code>.</p>
<pre>debra sourceinstall "$DESTDIR" \
"ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-$VERSION-p$PATCH.tar.gz" \
-b"sh -c 'echo fcntl\\\nopenssl\\\nreadline\\\nzlib >ext/Setup'" \
-p/usr -f"--program-suffix=-$VERSION"</pre>
<p>(See <a href="http://rcrowley.github.com/debra/sourceinstall.1.html"><code>sourceinstall</code>(1)</a> for more options affecting the build process.)</p>
<p>The last artifact a Debian package needs is a control file for version and dependency metadata. Other metadata like the MD5 sum of each file are handled automatically by <code>debra</code>.</p>
<pre>cat >$DESTDIR/DEBIAN/control <<EOF
Package: ruby-$VERSION
Version: $VERSION-p$PATCH
Section: devel
Priority: optional
Essential: no
Architecture: $DPKG_BUILD_ARCH
Depends: libc6, libssl0.9.8, libreadline5, zlib1g
Maintainer: Richard Crowley <r@rcrowley.org>
Description: Ruby $VERSION.
EOF</pre>
<p>With that, we can build the package in a file that follows the Debian convention of <code><em>package</em>_<em>version</em>_<em>architecture</em>.deb</code>.</p>
<pre>debra build "$DESTDIR" "ruby-${VERSION}_${VERSION}-p${PATCH}_${DPKG_BUILD_ARCH}.deb"</pre>
<p>This package is useful in development and production, can be uninstalled or upgraded with ease, coexists peacefully with the system’s Ruby plus other Rubies built this way, and isn’t gated by anyone else’s feature freezes or patch schedules. This whole example is available at <a href="https://gist.github.com/891447">https://gist.github.com/891447</a>.</p>
</article>
A blueprint is not a diff but it doesn’t matter
http://rcrowley.org/articles/blueprint-diff-analogy.html2011-12-31T07:53:25Z2011-12-31T07:53:25ZRichard Crowley
<article>
<header>
<time datetime="2011-05-07" pubdate>2011-05-07</time>
<h1>A blueprint is not a diff but it doesn’t matter</h1>
</header>
<p>The support we at <a href="http://devstructure.com/">DevStructure</a> have seen since opening the source to <a href="https://github.com/devstructure/blueprint">Blueprint</a> in February has been fantastic and addicting — thank you all. As a first introduction to how Blueprint works, I want to address an analogy I’ve seen many times: that a blueprint is a diff of your server. It’s a perfect analogy and something instantly understood by developers and operators alike.</p>
<p>The funny thing is: it isn’t a diff. Even better: it doesn’t matter.</p>
<p>Blueprint aims to describe the absolute state of a server in a concise format that is both human- and machine-readable. That’s the JSON format you see if you run <code>blueprint show <em>name</em></code> without a format argument (<code>-P</code>, <code>-C</code>, or <code>-S</code>). There are many, many things that are a part of the absolute state of a server that we can omit in favor of brevity without sacrificing correctness and we take advantage of several.</p>
<p>Building Blueprint around the notion of a diff was a non-starter because it would impose an order of operations on the configuration process. To diff between the current and some past state of a server, one could rely on file modification timestamps or explicitly invoke <code>blueprint start-paying-attention</code>. Either option requires knowing the time at which you began the configuration process. Worse still, the latter requires you to know ahead of time and install Blueprint first.</p>
<p>When faced with the prospect of figuring out what you did to configure a server, "You should have installed Blueprint before you started," is not the answer I want to give. Blueprint is just as happy given a server you’ve been tweaking since 2007 as it is on a pristine machine. This is because Blueprint deals in absolutes.</p>
<p>An image or a giant tarball certainly declare the absolute state of a system but miss the boat entirely on human readability and conciseness. To have our cake and eat it, too, we needed to rise above the abstraction of files and think about packages, the more useful building blocks of Linux systems.</p>
<h2>Reducing the noise</h2>
<p>Of course, there are hundreds of packages on any given Linux server and most of them can be assumed and therefore omitted from a blueprint. Packages like <code>coreutils</code>, <code>grep</code>, and <code>libc6</code> are essential to a Linux userland — Blueprint itself won’t even work without them. The system package managers like APT and Yum are easy to interrogate and can be asked about exactly these essential packages directly. In addition, packages like <code>ubuntu-standard</code> and all their dependencies are omitted for the same reasons.</p>
<p><code>/etc</code> is home to thousands of configuration files, the vast majority of which simply don’t matter to a blueprint. By examining file metadata and comparing the MD5 sums of file content to both the manifest of files in installed packages and a secondary list of files maintained within Blueprint, almost all of those files may be confidently omitted from a blueprint, leaving only new and modified files.</p>
<p><code>/usr/local</code> is a special directory. It is the de facto standard home of programs compiled and installed from source. Sort of. There are plenty of files and plenty of alternative package managers (like Python’s <code>easy_install</code> and <code>pip</code>) that place files in <code>/usr/local</code>. Blueprint intelligently trims away the fat and manages only the files that are truly important so you can compile from source knowing that Blueprint will faithfully package your build up for later.</p>
<h2>Sort of a diff, after all</h2>
<p>Blueprints certainly look and feel like diffs from outside and that’s to a great degree by design. The noise reduction techniques used to keep blueprints concise and understandable are very much in the spirit of diffing tools but operate outside the notion of points-in-time.</p>
<p>A blueprint is a bit like a diff between the minimal Linux installation, as best can be introspected from the installation itself, and the current, running state of the system. Create blueprints from your perfectly configured development environment and deploy with confidence.</p>
</article>
Blueprints in the new AWS CloudFormation
http://rcrowley.org/2011/10/04/aws-cloudformation.html2011-12-31T07:53:25Z2011-12-31T07:53:25ZRichard Crowley
<article>
<header>
<time datetime="2011-10-03" pubdate>2011-10-03</time>
<h1>Blueprints in the new AWS CloudFormation</h1>
</header>
<p><em>Cross-posted from the <a href="http://blog.devstructure.com/blueprints-in-the-new-aws-cloudformation">DevStructure Blog</a></em></p>
<p>Just last week, Amazon Web Services rolled out the <a href="http://aws.typepad.com/aws/2011/09/amazon-linux-ami-production-status-new-features.html">first production release of the Amazon Linux AMI</a> and with it some <a href="http://aws.typepad.com/aws/2011/09/powerful-new-features-for-aws-cloudformation.html">powerful new features for CloudFormation</a>. There’s a lot there (like updating stacks and IAM integration) but we at DevStructure are most excited about Application Bootstrapping.</p>
<p>The new Amazon Linux AMI comes with the <code>aws-cfn-bootstrap</code> package, which can unpack tarballs, place configuration files, install packages, and restart services at provision time. Sound familiar? That’s right, this new section of the CloudFormation template language uses the <a href="http://devstructure.github.com/blueprint/blueprint.5.html"><code>blueprint</code>(5)</a> file format!</p>
<p>Amazon’s PDF whitepaper, <a href="https://s3.amazonaws.com/cloudformation-examples/BoostrappingApplicationsWithAWSCloudFormation.pdf">Bootstrapping Applications via AWS CloudFormation</a>, walks through building a complete stack template by hand. To summarize, you’ll need to create an <a href="http://docs.amazonwebservices.com/AWSCloudFormation/latest/UserGuide/index.html?aws-properties-iam-user.html">IAM resource</a> and declare your EC2 instances like this:</p>
<pre>"Resources": {
"DemoInstance": {
<strong>"Metadata": {
"AWS::CloudFormation::Init": {
"config": <em>THIS IS WHERE THE BLUEPRINT GOES!</em>
}
}</strong>,
"Properties": {
"ImageId": {"Fn::FindInMap": [
"AWSRegionArch2AMI",
{"Ref": "AWS::Region"},
{"Fn::FindInMap": [
"AWSInstanceType2Arch",
{"Ref": "InstanceType"},
"Arch"
]}
]},
"InstanceType": {"Ref": "InstanceType"},
"KeyName": {"Ref": "KeyName"},
"SecurityGroups": [{"Ref": "DemoSecurityGroup"}],
<strong>"UserData" : {"Fn::Base64" : {"Fn::Join" : ["", [
"#!/bin/sh\n",
"/opt/aws/bin/cfn-init",
" -s '", {"Ref" : "AWS::StackName"}, "'",
" -r 'DemoInstance'",
" --region '", { "Ref" : "AWS::Region" }, "'",
" --access-key '", {"Ref": "DemoKey"}, "'",
" --secret-key '", {"Fn::GetAtt": ["DemoKey", "SecretAccessKey"]}, "'",
"\n",
"/opt/aws/bin/cfn-signal",
" -e $?",
" '", {"Ref" : "DemoWaitConditionHandle"}, "'",
"\n"
]]}}</strong>
},
"Type": "AWS::EC2::Instance"
}
}</pre>
<p>The user-data calls <code>cfn-init</code> with the newly-generated IAM credentials to fetch and process the metadata, and <code>cfn-signal</code> to report success or failure via a <a href="http://docs.amazonwebservices.com/AWSCloudFormation/latest/UserGuide/index.html?aws-properties-waitcondition.html">WaitCondition</a>.</p>
<p>Packages managed by Yum, Python’s <code>easy_install</code>, and RubyGems plus files and services all work natively within CloudFormation. Source tarballs will work if you upload them someplace and provide the fully-qualified URL.</p>
<p>Going the other direction, the metadata from an existing CloudFormation template can be loaded into Blueprint by copying out the <code>"config"</code> document fragment and passing it on standard input to <a href="http://devstructure.github.com/blueprint/blueprint-create.1.html"><code>blueprint-create</code>(1)</a>.</p>
<p>Our thanks to the Reto Kramer, Chris Whitaker, and Adam Thomas for making it even easier to deploy blueprints to Amazon EC2.</p>
<p>Today, we’re releasing Blueprint 3.1, which includes <a href="https://github.com/devstructure/blueprint/compare/v3.0.8...v3.1.0">a number of fixes and improvements</a> but most importantly, understands all of Amazon’s extensions to the <code>blueprint</code>(5) format, allowing seamless transition to and from AWS CloudFormation. Get 3.1 from <a href="https://github.com/devstructure/blueprint">GitHub</a>, <a href="https://github.com/devstructure/blueprint/wiki/Installing-with-a-package-manager">DevStructure’s Debian archive</a>, or from PyPI.</p>
</article>
Monads without pretension
http://rcrowley.org/2011/09/21/monads.html2011-12-31T07:53:25Z2011-12-31T07:53:25ZRichard Crowley
<article>
<header>
<time datetime="2011-09-21" pubdate>2011-09-21</time>
<h1>Monads without pretension</h1>
</header>
<p>The Strange Loop conference this week took a slight detour from its usual celebration of programming languages no one uses to really highlight Lisp in all its abstract glory. Suddenly, Scala seemed like the most mainstream topic of discussion and monads were everywhere except within the understanding of most attendees.</p>
<p>I read <a href="http://homepages.inf.ed.ac.uk/wadler/topics/monads.html#essence">The Essence of Functional Programming</a> for the second or third time on the plane and am not a professor or language author so I'd like to take a stab at explaining monads to folks accustomed to imperative programming languages.</p>
<p>This is not an exhaustive discussion of monads. This is what I consider to be the prerequisite material that no one bothered to cover in their talks. These are the three monad laws implemented and tested in Python.</p>
<script src="https://gist.github.com/1232809.js"> </script>
<p>Such contrived code has no place in a real Python application but now with monads solved your only problem is homoiconic syntax. To take monads to the next level, reimplement <code>M</code>, <code>unitM</code>, and <code>bindM</code> to encapsulate side effects (such as the primitive logging above), propagate error messages and statistics (both as demonstrated in the paper), or perhaps automatically memoize <code>k</code>.</p>
</article>
Git Paid, the programmer’s time tracker
http://rcrowley.org/2011/01/13/gitpaid.html2011-12-31T07:53:25Z2011-12-31T07:53:25ZRichard Crowley
<article>
<header>
<time datetime="2011-01-13" pubdate>2011-01-13</time>
<h1>Git Paid, the programmer’s time tracker</h1>
</header>
<p>As part of an effort to learn hands-on about DevStructure’s potential customers, I’ve started working with small companies helping with their Puppet infrastructure and general operational polish. It became quickly clear to me that tracking my time in TextMate was a losing proposition so I took a late evening to whip up Git Paid.</p>
<p>Git Paid includes three programs. <a href="http://rcrowley.github.com/gitpaid/gpbegin.1.html"><code>gpbegin</code>(1)</a> and <a href="http://rcrowley.github.com/gitpaid/gpend.1.html"><code>gpend</code>(1)</a> mark the beginning and end of a work session by making commits to a Git repository. By default, this repository is <code>~/.gitpaid</code> but it can be overridden by the <code>GITPAID_REPO</code> environment variable or the <code>-r</code> option. Likewise the default branch is <code>master</code> but it can be overridden by the <code>GITPAID_BRANCH</code> environment variable or the <code>-b</code> option.</p>
<p><a href="http://rcrowley.github.com/gitpaid/gpinvoice.1.html"><code>gpinvoice</code>(1)</a> tabulates the time spent in all work sessions and generates a nice work summary, which includes all the commit messages given to <code>gpbegin</code>(1) and <code>gpend</code>(1) with the <code>-m</code> option. It respects that not all hours are billable so the <code>-t</code> option to <code>gpend</code>(1) will cause <code>gpinvoice</code>(1) to count an arbitrary amount of time for that session. For example:</p>
<pre>
$ gpbegin -b client-name
$ ...
$ gpend -b client-name -m "Shaved the yak."
$ gpbegin -b client-name
$ gpend -b client-name -t 1:45 -m "Faked the time."
$ gpinvoice -b client-name
# Invoice
Thu Jan 6 18:27:32 UTC 2011
from client-name branch of /home/vagrant/.gitpaid
## Work log
Began: Thu, 6 Jan 2011 16:17:29 +0000
> Shaved the yak.
Ended: Thu, 6 Jan 2011 17:47:42 +0000
Billed time: 1:30
Began: Thu, 6 Jan 2011 18:27:23 +0000
> Faked the time.
Ended: Thu, 6 Jan 2011 18:27:25 +0000
Billed time (adjusted): 1:45
## Summary
Total billed time: 3:15
</pre>
<p><big><a href="https://github.com/rcrowley/gitpaid">Git Paid on GitHub</a></big></p>
</article>