March 2010 Archives

Continuous deployment sounds terrifying. It means that with each commit you make to your version control, your code is pulled down, all tests are run, and, if everything passes, that code is then pushed to production. Let me repeat: that happens with every single commit.

We can all see the risks here. HHStuffedHand-lg.jpgOne slightly screwy checkin could create a bug that's not covered by unit tests. That code then goes live to your production environment, where customers see it and get angry. They then cancel their service, your company goes out of business, and you're forced to eat nothing but Hamburger Helper for the rest of your life.

I was well aware of these risks, but when I came onboard as CTO of Famigo, I was determined to give continuous deployment a try. Why? Well, I was initially intrigued by Eric Ries' writings on the subject. The more I thought about it, the more advantages I discovered.

First, continuous deployment makes deployment a non-event. Before Famigo, I worked for a great company called Grasshopper. There, we would deploy around a dozen separate projects to nearly 50 machines every 2 weeks. Even though we deployed relatively frequently, it wasn't uncommon for our deployments to last 10 hours and cause enormous stress on the team.

There were a number of reasons why deployment was such a pain point. The structure of our environment required a lot of manual intervention for deploying. Also, we included a lot of manual testing at various points within the deployment so we could rollback quickly if we spotted an issue. The great thing about continuous deployment is that it addresses both of these problems, as you have no choice but to automate everything. If a deployment fails, it fails quickly.

Second, continuous deployment forces you to build great tests. There's simply no way around this. As developers, we all know that we should be religiously writing tests for our code, but it's incredibly easy to cheat and say, "I'll catch up on my tests later," or "This test can fail for right now." The problem is that we never, ever clean that up. With continuous deployment, you absolutely can't operate in that manner. Without repeatable and thorough testing, continuous deployment is continuous nightmare.

Will you encounter production bugs that you didn't catch with a test? Of course, you will. At that point, you write a test that covers the bug, you watch the test fail, you fix the bug, you watch the test pass, and then you commit. Like magic, the fix goes out to production and you have a little more confidence in your code.

beverly_hills_cop_ii.jpgWhat if your code currently has horrendous test coverage? How do you get around the problem? Simple: start writing tests. I spent nights and weekends writing unit and integration tests for Famigo's API before I started work officially. I fully expected that time investment to pay immediate dividends once I was ready to start pushing my work for our third party mobile developers, and it has.

Third, continuous deployment encourages good source control practices. You know what would be hilarious? Another Beverly Hills Cop movie. You know what would be even more hilarious? Someone trying to continuously deploy with 50 developers all checking into the trunk all the time.

All non-idiotic source control systems have branches; use them! If you're building something great, you're going to have to make breaking changes to your code. I'm a big fan of branching for these breaking changes/big features/huge refactors, and then merging that back into the trunk once you've stabilized the code. Not only does that keep the build happy, that keeps your teammates happy because they don't have to suffer through you sending out emails to everybody saying, "I know I broke the trunk, it'll be fixed in 2 days or I'll come to work dressed as a leprechaun." So far, this branching strategy has been crucial to scaling the deployment process to larger teams.

Another good source control practice that's enforced through continuous deployment is tagging. It's trivial to script your deployment out so that it tags each release you push to production. That is very helpful in the event you need to rollback.

Fourth, and most importantly, continuous deployment is fun. You know what I love about making software? It's when someone tells me about a major problem that I can fix simply. I am inundated with warm, fuzzy feelings when I see that simple, but important change go live just a few minutes after I commit. It makes me feel like I'm making a difference.

Not only is that sort of immediacy fun and rewarding, it actually makes you more productive. Imagine you do monthly deployments. Are you actually writing new code every day of that month? Of course not. Once you get close to the release, you do a code freeze and then you test for a while. If there's a bug with something you wrote 3 weeks ago, you have to figure it all out again and then start up the testing cycle once more.

If you're deploying continuously, you shorten that loop considerably because there's so little work in progress. You don't have to code freeze, and you don't have to test for a week to ensure the build is stable. That wasted time goes away.

How do you set up continuous deployments? It's simpler than you might think. First, download a continuous integration service; I really like Hudson, which is free and easy to administer. Once you have your continuous integration service installed, create a job inside of it that polls your source control repository every few minutes for changes. When changes are found, you want to build your code (if necessary) and run your unit tests. So far, this is straight up continuous integration.

The deployment angle comes in after the tests have been run. Here, I just wrote a simple shell script that my Hudson job calls before it completes. The script zips up the current production code and moves it to an archive directory, then it copies the files over from my continuous integration environment to our production environment. We do multiple projects in a couple of seconds this way. It took me an hour or two to figure out this shell script, so allow time for experimentation here.

Continuous deployment is an ideal solution in any environment where you must me able to iterate very quickly. At our startup, that is a must. If you give it a shot with one of your simpler projects, I think you'll soon be won over, just as I've been.

If you're going to be in Austin and want to talk lean startup or mobile development, shoot me an email (cody at cody powell dot com) or ping me on Twitter! First beer's on me.

Work Clean!

| 2 Comments | 1 TrackBack

If we as software developers want to both build great products and have fun doing so, we must be able to focus on the task at hand. Thus, our code should always compile. Our unit tests should always pass. We shouldn't have to battle a ton of misleading warnings and errors as our code is executed. In short, we must hold ourselves and our teammates to a high standard.

Why is all of that important? It's because writing software, and doing a good job of it, is hard. It's difficult enough just to implement a single new feature; it becomes a truly monumental task when, in addition to implementing the new feature, you have to figure out why the code no longer compiles, the database scripts error out, and why this one unit test, written 2 years ago, causes the build server to spit out a bunch of logs in Afrikaans and then burst into flames. Everything besides that feature is a distraction that will keep you from doing a great job on the task at hand while frustrating the lederhosen off of you.

I don't think any of that was a shocker to anyone. (If it was, you have my sympathy.) What's interesting to me is that this idea of keeping all of your working materials in line all of the time actually applies to a lot of different professions. 2008_01_bourdain.jpg Check out this great example, from a seriously delightful book:

"As a cook, your station, and its condition, its state of readiness, is an extension of your nervous system - and it is profoundly upsetting if another cook or, God forbid, a waiter disturbs your precisely and carefully laid-out system. The universe is in order when your station is set up the way you like it: you know where to find everything with your eyes closed, everything you need during the course of the shift is at the ready at arm's reach, your defenses are deployed. If you let your mise-en-place run down, get dirty and disorganized, you'll quickly find yourself spinning in place and calling for backup.

I worked with a chef who used to step behind the line to a dirty cook's station in the middle of the rush to explain why the offending cook was falling behind. He'd press his palm down on the cutting board, which was littered with peppercorns, spattered sauce, bits of parsley, bread crumbs and the usual flotsam and jetsam that accumulates quickly on a station if not constantly wiped down with a moist side towel. "You see this?" he'd inquire, raising his palm so that the cook could see the bits of dirt and scraps sticking to his chef's palm. "That's what the inside of your head looks like now. Work clean!" - Anthony Bourdain, Kitchen Confidential

The great thing here is that working cleanly is just accepted in the world of cooking. You can Google for hours and I'm confident that you'd never find a thousand post thread on the culinary equivalent of StackOverflow, asking, "Should I keep my mise-en-place clean? What if I clean it right before I need something? What if I just move to a new kitchen for each meal?" They get it, just like we need to get it. (And since I'm quoting Anthony Bourdain here, I feel like he is now obligated to come to my house and make me a grilled cheese sandwich.)

I get tempted, once a month or so, to just hack something together and check it in. I know this is happening when I say a line that's as evil as non-dairy creamer: "I don't have time to do it right!" When I do check this code in, almost inevitably, something will come up shortly thereafter. A feature, quite possibly something I didn't even touch, isn't working right. I start investigating and I see, "Oh, it's this function." I dig a little deeper, "Oh wait, it's the crappy code I did last week." I go further still, "Hang on, maybe it's not this crappy code. Or maybe it is that crappy code, and I just happened to break a whole lot of stuff without realizing it." It's a painful lesson over and over again that there's no such thing as a shortcut in the world of development.

During times like that, I wish someone would press their palm onto my spaghetti code and hold it up. Then they could say, "You see this? That's what the inside of your head looks like now!" They'd be right.

About the Author

The Art of Delightful Software is written by Cody Powell. I'm currently Director of Engineering at TUNE here in Seattle. Before that, I worked on Amazon Video. Before that, I was CTO at Famigo, a venture-funded startup that helped families find and manage mobile content.

Twitter: @codypo
Github: codypo
LinkedIn: codypo's profile
Email: firstname + firstname lastname dot com