Who's Afraid of Continuous Deployment?

| 5 Comments | No TrackBacks

Continuous deployment sounds terrifying. It means that with each commit you make to your version control, your code is pulled down, all tests are run, and, if everything passes, that code is then pushed to production. Let me repeat: that happens with every single commit.

We can all see the risks here. HHStuffedHand-lg.jpgOne slightly screwy checkin could create a bug that's not covered by unit tests. That code then goes live to your production environment, where customers see it and get angry. They then cancel their service, your company goes out of business, and you're forced to eat nothing but Hamburger Helper for the rest of your life.

I was well aware of these risks, but when I came onboard as CTO of Famigo, I was determined to give continuous deployment a try. Why? Well, I was initially intrigued by Eric Ries' writings on the subject. The more I thought about it, the more advantages I discovered.

First, continuous deployment makes deployment a non-event. Before Famigo, I worked for a great company called Grasshopper. There, we would deploy around a dozen separate projects to nearly 50 machines every 2 weeks. Even though we deployed relatively frequently, it wasn't uncommon for our deployments to last 10 hours and cause enormous stress on the team.

There were a number of reasons why deployment was such a pain point. The structure of our environment required a lot of manual intervention for deploying. Also, we included a lot of manual testing at various points within the deployment so we could rollback quickly if we spotted an issue. The great thing about continuous deployment is that it addresses both of these problems, as you have no choice but to automate everything. If a deployment fails, it fails quickly.

Second, continuous deployment forces you to build great tests. There's simply no way around this. As developers, we all know that we should be religiously writing tests for our code, but it's incredibly easy to cheat and say, "I'll catch up on my tests later," or "This test can fail for right now." The problem is that we never, ever clean that up. With continuous deployment, you absolutely can't operate in that manner. Without repeatable and thorough testing, continuous deployment is continuous nightmare.

Will you encounter production bugs that you didn't catch with a test? Of course, you will. At that point, you write a test that covers the bug, you watch the test fail, you fix the bug, you watch the test pass, and then you commit. Like magic, the fix goes out to production and you have a little more confidence in your code.

beverly_hills_cop_ii.jpgWhat if your code currently has horrendous test coverage? How do you get around the problem? Simple: start writing tests. I spent nights and weekends writing unit and integration tests for Famigo's API before I started work officially. I fully expected that time investment to pay immediate dividends once I was ready to start pushing my work for our third party mobile developers, and it has.

Third, continuous deployment encourages good source control practices. You know what would be hilarious? Another Beverly Hills Cop movie. You know what would be even more hilarious? Someone trying to continuously deploy with 50 developers all checking into the trunk all the time.

All non-idiotic source control systems have branches; use them! If you're building something great, you're going to have to make breaking changes to your code. I'm a big fan of branching for these breaking changes/big features/huge refactors, and then merging that back into the trunk once you've stabilized the code. Not only does that keep the build happy, that keeps your teammates happy because they don't have to suffer through you sending out emails to everybody saying, "I know I broke the trunk, it'll be fixed in 2 days or I'll come to work dressed as a leprechaun." So far, this branching strategy has been crucial to scaling the deployment process to larger teams.

Another good source control practice that's enforced through continuous deployment is tagging. It's trivial to script your deployment out so that it tags each release you push to production. That is very helpful in the event you need to rollback.

Fourth, and most importantly, continuous deployment is fun. You know what I love about making software? It's when someone tells me about a major problem that I can fix simply. I am inundated with warm, fuzzy feelings when I see that simple, but important change go live just a few minutes after I commit. It makes me feel like I'm making a difference.

Not only is that sort of immediacy fun and rewarding, it actually makes you more productive. Imagine you do monthly deployments. Are you actually writing new code every day of that month? Of course not. Once you get close to the release, you do a code freeze and then you test for a while. If there's a bug with something you wrote 3 weeks ago, you have to figure it all out again and then start up the testing cycle once more.

If you're deploying continuously, you shorten that loop considerably because there's so little work in progress. You don't have to code freeze, and you don't have to test for a week to ensure the build is stable. That wasted time goes away.

How do you set up continuous deployments? It's simpler than you might think. First, download a continuous integration service; I really like Hudson, which is free and easy to administer. Once you have your continuous integration service installed, create a job inside of it that polls your source control repository every few minutes for changes. When changes are found, you want to build your code (if necessary) and run your unit tests. So far, this is straight up continuous integration.

The deployment angle comes in after the tests have been run. Here, I just wrote a simple shell script that my Hudson job calls before it completes. The script zips up the current production code and moves it to an archive directory, then it copies the files over from my continuous integration environment to our production environment. We do multiple projects in a couple of seconds this way. It took me an hour or two to figure out this shell script, so allow time for experimentation here.

Continuous deployment is an ideal solution in any environment where you must me able to iterate very quickly. At our startup, that is a must. If you give it a shot with one of your simpler projects, I think you'll soon be won over, just as I've been.

If you're going to be in Austin and want to talk lean startup or mobile development, shoot me an email (cody at cody powell dot com) or ping me on Twitter! First beer's on me.

No TrackBacks

TrackBack URL: http://www.codypowell.com/mt/mt-tb.cgi/11

5 Comments

What about rollbacks? Do you automate those to?

sounds good but how do you convince your boss to go along with this?

Hi. I treasured to drop you a quick note to impart my thanks. I've been observing your blog for a month or so and have picked up a heap of good information as well as enjoyed the way you've structured your site. I am attempting to run my own blog however I think its too general and I would like to focus more on smaller topics.

About the Author

The Art of Delightful Software is written by Cody Powell. I'm a dev manager at Amazon where I work on the Instant Video, Mobile Clients team. Before that, I was CTO at Famigo, a venture-funded startup that helps families find and manage mobile content.

Are you interested in building great mobile apps for Amazon Instant Video? Email me!

Twitter: @codypo
LinkedIn: codypo's profile
Personal blog: Goulash
Email: firstname + firstname lastname dot com

Categories