August 2013 Archives

Donut Driven Development

| No TrackBacks
We had a problem: our continuous integration environment had turned into a continuous breakage environment.  Around half the time, our I builds would fail, either due to compilation issues or failed unit tests.  Each broken build generated an email to the team, and due to their frequency and verbosity, we began to tune these emails out.  That was even worse, as it meant multiple broken builds could pile onto each other before anyone noticed.  No longer did we have to fix a single bad change, we had to unwind multiple bad changes.  And this wasn't just a build issue: each engineer would end up pulling down this broken code and break their own workspace.  It was like Voldemort himself designed this process to drive us all crazy.  

In all fairness, it was very, very complicated.  We had dozens of engineers contributing to 20 packages, with lots of explicit and transitive dependencies across these packages.  Many times, the build would work in person A's workspace and fail in CI because a dependency had changed or because they had changed some implicit behavior that another package relied upon.  It wasn't easy, in short.

Initially, several of us tried to solve these broken builds the same way: nagging emails.  I sent plenty of these on the sanctity of the build, laying out high-minded build tenets and describing what a big cost it was to the team when our builds failed.  I put all of this onto a wiki that I made all new hires read.  I was seriously considering buying a sandwich board, writing 'THE BUILD APOCALYPSE IS NIGH!  RUN YOUR UNIT TESTS!' on it, and walking up and down the halls.  None of this accomplished very much.

Then, one night, I got desperate.  After a particularly horrendous streak of failures, I sent a team email saying, "Guys, the build is all screwed up and we need to fix it immediately.  If you fix the build, I'll buy you donuts."  Yep, I resorted to bribery.  I'm sure there were other motivational tactics available, but man, I work hard already and there's a donut shop right down the street.

You know what happened next?  Someone fixed the build within the hour.  Once I saw that, I made it a new build policy: if you fix a build that you didn't actually break, you get a dozen donuts the next morning.

Now, my donut incentive plan hasn't fixed all broken builds forever, but it has led to immediate improvement.  Roughly every week, people stay late to fix a broken build specifically so they can claim their donuts.  Contrast that to the previous scenario, where people theoretically would've stayed late to fix a build in order to save team productivity.  That happened once a year, maybe.  Science shows then the donut incentive plan is 5200% more effective in preventing broken builds.

I work with talented people who are paid well; they don't need me to buy them donuts.  And yet, once that tiny, tangible incentive is there, behavior changed dramatically.  What does all of this mean?  Why did this work better than lots of emails about how important successful builds are to team productivity?     I... don't know.  Maybe it's the fact that it's a small sign of appreciation, something which can be shared with teammates.  Maybe it's the fact that the donut incentive is visible, kinda funny, and has since become baked into team culture.  Then again, maybe people just really like donuts.

There are bigger solutions here, like simplifying the build process, and we're working on all of that.  In the meantime though, our existing process need to continue to work.  If you find your team in a similar situation, look to the donut shop.

About the Author

The Art of Delightful Software is written by Cody Powell. I'm currently Director of Engineering at TUNE here in Seattle. Before that, I worked on Amazon Video. Before that, I was CTO at Famigo, a venture-funded startup that helped families find and manage mobile content.

Twitter: @codypo
Github: codypo
LinkedIn: codypo's profile
Email: firstname + firstname lastname dot com