Maintenance Commandments

Woah, it's been a while since I updated here. For what it's worth, I blame it all on professional upheaval and sunspots (probably a .5/.5 distribution). To dive back in, I've decided to write a little about what I'm doing right now and what I'm learning from it.

Right now, my employer runs a large sensor network. There's data coming in from all over North America in a variety of ways (satellite, analog cell, digital cell, passenger pigeon), and it's our job to decode all of these proprietary formats and make sense of the data. That sounds kind of cool, and it is. However, the volume and nature of the data impose some constraints on us. One constraint is that, if we ever have a problem with an internal machine or service, we don't have time to discover what went wrong or why it went wrong; we need to get back online in chop-chop, mush-mush fashion.

As a result of that, we have critical services spread over a bunch of machines. If something goes wrong, the main course of action is to reboot this long string of machines in very precise order. (I'll leave it your imagination to determine how well that'd work in a worst case scenario.) My job right now is to consolidate all of these services onto one or two machines, and eventually to redesign/rewrite all of them. It's not quite as fun a job as professional hamburger taster, but it pays the bills and it's giving me the opportunity to learn some interesting stuff.

Now, based on the lessons of the past few weeks, I present an ordered list of things I've learned in trying to maintain and consolidate these services!

1. It's a lot harder to be rigorous in a maintenance scenario. If you're working with a fresh codebase, how difficult is it to write unit tests and implement patterns? Hopefully, you're starting with that mindset, but even if you're not, it's a snap. However, when you inherit a giant, unseemly, Jabba-the-Hut-like heap of code, it becomes a nightmare; there's so much accumulated cruft, it's hard to know where to start. In fact, it's tempting to just add a little more to the Jabba pile yourself.

2. However, you get the better return on your effort writing tests for maintenance code. New code is less likely to be deployed and to be depended upon. As a result, if it doesn't work that well, it's not a huge loss. However, your company does depend on the skeezy maintenance code you're looking at; if they didn't, you wouldn't be looking at it.

3. Resist the urge to point fingers. No matter how crappy the code looks, keep in mind that three years from now, someone will be maintaining your latest whiz-bang design. They won't be saying, "Someone give me this dude's phone number, I must shower him with the candy of his choice!" They'll be saying, "With skills like these, how in the hell did this guy master potty-training?"

4. As weird as it looks, there may be a reason for it. If you're ever looking at a dependable old piece of code and notice 183 nested if statements, entertain the idea that they might be there for a reason. If the code works, look at it, don't rip it out.

5. Watch out for old documentation. Just because your pile of ancient crud comes with some docs doesn't mean you should read those docs. Five years ago, gas was $1.25, people listened to Creed, and your documentation reflected your app. Things might've changed since.

And I've winded myself. Perhaps more will come later.

Posted by Cody at April 3, 2007 06:25 PM