When I woke up a few hours later, I saw roughly 38 gazillion more alerts. Based on the amount of load we were seeing, you'd think Barack Obama had interrupted all network programming to give a Famigo plug (note to Barack: you should do this). Still, I didn't actually see many users on the site, just a lot of load. Also, we're load balanced in production, and each traffic-handling node was under heavy load; it wasn't just one machine.
Maybe It's Not a Problem?
Saturday mornings are a key time for us at Famigo because we send out a report to each activated user, showing what apps their kids played along with some personalized app recommendations. When I saw the load, I immediately thought something was wrong with the email generation process. Maybe we dropped an index somewhere and the queries were suddenly taking a long time? I resolved not to worry any more about this until the emails were done.
Here's the problem: the emails wouldn't finish. They were going so slowly due to the load that we'd be sending these emails for days. I now began to worry more. I canceled our email task and began troubleshooting in earnest.
Bizarre Facts Emerge
As I SSHed into our various boxes, I noticed, via a top command, that it was the Python process serving our website and API that was consuming 100% of the CPUs. That was intriguing. I restarted all of the usual suspects (MongoDB, lighttpd, our web app), only to see things quickly begin to degrade again. Within a few minutes, Python was once again consuming 100% of the CPU.
Like most web apps, we don't really do much work that's CPU-bound. In fact, one of the laws of performance I've learned is that if you do have work that's CPU-intensive, always do it in the background. And yet, we were clearly taxing the CPU. Maybe we accidentally pushed a commit that attempts to generate pi to 1 million digits every time someone made a web request?
It seemed clear that the issue was with our code. Here's another weird thing: we hadn't changed much lately. I went through all of the commits for the past 2 days, and it was all pretty boring stuff. Just to verify, I pointed my dev instance to our production database (don't try this at home, kids) and began to actually navigate through these recent changes. Like I initially thought, there was nothing earth-shattering; it was all pretty standard web stuff.
At this point, I began to think I was hallucinating. I hadn't seen Girl Interrupted, but I imagined that Winona Ryder got committed in that movie because of the mental strain of debugging in production.
A-ha! (Or, How to Solve Problems Through Random Mouse Clicks)
I didn't know what to do. I just began clicking around our site on my dev instance, monitoring page load time in Chrome. Everything looked just fine on my dev instance at first, and then I began to notice one trivial view taking a little bit longer than it should've. The page itself should've rendered in a second or so, and it was taking closer to 1.5 seconds. The more I reloaded, the longer it took. This was particularly interesting because this view was our application view (here's an example). We can render over 30,000 apps with that view, so while the logic is very simple, it's constantly being rendered because of all the traffic.
Now, we get to the funny part. As I mentioned, we analyze tens of thousands of apps and, if you knew the right app slug, you could actually render any app, even dirty ones, with that view. While we didn't advertise this fact, you could get to stuff like famigo.com/app/super-sexy-sex-time/. Those are clearly not the apps we want families to see, even progressive European families. So, I had recently put in just a bit of work to keep sex apps from rendering. Every time someone requested an app, we'd check to make sure it was in the set of allowed, non-sex apps before we rendered anything.
It's really just one line of code that does the no-sex-app check. The no-sex-app check wasn't being done in the database; I was basically saying, in Python, 'raise 404 if app not in good_apps'. That's so simple! On my development instance, it worked fine. It didn't slow down our unit tests. However, when I wrote a quick bash script to request that URL 10 times simultaneously, things began to explode. In production, when we regularly have 25+ visitors requesting that URL at all times, everything truly burst into flames. It turns out that this particular view was CPU-bound, I just didn't know it until it encountered some scale.
Scale Drives You Mental
I think this is a fascinating bug. It's very simple logic that would work fine if we had 3 or 5 people on the site at any moment. With 10x that traffic, it was catastrophic to the rest of the platform. Imagine how crazy this gets at 100x or 10,000x that traffic. That's what makes scalability fun: gigantic issues at scale often come from very innocuous code. At least for me, the root cause is never what I expect.
How do we prevent this from happening in the future? I'm not entirely sure. It's not really something that fits into a unit test or an integration test. As a stopgap solution, we wrote a Python decorator that wraps all of our views and logs how long they took to render. Based on that, we can calculate how long a view should take to render, and alert ourselves if the render time is outside a reasonable span of time. It's not perfect, but it's a start. Anybody have a better idea?
]]>A Harder Problem
At Famigo, we do many, many distinct, complex queries when it comes to recommending apps for families (eg, give me the top 1000 puzzle games for young adults that are free on the Amazon App Store, sorted by user rating). Doing all these queries on demand proved to be a little bit slow (average query time is about a second), so I decided to cache the results of each distinct query for 8 hours. That's slightly more complex, but it wasn't like I was writing an Erlang compiler in Visual Basic.
Initial Approaches
Take 1 of cache implementation: Use memcached for the cache. Ten minutes later, curse memcached for not having an ordered datatype.
Take 2 of cache implementation: Use MongoDB for the cache. Many minutes later, celebrate success (prematurely).
What did our cached query results look like in MongoDB? Each document in the cache had a cache key (eg, most-popular-puzzle-games-for-young-adults), an expiration date, an ordinal, and a reference to the application document that we wanted to render.
Warning Signs
There were already hints that I was doing it wrong. Case in point: I had to manage all of the cache expiration myself. In MongoDB, you can specify a maximum number of documents that a collection can store (which I was doing; I specified a max of 500k docs), but that's not at all the same thing as caching these results for exactly 8 hours. Speaking of which: hey 10gen, we want TTL collections!
Another sign I was doing it wrong: I had to do a lot of index tuning to make my interactions with the MongoDB cache fast. Every time I checked the cache, I had to specify the cache key, expiration date, and sort by the ordinal; for that to be fast, all of those needed to be covered by an index. While the index sped up my finds, it slowed down my inserts. I had a hell of a time finding the right balance.
Unfortunately, I'm not yet done listing the signs that I wasn't doing it right. You can't delete from a MongoDB capped collection. That's no problem if you're just collecting logs, but from time to time, we must invalidate our cache. Since I couldn't delete these documents from the cache, I had to add another column that stored an Active status, which also required an index, since we had to query by it every time.
How Did It Work? (Spoiler Alert: Not So Great.)
We ended up running in production on my MongoDB app query cache for a month or two. It was definitely faster than performing all of the complex queries in real time (~300ms instead of 1s), but there was a new delay when we had to add results to the cache (~200ms). As both app data and users scaled up by an order of magnitude, it was clear that this would just burst into flames at some point.
A New Solution Emerges!
I decided to try something new. I knew that Redis had a sorted set datatype, so I started to play with that. Rather than cache these app query results in MongoDB, I created a sorted set of app ids for each query. I let Redis handle all of the cache expiration business by setting a TTL value for each key. When I wanted to pull from the cache, I did so, then did a find in MongoDB using the $in operator with all of the app ids, then I reordered that in Python based on the app ordering in Redis. I knew it wasn't as pretty, but was it effective?
For my first test, I merely timed how long it took to add a few hundred results to my Redis-backed cache. That was regularly around 200ms; it was now down to 1 or 2ms. Impressive... but then that should be fast. I refused to be impressed until I started pulling from the cache.
Was it faster to pull the app ids from Redis, use that to pull the documents from MongoDB, then use Python to reorder everything? Actually, yes. Thus far, getting from the cache takes 1/3 of the time that it did before. Meanwhile, adding to the cache is essentially free.
How Not to Do MongoDB, or Any Other Datastore
It turns out that, technically, I was correct. I could use MongoDB as a key-value store for caching, much like I could use my Mazda 3 as an amphibious assault vehicle. In practice, neither would be optimized for those use cases.
A key part of determining your architecture is understanding the strengths and weaknesses of your technology choices. The primary strength of MongoDB is how it allows you to simplify and decouple your data modeling via document-orientation. What about Redis? Its primary strength is how it enables very fast access to a few key data structures, like sets and dictionaries. With both of those stated, it becomes clear the situations in which you can combine MongoDB and Redis to build delightful software.
]]>Think back to your favorite prison break movie, whether it's the Great Escape, Shawshank Redemption, the cleverly-titled TV show Prison Break, or the Rock (which, granted, is about breaking into prison - further proof that Nick Cage doesn't play by your rules). What was involved?
A cast of quirky characters, drawn together by a shared goal.
On a prison break team, you might have a group of guys digging the tunnel, another group laying down the track inside the tunnel, and then some more guys securing fake IDs and afro wigs for once you've escaped. There's always (ALWAYS!) a scene earlier in the movie showing how none of these guys liked each other originally. Once they settle on the idea of a prison break, they quickly become inseparable and ready to lay down their lives for each other.
Between developers, marketing, biz dev, management, and investors, the typical software organization is a collection of people who'd never gather together for any other reason. It rivals the most dysfunctional family in the world. And yet, the end goal, be it world domination, billions of dollars, or just happier users, is enough get this odd bunch of folks to put in thousands of hours in extremely stressful situations. Weekends are skipped, holidays go unobserved, and your kids come to refer to you as 'That smelly guy who drops by occasionally to swear at us and change underwear', all for the sake of the software.
A group of antagonists, out to thwart the escape.
In the prison break movie, there's always a group of guards charged with preventing the prisoners from escaping. We might get a scene showing what happens when some other group tried and failed to escape. We come to learn that these guards are some rough hombres.
Similarly, your competition is there to keep you from the rich rewards you'll gain upon your glorious software release. This competition could be a competing company in this market, or it could be another department in your organization. Regardless, an inordinate amount of energy is spent worrying about these antagonists.
Fortunately, the antagonists aren't very smart.
Remember how, in all of those movies, the prisoners empty one handful of dirt at a time into the yard, so no one realizes they're tunneling out? And how they put a dummy made out of socks in their beds each night so the guards don't notice their absence? If the guards were actually intelligent and engaged, they'd probably notice that kind of thing. They never do.
Your software group's antagonists are probably the same. You'll spend a ton of time worrying that they've figured out what you're up to, based on a few sentences on your website or a line in some PR piece that slipped out. Oh God, maybe they even signed up for a beta account! And then you'll worry that once they figure this out, they'll beat you to market and steal the money, fame, and silk snuggies that accompany both of those. This will probably never happen either.
Months and months of drudgery.
Unsurprising fact: it's a lot of work to tunnel through a building, under the prison yard, and out to safety. Especially when all you have is a spoon.
Along the same lines, it's a lot of work to write software. No matter how many practices, methodologies, and tools we use, it is ultimately just a hell of a lot of typing. Not all of this is exciting; you'd probably rather dig through a wall with a spoon than revisit your password reset logic. Nonetheless, the goal depends on this, so we type and type and type.
Followed by short bursts of brain-melting terror.
What happens when the prisoners finally escape? All hell breaks loose: somebody realizes the fake IDs never showed up, there's a police-looking-guy in front of the rally point, and the prisoners can hear the guards and their dogs a few hundred yards away.
A software release is quite similar. What should be a triumphant moment quickly turns sour, as the production environment goes down, Google de-indexes us for some reason, and we discover a bug that's led us to charging some people 100x more than we ought to.
Hooray, we escaped from prison and everything's great! Except for when it isn't.
Here's the surprising part: for all the cliches, prison break movies diverge when it comes to the conclusion. Some end happily, with everyone free. In some, no one escapes. And then some end on a mixed note, with a few prisoners escaping and the rest being captured/shot/nibbled on by guard dogs. In general, the endings aren't anything to be depressed over, because there's always a high chance of a sequel.
Software works the same way. There are endings that are happy, non-happy, and all emotions in between. If that's the case and we're not guaranteed success regardless of the sacrifices, then we should focus on the practice itself of software development. Let's have fun, treat each other excellently, and learn. Let's understand that there's an element of luck in these results, but that shouldn't detract from our satisfaction of doing a job really well and building something wonderful.
If we take over the world in the process, that's all the better. Even if we don't, we'll still be in fine position for the sequel.
Many thanks to everybody who helped me think of prison break cliches on Twitter.
]]>Our take on MongoDB: it's a great tool, but it's not the only tool. MongoDB is fast, easy to administer, and has a great API for many different use cases. When you find yourself in a situation where those 3 no longer apply, you should use a different database. Let's consider these, point by point.
MongoDB is fast
This has long been one of Mongo's selling points; if you're gullible enough to believe database benchmarks, there's proof scattered about the web. Lately, there's been much debate about how some of 10gen's design decisions could potentially kill db performance, particularly Mongo's global write lock. Under a write-heavy load (as many Mongo instances are), Mongo could become CPU-bound, which would be catastrophic for performance.
That sounds terrifying (global write lock omgwtfbbq?!?), but we have never experienced this. Under a reasonable load, you're unlikely to run into this if you follow Mongo's one guiding performance principle: your working set must fit into physical RAM. I cannot overemphasize this point. Mongo relies on memory mapped files for performance, and you'll see a gigantic degradation in response time if it must go to the hard drive to read or write.
Remember, for outstanding performance, your working set must fit into physical memory, not virtual memory. If your data is too large for that, you should either shard it so that it will fit into physical memory on multiple machines, or choose another database.
MongoDB is easy to administer
It's quite simple to configure MongoDB for replication; if you know the difference between bash and batman, you could do it in less than 30 minutes. The same goes for sharding. Compared to the amount of time I've spent configuring and freaking out over Oracle clusters of similar size, my MongoDB administration time is a rounding error's rounding error.
There are scary stories here too, particularly with rebalancing shards and the availability of all the requisite services on a large, sharded databases under heavy load. Again, I wonder if MongoDB is the right choice. If you're looking for scalability and high availability via replication, I would try Cassandra. (Random asides: Cassandra's performance actually scales linearly as you add instances, which sounds like magic. Neato graph and other stuff here.)
MongoDB has a great API for many different use cases
Considering that Mongo uses a JSON-like encoding for all its data, the query language is simply amazing (awesomeness ahoy!). Not only that, but there's built-in support for map/reduce across your collections. When it comes to standard CRUD work or ad-hoc querying (via its querying language or map/reduce), Mongo delivers nearly everywhere.
Where isn't it so great? One example is full text searching. You can technically kinda do it, but it lacks basic functionality like stemming. Given the sheer number of simple, powerful full text search engines, you should just supplement Mongo with something like Solr for searching. That's what we do.
Conclusion
Okay, so MongoDB doesn't work superbly for all problems in all deployments at all levels of load. What does?
I like it that Mongo doesn't solve all my problems. One of the great aspects of the NoSQL movement is the sheer number of amazing tools available. I love that that, in the course of building great software, I get to work with Mongo, Redis, Solr, and others. It's fun, and I learn; these are good things.
]]>When I was 17 or 18 (think late 1990s), I really liked technology and writing code, but I wasn't sure if computer science was the right major for me. I was apprehensive because I'd had an internship that wasn't always a great experience. It sure seemed like I could use some advice from an expert on the matter, but I didn't know any experts. I had a flash of insight: hey, why not email somebody incredibly accomplished and see what they thought about my situation?
To this day, I have no idea why I chose Dennis Ritchie. There were roughly 500 million other programmers in the world who would've had more time to answer my my inane questions, but I chose him. I guess I assumed that language design, O/S internals, and career planning for high schoolers were all similar sorts of expertise. I got his email from his Bell Labs page, then I typed up a note on my predicament and sent it off to him.
Surprisingly, he answered! It wasn't some perfunctory response, either, it was actually quite encouraging. Unfortunately, the email disappeared into the ether long ago, but I can paraphrase what he said.
He said he was happy to offer me fatherly, no, probably more like grandfatherly advice (I recall that part distinctly). He said he himself didn't study cs as an undergraduate, and he didn't think it had limited his options. He went on to say that the field of software advances very quickly, but as long as you understood the fundamentals, you could do a lot of great work.
In retrospect, it was an incredibly nice gesture from an esteemed figure to a clearly deranged teenager. Thanks again, dmr.
]]>Starting Point
Before I made any changes, I did a simple load test just to see what we were starting with. I installed httperf, and then directed 20 GETs a second for a minute to the index of our site. The good news is that I didn't kill production entirely. The bad news is that half the requests timed out, and the average response time for all requests across the load test (9+ secs) was unacceptable.
I didn't get too upset here; I expected us to do poorly. After all, we'd spent all of our development time in actually building something people want, not on scaling up for thousands of imaginary users.
On a side note, I had absolutely no idea how much traffic to expect from our PR blitz. Regardless of that, I knew we had to stay up. PR is a long, expensive process involving many, many parties, and a company doesn't get many chances with these outlets or their readers. In order for us to truly benefit from all of this visibility, we had to stay up. As a result, when I was load testing, I used gigantic numbers to ensure we were absolutely prepared.
True App Performance
In my initial load test with httperf, I noticed something interesting: the response time to the initial batch of GETs weren't even that good. The initial problem wasn't one of scalability, but one of plain old web performance. Yay, that's considerably less terrifying!
After a few rounds of profiling our Django requests, I determined that the main bottleneck was with a database query. The queries required in rendering the index of famigo.com are preposterously complex. Why is that? It's because we allow our users to sort and filter apps by every data point in the known universe.
Before I began rewriting any of those insanely tough queries, I enabled profiling in MongoDB and ran the queries again. This showed me something wonderful: the problem wasn't wiht the queries, but with a lack of indexes. (Edit: we weren't lacking indexes entirely, but we had changed the query recently and we were now lacking 2 important indexes.) Hooray for easy solutions! I created the indexes, reran the queries, and saw the immediate improvement. We were now rendering our site index in 1.5 seconds or less, compared to 3 seconds before.
Http Requests and Page Size
My next step was to use YSlow to further analyze our site's performance. Side note: YSlow is awesome, and it singlehandedly redeems Yahoo as a company in my eyes.
I learned a few surprising facts about our site in YSlow. First, holy moly, we send a LOT of data over on each request; initially, this was over 1 MB. Now, if you're on a cable modem or DSL, you probably wouldn't notice the transfer time. Someone on 3G most certainly would, though. There's a scalability aspect to this as well. The larger our response is, the longer it takes our webserver to respond to a request, and thus the fewer requests we can actually respond to.
Why was our site index over 1 MB? It was all due to image size. I'm roughly as talented graphically as a manatee, but downgrading images is a problem even I can solve. Once I did that, our response went down to 300k. YSlow found a few other items to correct, like the fact that our Javascript wasn't at the bottom of our HTML, a few scripts that weren't gzipped, etc. These were more easy fixes.
Cache
Now that the site was responding quickly and with a minimum amount of data, I set to work on our cache. We use memcached to cache the results of certain, expensive, out-of-process operations, like complicated queries and requests to certain APIs. Typically we cache all of this for an hour. That's typically an okay trade-off: it keeps the data on our site reasonably fresh as you navigate through it, and it's only when we refresh the cache that requests take a little while.
With all of these new visitors, I wasn't particularly concerned if the data was fresh, we just needed to respond quickly. We needed to approach this like a rapper in a gentleman's club: straight cache, homey.
I cranked our cache life to 24 hours, and then I warmed the cache up myself by clicking every dang page I could find. A random visitor would have to do something very strange in order to request something that hadn't already been cached.
Load Balance
All of our web requests typically go to the same EC2 instance. I began to worry: What would happen if that instance burst into flames from all of the traffic? We had identical VMs that could serve those requests, but there'd be definite downtime as A) I responded to the 'Kaboom!' alert and B) DNS changes propagated.
The clear solution was to use a load balancer to spread our web requests across those EC2 instances. Amazon makes this both simple and impossible. It's trivial to create an elastic load balancer, and point a subdomain (like www.famigo.com) to the load balancer. It is not at all simple to point the domain itself, unless you use Amazon Route 53 to create a hosted zone for your domain, then use Elastic Load Balancing to add a zone apex alias to your hosted zone. I copied that from Amazon's help; I still have no idea what it means.
Due to this, the load balancer handled requests to www.famigo.com, but famigo.com was still pointing to just one instance. I then spent the next several hours hoping journalists would direct visitors to the right URL. Fortunately, they did. For this, I thank the media as a whole.
Conclusion
In the end, we did not go down. The last round of load tests with httperf showed us handling 200 GETs a second indefinitely, with an average response time of about 1.2 seconds. We served thousands of requests without even a hiccup, and we did it without spending any money.
If there's one thing I learned from Spider Man, it's that great power comes with great responsibility. (Secondary lesson: everybody loves a guy in tights.) Cloud hosting delivers on the great power, but it's not helping us with the great responsibility.
Let me explain what I mean by responsibility by pointing to a typical cloud hosting scenario for a growing enterprise. This company has a few developers and a few VMs. From time to time, a developer needs to spin up one or more new instances to help with particularly complex computation; hey, no problem, that's like $1 an hour! This company backs up the important stuff, maybe partial or whole images, and shoves it all onto S3 or Rackspace Files. Not much is monitored or analyzed, but in the short term, this is all just fine.
The problem arises when the company continues to operate this way. Even if it's a small technical team, they'll quickly reach a point where no one knows what all these VMs are for. No one will know what backup belongs to what instance, and if these backups are even operational. The company will see a gigantic bill and a ton of resources, and not know what of this is waste and what is actually powering the business.
Think back to the crufy old IT department at your last job. They were a pain to deal with, because they insisted on process for everything. Want a new server? Justify your case, go through the proper channels, and you'll get it in a couple of weeks. It wasn't fast or efficient, but it was... responsible.
That wasn't the end of their responsible behavior. Once you got your server, those IT dudes monitored it, kept its OS up to date, backed it up, and ensured that the failover procedure actually worked. Again, this is all a lot of work and process and time, but there are some great side effects. You know how much you're spending; you know who's using what; you know that if something goes wrong, you can get back up quickly.
Like a lot of other folks, I left a larger IT organization for a startup, and along the way, I eschewed hosting or colocating my own servers for the cloud. We get a ton more power at a fraction of the cost, and we can scale up and down on the fly. In moving to the cloud though, a lot of us have forgotten something: Amazon or Rackspace provides the tools, but they don't fulfill the responsibilities.
This brings me to two separate points. First, there's a huge opportunity: find out the ways we're using the cloud irresponsibly, and fix it for us. I know one startup looking at this, but there's just so much to do.
Second, there's an opportunity for all of us to learn a lot here. Imagine something like a cloud maturity model, where the people who've had a lot of success here can help lay out a roadmap for the right way to use all of these powerful tools. Given the scenario I described above (which I think is totally common), where do we go next? Anybody have any ideas on how to get started here?
By maturity model, I do NOT mean something huge and bloated, like the Capability Maturity Model. I just picked a catchy name. Something short and sweet, like the Agile Manifesto, would be better.
]]>At Famigo, we house all of our valuable data in MongoDB and we also serve all requests from Amazon EC2 instances. We've devoted many mental CPU cycles to finding the right architecture for our data in the cloud, focusing on 3 main factors: cost, reliability, and performance.
Reliability
If you're hosting anything in the cloud, you must keep one scary question in mind: what if this node disappears? The node could disappear because of an availability zone outage (wuzzup, EC2?), an actual hardware failure (remember, it's still hosted somewhere on something), or a werewolf attack on a datacenter. Regardless of the cause, it's safe to assume that this will happen at some point.
Since our data is crucial to our business and hard downtime is not an option, we want our data replicated across multiple nodes. Not only that, the ideal scenario would have automatic failover, so that if one node dies, another can take its place without any human involvement. In addition, it'd be great if, when a flaky node comes back up, it automagically rejoins in the replication.
MongoDB replica sets satisifies all of these requirements, and they're quite easy to set up. Replica sets are like asynchronous master/slave replication on steroids. A replica set has 1 primary node, and at least one secondary node. Each node has a special oplog collection, which is an ordered list of writes performed on the data. All writes occur first on the primary node (and represented in its oplog). Secondary nodes are alerted to changes in the primary node's oplog, and so all data changes are made quickly in the right order.
One cool feature of replica sets is automatic failover. When a primary node goes down, an election is held and a new primary node is elected within a couple of seconds. Yep, within seconds; there are no electoral college shenanigans here. Another great feature is automatic recovery. When a node falls behind, it catches up by iterating through the primary node's oplog.
Replica sets sound awesome, but they can be a little bit complicated for the following reason: a set must contain at least 3 nodes. Why? The primary node is determined by voting, so you want an odd number to break any ties. Thankfully for our bank account, there are special, lightweight nodes called arbiters that don't actually store any data, but exist solely to vote. While you do need 3 nodes, you don't need 3 full, high-performance nodes.
Replica Set Performance
Replica sets sound as if they could be slow, since all writes must occur on one particular node first, then filter down to every other node. Also, by default, you can't even read from non-primary nodes. So, that means all writes AND all reads must occur on this one node that's also orchestrating all of the replication. Heavens to Betsy, this sounds terrible!
Thankfully, there is a setting you can apply to your replica set to allow reads on your secondary nodes. This setting is rs.slaveOk() and can be entered into your mongo client at the command line. You can't guarantee that the data from these reads are completely up-to-date with the canonical dataset on your primary node, but practically speaking, it's good enough. I've found this to be a very worthwhile trade-off, and so we always set rs.slaveOk().
Thus, with just a few keystrokes, a replica set allows you to scale your reads across many nodes. We scalability nerds should find that pretty exciting.
Node Performance
What are the ideal specs on each node in the replica set? At the very least, each node needs a 64 bit processor. This is because MongoDB uses memory-mapped files for performance, and so any instance with a 32 bit processor can only access 2.5 GB of data.
With regards to RAM, the best rule of thumb I've seen is that a node should be able to fit its working set into memory for best performance. That's not necessarily your whole database. If you're using a tool like top, it actually looks like Mongo is using a boatload of memory. This is an illusion, since much of this is cached.
If each node needs a 64 bit processor and (probably) a lot of RAM, it's starting to sound like it can get expensive. Thankfully, there's a shortcut here. Remember, if your node is an arbiter and only useful for voting, it does NOT need a 64 bit processor, nor does it need a lot of RAM. An arbiter can be the tiniest instance you can possibly get away with. If you have a Casio watch that you bought for $3.50 at a garage sale, I'd seriously consider using that as your arbiter.
Cost
On Amazon's EC2 service, I've found that a Large instance works best for each primary or secondary node. These are 64 bit instances with around 8GB of RAM. Currently, you can spin one of those up on-demand for $0.34 an hour. Sadly, these are Amazon's cheapest 64 bit instances.
One cool thing about Rackspace Cloud instances is that they're all 64-bit. For a primary or secondary node, you can choose the cheapest instance with enough RAM. If you're looking for something equivalent to the EC2 Large instance, it's currently $0.48 an hour.
Yes, they are expensive, but you don't have to devote these instances solely to MongoDB. They can also host your API, web server, etc.
For an arbiter, I recommend finding the smallest instance you possibly can. I use an EC2 micro instance here, which you can get for free in certain conditions or spin up an on-demand instance for $0.02 an hour. The Rackspace equivalent is $0.015 an hour.
I should note that if you're willing to prepay for these instances, the price goes down significantly. Also, if you have incriminating photos with which to blackmail executives at Rackspace or Amazon, the price goes down further still.
Result
The end result here is that you get your data replicated across multiple nodes, automated failover and recovery, and scaled reads on highly-performant machines for less than $1 an hour. Not too shabby, MongoDB!
Unfortunately, continuous deployment doesn't work for all software. Imagine you're writing the firmware for a missile; there's no room there to experiment with your coordinate system, and push a quick update in the event you accidentally blow up Euro Disney. The same has been true for software that goes through a gatekeeper, like Apple for iOS apps. There, you submit a release to Apple, and wait a week or more as they test. While you wait, you might've deployed your API and website 50 times. It seems arbitrary that you can't do the same thing with your user interface, but then in Apple's store, Apple makes the rules.
The great news for mobile developers is that Android is completely different. When you push a new build of an Android app to the Android Market, it goes live within an hour. While you definitely can deploy and iterate very rapidly in the Android Market, you can't automate these deployments: the Android Market has no API allowing for this. Without that, truly continuous deployment isn't an option for the Android Market.
This does not rule out continuous deployment for Android apps, though. Almost all Android devices have a setting that, once configured, allows them to install non-market apps. It's through this setting that third party app stores like Amazon work.
Here's the interesting part: if this setting is configured properly, you can distribute your own Android application. Since you're distributing the app, you can also automate the building and deployment of your Android app. This can all be done in a simple shell script inside of Jenkins or any other continuous integration framework. You run your tests, you build in release mode using ant, you sign the apk using keytool and jarsigner, then you place your apk at a public URL. Once you do that, bam, you're continuously deploying your mobile app!
We actually tried this at Famigo, and the results were... not good. Okay, they were bad. Well, let me be a little more descriptive: they were straight up cover-your-eyes, hide-the-children, ye-gods-what-have-we-done? bad.
We have a site where we review hundreds of Android apps to determine if they're safe for your family; this content is visible via the web, but you can also access it via an app. Once we detected a visitor to the site was on an Android device, we'd render a little banner saying, "Hey, download our Android app!" We weren't linking to the Android Market there, we were linking to our own, continuously deployed Android app, residing on our web server.
While we were running this experiment, we saw several users follow the link to install the app (don't have the numbers, but it was greater than 20). How many actually installed the app? Zero. The results were bleak enough that, after 2 days, we started redirecting people to the Android Market again. Our conversion rate quickly went way up, effectively an infinite percent increase. Uhh, hooray?
It was a beautiful strategy, but it failed. Why? First, many Android users don't have "Allow install of non-Market applications" checked. In fact, many devices (looking at you, ATT!) actively prevent their users from setting this. (This is also an interesting commentary on the relative popularity of third party app stores.) Even if the user could set that option, they were unwilling to do so for us. That brings us to the second, larger reason this failed: no one else is distributing their own app, so users assumed we were doing something sketchy. After all, if it's just a regular app, why isn't it in the Market instead of some random website?
As more third party app stores emerge, users may no longer be wary of apps distributed outside of the Android Market. Until then, continuous deployments of your Android app are a technical success that will probably lead to a business failure.
]]>"We'll make mistakes, but I'd much rather our mistakes come from us going too quickly than us going too slowly," I said.
He smiled when I said that. "That reminds me of the distinction between function and purpose," he said.
I looked at him blankly. Do what now?
"Think about the brakes on a car. What's the function of car brakes? To slow down the car. What's the purpose, though?"
I thought for a moment. "Hmm, to save your life?"
"Close. The purpose of brakes is to allow you to drive fast."
Think about any practice we employ to build great software: unit tests, code reviews, continuous integration, and a zillion other things. As developers, we understand the function of all those very well, but I think we often forget about the purpose.
Let's pick on unit tests for a minute. What's the function of unit tests? They allow us to verify that software behaves the way we expect it to, in an automated manner. What's the purpose of unit tests, though?
The purpose isn't so you have great code coverage numbers or the warm, fuzzy feeling you get when you see them pass after a change or the awesome blog posts and tweets you can publish about your sweet testing culture. It's pretty much the same purpose as brakes on a car: writing unit tests allow you to go fast. (In fact, I'd say they allow you to go fast the right way.)
I actually think most good development practices share that same purpose. Sure, you're doing completely different things when you're writing a unit test versus reviewing someone's code versus scripting out your deployment process, but they all lead to the same result: better software, done faster. If you find yourself following a practice that doesn't share that purpose, you're following the wrong practice.
I think this distinction is important to remember. The next time you find yourself spending hour after hour debating methodology, remember the purpose behind what you're doing and try not to get swept away in the function.
]]>Hence what, for lack of a better name, I'll call the Python paradox: if a company chooses to write its software in a comparatively esoteric language, they'll be able to hire better programmers, because they'll attract only those who cared enough to learn it. And for programmers the paradox is even more pronounced: the language to learn, if you want to get a good job, is a language that people don't learn merely to get a job.
Paul wrote that in 2004, and since then, many companies have embraced these languages. For proof, check out the percentage of jobs on indeed.com that mention Python or Ruby. According to indeed, 0.6% of all job postings for all positions reference Python. We have reached the point where these technologies are no longer 'comparatively esoteric', where plenty of developers learn these languages just to get a job.
It's tempting to say that the logic behind the Python Paradox still holds true, and that we simply need to find other languages. To attract curious, great programmers, maybe we should be building our software in the Clojure or Erlang. I don't think this is right. Even back in 2004, it was easy to build great web apps with Python, while I wouldn't wish building a non-trivial Haskell web application on my worst enemy. The current wave of new languages are interesting, but I think they work best for a subset of problems and that many of these problems won't be encountered by most web startups.
I still do think that startups can find great developers based on technology choices, just not languages choices.
Ladies and gentlemen, I present the NoSQL Paradox: if a company chooses to store its data in a comparatively esoteric datastore, they'll be able to hire better programmers, because they'll attract only those who cared enough to learn it. And for programmers the paradox is even more pronounced: the datastores to learn, if you want to get a good job, are the datastores that people don't learn merely to get a job. See what I did there?
NoSQL stores like MongoDB and Cassandra are where Python and Ruby were, comparatively speaking, 7 years ago. They're featureful, they can be applied to many different problems, they're driving a lot of innovation, they have passionate followings in the development world, and best of all, the vast majority of companies aren't using them. To prove that last point, I point you to the indeed figures for job postings mentioning MongoDB and Cassandra, each of which are about 0.02% of all postings or 1/30 the Python figure listed above. While the vast majority of web companies may not need Erlang, they can certainly use one of these databases.
Anecdotally, I can vouch for this. At Famigo, we build everything in Python on top of a variety of different datastores, including Mongo, Redis, Solr, and CouchDB. As we're looking to hire more people (if interested, email codypo at gmail!), I talk about our stack with a ton of programmers. The good ones want to talk about Python, while the great ones want to talk about Redis and Mongo.
Just like other companies eventually grokked the Python Paradox and hired accordingly, they'll do the same here and mitigate this advantage. It'll take a while though, and until then, I'm going to build and hire accordingly.
]]>
The whole idea behind indentation is to clearly define where a block of control starts and ends. Especially when you've been looking at your screen for 20 straight hours, you'll find it a lot easier to see how the indentation works if you have large indentations.Now, some people will claim that having 8-character indentations makes the code move too far to the right, and makes it hard to read on a 80-character terminal screen. The answer to that is that if you need more than 3 levels of indentation, you're screwed anyway, and should fix your program.
- Linus Torvalds, Linux Kernel Coding Style
What makes indentation evil? Every additional level of indentation is one more thing that you must remember. The more you indent, the harder your job gets. Imagine your code looks like the following:
for x in mylist:
if x meets some condition:
for y in x.list_of_thingees:
if y meets some condition:
do a
else
do b
else:
do c
That's pretty common; there's probably code like that lurking all over your codebase. Now imagine that I ask you to add a new feature here. We only want to make a call to a if we haven't already made a call to b for at least one member of x.list_of_thingees. It sounds easy enough, but think about actually implementing it. Where are you going to spend most of your time? Is it in dropping those couple of lines of code in, or is it figuring out just where in the hulking beast of indented craziness to drop those lines into?
My brain is like a stack. Each level of indentation that I have to think about gets pushed onto that stack, and is more for me to manage. When the time comes to maintain that code, not only do I have to load that entire stack up into my brain again, but then I have to traverse multiple times to figure out where my modifications go. I think that analogy made me sound a lot like that Keanu Reeves movie, Johnny Mnemonic, which is totally what I was going for.
If indentation makes code hard to understand and maintain, how do you fix it? Remove the indentation! It sounds scary, like walking around public in the nude. I assume you though that in the end, it's very liberating, just like walking around public in the nude.
Your language surely contains some great keywords that will aid you in your fight against needless complexity. Here's an example of the above code rewritten using one of these magic keywords, continue.
for x in mylist:
if x doesn't meet some condition:
do c
continue
for y in x.list_of_thingees:
if y meets some condition:
do a
continue
do b
Look at all of the indentation saved, and that's with the addition of only one word! There are far more tools you can use here besides just continue, like the break keyword, functional programming ideas like map and filter, the strategy pattern, and, if your language supports them, list comprehensions. I should warn you, though, that utilizing the concepts I just mentioned might have the horrible side effect of causing you to learn more about your language of choice and programming in general.
]]>