Saving our bacon with evil cucumbers
As Wunderlist 3 neared launch in 2014, we needed to know whether our rebuilt backend could survive real-world load. The answer came from thousands of simulated users that broke servers, exposed bad assumptions, and made launch day mercifully quiet.
We knew the new server architecture for Wunderlist was fast. So fast, in fact, its performance surprised us. Changes made to a list on one device would show up on every other device so quickly, it was hard to believe that the communication between devices was going from Berlin to Ireland and through a half dozen servers before making its way back. It felt like the kind of speed your TV responds with to your remote control.
On the other hand, we knew that perceived performance with only a few hundred users on the system would not be any indicator of how well it would work after we released Wunderlist 3. I wasn’t at the company for the launch of Wunderlist 2, but everyone that was recalled that it performed impressively well in testing. Minutes after launch, however, it melted like an aluminum can in a hot fire. We didn’t want a repeat of that, especially not after rebuilding our backend infrastructure from the ground up.
What we needed to do, of course, was to load test.
Over a coffee at a local café, Chad Fowler and I sketched out a plan to take the Java-based sync library from our Android client and use it to build a service that would simulate — or at least approximate — real-world load. In our service, we would run tens of thousands of virtual users which would each do a semi-random set of actions, such as create lists and tasks, share them with other virtual users, and then delete them. For reasons that can only be blamed on our foamy beverages, we decided to call our simulated users Böse Gurken — German for evil cucumbers. What can I say other than the visual of evil cucumbers beating down our servers amused us?
We built our Gurken as actors running in an Akka actor system deployed onto multiple machines in our immutable infrastructure. Each actor simulated a user that would create an account and then log in with two clients. The idea was that one client would represent Wunderlist on a phone and the other would represent Wunderlist on their desktop. As each Gurke worked, its data would sync between the two clients.
To give you a quick feel of how we made this—don’t worry, you can safely skip over the code blocks if Scala makes your eyes glaze over—here’s a simplified sketch of how we start up our Gurken with a Gurke Meister:
val userCount = config.getInt("userCount").orElse(1024)
val meister = Akka.system.actorOf(Props(new GurkeMeister))
meister ! CreateUsers(userCount)The Meister, in turn, handles the CreateUsers message (sent with the ! operator on the last line of the code block above) like this:
def receive = {
case CreateUsers(userCount) => {
val user = context.actorOf(Props(classOf[GurkeUser]))
user ! SignUp
user ! LogIn
user ! CreateGurken
}
}Each user actor signs up for an account, logs in, and then creates two Gurken—the clients that simulate a running copy of Wunderlist on a phone or a laptop. Once all the Gurken have been created, they kick into action and start up an internal timer which sends a Tick message on a regular basis. This is where things get fun, for some definition of fun.
def receive = {
case Tick => {
roughlyHalfTheTime { createAList }
prettyOften { createATask }
roughlyHalfTheTime { completeAtTask }
// and so much many more actions...
}
}The infrequently, roughlyHalfTheTime, and prettyOften functions aren’t pseudocode, even though they might look like it. They are our quick and dirty mechanism to give some amount of variability into what happens on each cycle. Otherwise, our Gurken might behaive a bit too predictably. They’re implemented like this:
private def prettyOften = percentOfTheTime(8) _
private def roughlyHalfTheTime = percentOfTheTime(2) _
private def infrequently = percentOfTheTime(5) _
private def percentOfTheTime(chance: Int)(f: => Unit) = {
if (Random.nextFloat <= chance * 0.01) { f }
}Of course, there’s a bit more to Böse Gurken including the details of creating and completing tasks. As well, we put a throttle around the creation of users to keep from flooding the database that handles setting up new users. But, this is the essential structure of how the Gurken operate.
Even in the first iterations, running large numbers of Gurken had a significant effect on our infrastructure. It didn’t hurt that our bad pickles were running at a pretty fast pace. A normal user creates a few tasks at a time over a couple of minutes. Our simulated users were performing several actions every few seconds. Imagine, if you will, that our simulated users were pickled in a bath of simulated methamphetamine—Heisenberg’s best, of course—and you get the picture.
Destroying servers was something the Gurken turned out to be really good at. Every iteration brought new information on how our servers could be brought down which, in turn, showed us what to work on next. It became a bit of a game of cat and mouse. As we continued to iterate, the Gurken became legendary inside the company. Any time anything went wrong for any reason, the first thing our QA team would ask is whether the Gurken were at work.
Even better, running Böse Gurken was downright cheap. We ran them on Amazon EC2 c3.large instances that cost $0.105 an hour. Since most of our test runs were less than an hour long, even running a hundred instances during a run cost less than $10.
The work of repeatedly punishing our servers started to pay serious dividends. After a few weeks, it took ten times the number of Gurken to break our servers. Then twenty. Then fifty. As the Wunderlist 3 launch date approached, we gained confidence. We knew we had to go further, however, so we reworked the Gurken and slowed them down to more closely emulate the speed of a real users. We then increased their numbers by several orders of magnitude and ran them across a fleet of dozens of E2C instances.
This change quickly snapped our infrastructure in new and novel ways and exposed some issues with a few of our fundamental assumptions. For example, it turns out that RabbitMQ — the message queue server that we use to coordinate changes in user data and push those changes back out — is quite a bit more sensitive to network topology than we thought. The documentation says not to cluster Rabbit instances over a WAN, but we found the point at which even the network separation between availability zones in a single AWS data center would break a Rabbit cluster.
If you’re going to break your core messaging server, doing it in the weeks before you launch is so very much better than after you’ve gone live.
In the end, we replaced a naïve belief in whether or not the system would work with an understanding of how the system would break under load. More importantly — although I don’t think we realized how important this was at the time — we got a lot of practice at bringing our system back up in response to intense stress.
When launch day came, we braced for the impact of real users on our system as customers migrated to Wunderlist 3. First, the Android client went public in the Google Play store. Then, a half hour later, the iOS and Mac clients started rolling out worldwide. Our connection numbers went up and our system held, with only a few anxious moments when traffic levels spiked in response to release announcements. We had big expectations in terms of usage and activity and the launch met them.
Things went so well, and we were so prepared for failure, that it was a bit anti-climatic after weeks of terrifying load tests destroyed servers. Not that I’m complaining, mind you. If we hadn’t broken our system a dozen different ways with our evil cucumbers, it’s certain that our launch day would have had a much different ending.
Sadly, a few days after launch, an issue with one of our database servers cropped up and had us scrambling. Evidently, Murphy couldn’t let us off that easy. Thanks to all the testing and the launch day going well, however, at least we knew that it wasn’t a scaling problem that we were facing.