Truth and lies

Sync gets easier when you separate the authoritative data you know from the provisional data that makes a product feel immediate.

September 19, 2023

Sync is one of those problems that you’d think was solved long ago. One that any software developer can easily handle. And yes, if you’re utilizing a service like iCloud or another data layer that manages sync for you, it’s relatively straightforward (at least, for the most part). But, if you don’t have the ability to use a service like that, it’s still pretty rough — even in 2023 — when you have to access data over there using a copy of it over here.

When I was at Wunderlist working on real-time sync, we fought this problem in a deep and hard way. Our source of truth was our database servers in our AWS account, and each of our clients needed to stay in sync with it. We learned a lot of different ways to mess this up, but at some point somebody much wiser than I gave a key insight into how to approach it.

The insight is: When you’re dealing with a local view of a remote database and need to make locally visible updates as well, you have to separate your view of your local data into truths and lies.¹

Truths are the data that you keep up to date via the best, most authoritative API you have. At Wunderlist, this was using our REST API. Sync using this API wasn’t slow by any means. We had a bookkeeping system that let us incrementally update our collection of truths over time. But even though it was quick, essentially just asking for a last-modified revision update to see if anything changed before asking for anything else, rapidly polling a remote master for state change at a rate that would look real-time is a horrible thing.

Lies are the data that you apply on top of the truth as a view in order to drive the client user experience. In our case, some of these lies were tasks that a user had just created and which may not have been transmitted back to the server, such as when a user was offline. Other lies were messages received over either a websocket or other kind of push notification that had new information that would be immediately useful to display to the user, even before running a full synchronous sync.

These lies were incredibly useful, especially the ones that came via push notifications. This is what made Wunderlist sync appear instant to people even if the real sync process was something that was only run on an infrequent basis. Data would be displayed and could be acted on instantly even as the underlying machinery caught the real source of truth up. It worked so well that we were able to rely on these notifications to drive updates in almost every case other than application startup or the device coming online.

And doing this allowed us to side-step the need for a multi-master sync. Like cryptography, true multi-master sync is something that mere mortal software developers probably shouldn’t roll on their own. Most of the time, boring parent-child sync with a fast channel for transmitting lies that can be used for optimization purposes is a much more approachable solution.