If you are reading this, chances are that you’ve got some experience with personal finance software. You’ve probably tried a lot of apps until you found MoneyWiz… and you probably noticed that MoneyWiz is the only finance app that offers automatic live sync.
But have you ever wondered why? As a matter of fact have you ever wondered why so few apps in the world actually offer a decent sync in year 2012? If you have, read on… I will try to explain how these things work and how our own syncing service, Sync Everything, works.
How apps work with data
Every data-heavy app, such as a personal finance app or a to-do app, works pretty much the same way when it comes to storing and reading its data.
Such apps store data into database files. A database is basically a file, that allows us – the developers – to organize the data into tables with columns. Pretty much like an Excel spreadsheet but much more complex.
There is one more important thing here – where is this database file stored? The simple answer is that it could be stored either on your device (be it a mobile device or a desktop computer), or it could be stored online. Online storage and exchange of data is as old as the Internet, but these days we use this geeky-cool name – “cloud”. Did you notice that for the past year or so, everything has been going to the cloud…well if you ever wondered where’s that popular cloud that everybody knows of, and you don’t – wonder no more. It’s just the good old Internet.
What about syncing?
So, what’s sync? It stands for synchronization and it’s basically the exchange of data between two or more devices, until the data becomes the same on all of them.
There are two ways to sync:
You can sync directly between the devices…Bluetooth and WiFi sync do that, when you connect the devices on the same network, or just pair them with each other…a.k.a. peer-to-peer.
You can sync via an arbitrary third device. That device is called a server, and when it’s on the Internet, we call it “cloud server”. One of the devices would upload data to the cloud server, and the other will download it from it, but the two devices would never communicate directly to each other.
Syncing devices directly, via WiFi or Bluetooth is too much hassle. It’s just and old and obsolete way of doing this. Add a third device, and you will be looking into a serious waste of time. Also, the devices need to be physically close to each other for this to work. So, this sort of syncing solution is off the table…at least for us.
Cloud syncing, in the other hand, looks like the cooler thing to have. It gives more independence, it works with unlimited amount of devices, and it can be automatic. The devil is in the details though…
How do the popular sync services work?
Lots of people ask us if we’ll do iCloud or Dropbox sync for MoneyWiz…so let me briefly introduce you to how these services work and why we won’t integrate with them…
Lots of people know and use Dropbox, and it’s a really great service for automatic files syncing. What Dropbox does is that it gives you a place (a folder on your disk) to put your files into, and when you put them there, they are automatically uploaded to Dropbox’s servers. This of course is possible after downloading Dropbox software that will take care of the uploading and all. That same software, which is also known as a “client software” will also check periodically to see if something new has been uploaded to your Dropbox folder, but from another device, and if that happens it will automatically download it on your current device. All of this is possible because ultimately, all of your data is located on a cloud server, and all of the devices you use just collaborate with this cloud server.
iCloud is actually a set of many services accessible with single username and password. Some of these services include IMAP to sync your emails, service to sync your address book and calendars, iTunes Match to sync your music and so on… One of the iCloud services allows us to sync the databases of apps. This is what we, the developers, can do with iCloud – sync our app database across all of the devices that a certain iCloud use has. The problem with that is just one – this is pretty much the same as Dropbox, because at the end of the day, we will be syncing the whole database file.
You may be asking yourself now – “Why not sync the whole database file?”… I will give you 5 good reasons!
- Imagine that you have an iPhone and an iPad. You may want to enter some transactions on your iPad and some other on your iPhone, but unfortunately one of the devices doesn’t have Internet connection at the moment. You would obviously want to have both devices sync as soon as both of them regain Internet connection. Okay, but you made changes on both devices, therefore you’ve changed the database on both devices. If we were to sync the entire database file, this would mean that either your iPad will send its database file to your iPhone, or your iPhone will send its database file to your iPad. Either way, you lose the changes made on one of the devices, because that device just received a database file that overwrites its own database file. That would be pretty annoying, wouldn’t it?
- If we were to sync the entire database file, this would mean that every time you make a new transaction, we have to upload and download the entire database file that also contains all of your transactions. If you have 1000 transactions, we are looking at at least 2MB database file size. Every time you make a new transaction your Device X will upload 2MB to the cloud and Device Y (your other device), will download them afterwards. If you make 5 transactions per day, using just 2 devices, this would generate approximately 600MB data traffic per month. That’s expensive if you are on 3G!
- Many families use MoneyWiz for their home finances. Such families often have 3 or more devices in total, that are used by different family members. If we used iCloud it would mean that all family members must use the same iCloud account on their devices, otherwise the sync won’t work. iCloud credentials can’t be set per app!
- Some people would propose to workaround the above problems by syncing manually as opposed to automatically…in other words – to ask you, the user, to tap some buttons when you want to sync… That would obviously prevent these issues, but the last time I checked we were living in year 2012… Technologies are powerful enough to allow us to do the syncing automatically for you. Anything less is just a poor excuse.
- iCloud is controlled by Apple. That means that if we went for iCloud, we wouldn’t be able to make MoneyWiz for Web and for Windows for example. That’s a limitation we don’t want.
From Dream to Reality – Sync Everything!
When we started developing MoneyWiz two years ago, one of the major things we wanted to do was proper syncing.
We imagined the perfect sync to take a minute to set up and no time at all to manage. It had to do everything automatically!
We first wanted to find an existing solution that would save us some development time, but it turned out nobody had done this before. So it took us about 2 months just to figure out how to do it and another 6 months to actually develop and test it. Some of the challenges we had were:
- It had to be secure
- It had to be scalable (i.e. allow us to have at least 100,000 active users, and then allow us to scale more on demand)
- It had to be fast
- It had to be reliable…meaning – no downtime.
- It had to properly figure out what to do in all sorts of situations, like when you have many devices, some online, other offline, and you make changes on all of them at the same time.
We knew from the very beginning that syncing the entire database won’t work well. It would’ve been the easy solution, but not the right one. So we came up with something very different… We’ve figured that instead of syncing the entire database file, we will sync the changes only. So basically we’ve decided to sync commands that describe what you do. This way, we could send these commands to all other devices running MoneyWiz, and MoneyWiz would execute them, just as if you did these changes yourself.
It was essential to know the order in which the changes were made…for example, the result will be different if you created some transactions and then adjusted the account balance, and if you adjusted the account balance and created the transactions after that. So, we’ve decided to give each change an incrementing number… We call it “command number”, because essentially changes and commands are the same thing.
It was also important to be able to tell which commands come from different devices. If we didn’t know that, we couldn’t possibly merge the data correctly (if changes were made on several devices at the same time). So we’ve decided to group the changes made by one device. Each group we call “revision”. MoneyWiz uploads and downloads data from the server every 20 seconds. All changes made within these 20 seconds go in one revision. If after another 20 seconds you’ve made more changes, they will go in a new revision and so on. Revisions are also incremental, so we can properly arrange them.
The process works like that…
When you first sign up for the Sync Everything! service, MoneyWiz will send commands for your entire database file… It’s pretty much like it will describe what you did so far, and save that informations to the server. Since all that information comes from just one device, it’s all grouped under one revision. This is revision number 1.
After that, every time you make some changes, MoneyWiz will upload them and create new revisions.
When we spot that there is a revision on the server that hasn’t been downloaded yet. MoneyWiz will download it. This would happen when you did something from another device.
Not only this solution proved to work beautifully, but it also brought another perk – we could revert data back to any previous revision… This way, if we made an unnecessary change or something just messed up with the data, we can always revert it back to the state we want. For now, this functionality is not available for the users, but we have it on our radar to add it in the future.
Security of Sync Everything!
So, what about security? Here’s what we’ve done:
- Your data is encrypted on the server, with industry standard AES encryption
- The entire server is backed up every day, and we keep the backups for the last 30 days
- Only you have access to your data, unless you provide your e-mail and password to someone else
- We use managed dedicated server, being a firewall and server administrators constantly make updates and routine checks to make sure the server is secure from the outside world.
- We’re in the process of obtaining several security verifications, such as Verify, TRUSTe and McFee Security.
Your data is not currently encrypted on the iOS device itself and the communication with the server is not via SSL yet. The reason is that to legally do this, a company needs an ERN number from the US government. We’re expecting our ERN number to be ready any moment now actually, so I expect to start encrypting the local database + make SSL communication in a month or two from now.
We want to improve Sync Everything! even more, so we have a few things on our to-do list that will help us do that:
- Start compressing data and merging old revisions automatically, to speed up the initial sync. Currently the initial sync (the one that happens when you just (re)installed MoneyWiz and you sync from existing sync account) can be slow when you have many revisions. There are people with more than 5000 revisions and the initial sync can take 1-2 hours for them. When merged, the speed improves around 10 times (we’re already testing a solution, and that’s what our benchmarks show). We also have few other significant performance improvements in mind, which should be able to make the initial sync many many times faster.
- Ability to sync attached images – we don’t sync them now, because we can’t afford to keep the service free and pay the server costs if everybody decided to sync all their receipts. We do plan to give a free disk quota for images, and anything above it to be on a subscription plan via In-App Purchase. That is to be expected in major version 2.0 of MoneyWiz, which is planned for summer 2013.
- We have such a great service there… why use it for MoneyWiz only? We could develop other great software, that syncs automatically…