inessential by Brent Simmons

Vesper Sync Diary Follow-up

This is probably the last of the Vesper Sync Diary posts — there’s not much left to write about the design. But I should go back and note which things I wrote about are no longer true, and why. I’ll go article-by-article.

Vesper Sync Diary #1 - Syncing Tags

I especially like this article because it introduced the day-phone/night-phone thing as a way to talk about syncing. Please feel free to use that yourself, if you like.

This article describes tags syncing exactly. No changes.

Vesper Sync Diary #2 - Core Data

I didn’t end up using Core Data.

Vesper Sync Diary #3 - Immutability, Deleting, and Calculated Properties

This article is accurate except for a few details. The main one is that we don’t have a DeletedObjects table — we have a DeletedNotes table. The schema on the server for deletednotes:

id bigint, userID bigint, noteID bigint, serverModificationDate datetimeoffset(3)

On the client:

uniqueID integer

Vesper Sync Diary #4 - In Another Country

I didn’t write about my biggest frustrations with JavaScript. That deserves its own post.

Vesper Sync Diary #5 - Sync Tokens and Efficiency


Vesper Sync Diary #6 - Merging Notes


Vesper Sync Diary #7 - Audibles

Partly accurate.

On the server, attachment metadata is stored in a column in the notes table. Stored as a JSON string.

On the client, attachment metadata is stored in a separate table, called attachments:


Vesper Sync Diary #8 - The Problem of Unique IDs

This article had many follow-up articles:

Vesper Sync Diary #8 part two - More about Unique IDs
Vesper Sync Diary #8 part three - Unique IDs and Hashing
Vesper Sync Diary #8 part four - Random IDs
Vesper Sync Diary #13: Unlucky Numbers
Vesper Sync Diary #13 part 2 - Maybe It’ll Be UUIDs After All<br/ >

In the end: unique IDs for notes are random 53-bit positive integers. The client checks for collisions. (A note ID only needs to be unique for a given user, so this works out.)

The notes table on the server has a primary key that’s a monotically increasing bigint. It has a constraint that each noteID/userID pair has to be unique.

Vesper Sync Diary #9 - Tutorial Notes Edge Case


Vesper Sync Diary #10 - Data Migration

This article references Core Data, but we’re using FMDB + SQLite. (I write the model layer of my dreams. If I had plans for other apps, I could use it. It’s exactly what I want.)

Instead of storing a boolean that says whether or not migration happened, I do this instead:

If the v1 database exists, run the migration. On completion, rename the v1 database. (Don’t delete it.) That’s it.

Vesper Sync Diary #11 - Scaling

This is more about performance than scaling.

(In Scaling and Performance I talk about scaling.)

The article mentions Amazon S3 — but we ended up using Azure blob storage instead. (Which is very similar.)

Otherwise the article is accurate.

Vesper Sync Diary #12 - Let’s Make This Change No Matter How Late

In this article I considered switching from SQL to table storage for account data. Table storage is much cheaper, and everybody likes NoSQL these days.

I did not make that switch. I stuck with SQL — for the reasons the article talks about, but also because all my experience in the last ten years or so has been with SQL, and it’s worth sticking with something I understand well.

Vesper Sync Diary #14 - Keys


Vesper Sync Diary #15 - Server Testing

I’m still writing tests.

Vesper Sync Diary #16 - Debugging Syncing

I haven’t had to do much debugging — but it sure is nice to control all parts of the system.

It does make for craziness where I have Xcode open with the client app, BBEdit open with the API server, several Terminal tabs open, and several Safari tabs opens (server log, for instance). And the app running in the simulator and the app running on my iPhone.

It may be a distributed system, but I can see and affect all the parts on my computer. Sure beats getting frustrated with someone else’s server bugs.