Evergreen Diary #7: Syncing and Immediate and Deferrable Actions

There are, as I mentioned previously, two types of syncable actions: immediate and deferrable.

An immediate action is something like adding a feed: it requires that the server is reachable right now, so that the feed actually gets added.

A deferrable action is something like marking articles as read. The sooner the server knows, the better, sure — but it can wait if necessary. It can even wait for weeks or months.

I’ve decided to make it simple and categorize these actions like this: any structural action that affects the list of feeds and folders (or tags) is an immediate action, while any action that affects the status (read, starred) of articles is deferrable.

Or, in UI terms: if it’s something you do in the sidebar, it’s immediate; it it’s something you do in the timeline, it’s deferrable.

Most importantly: you can read articles while you’re offline.

Discarded alternate approach

You could argue that all actions should be deferrable. For instance, Evergreen could remember that you want to add a given feed, and add that add-feed action to the sync queue and wait for the server to become reachable again.

But doing so would add more chance for conflicts. Consider the case where the add-feed action got queued (on your day Mac) and then just sat there. Then, on your night Mac, you add the feed and it succeeds — then you decide you don’t like it after all, then delete it.

The next day, back on your day Mac, the add-feed call finally goes through, and then you have to delete it again. (And then you report an Evergreen syncing bug.)

There are other issues as well: do you add the feed locally? And read it? No, because this gets ambiguous quickly, as Evergreen’s article IDs won’t match the article IDs the server has assigned. Evergreen might not even have found the same feed URL the server found (in the case where you just gave it a website URL).

So we’ll stick with these two categories, immediate and deferrable.

To refine the definitions: a deferrable action is one where the state change can be reflected locally without risking serious conflicts or ambiguity.

Immediate actions

These will result in an immediate API call, and the result will be reflected in the UI. This is straightforward.

Deferrable actions

These actions will be stored in a database.

The simplest way to do this is probably a set of tables: articlesToMarkRead, articlesToMarkUnread, articlesToStar, articlesToUnstar. Each table would have one single column: articleID.

This layout is probably the most efficient way to add/delete articles.

This is not actually an ordered queue, but I think that’s okay. Most of the time all four tables will be empty.

When articles are marked read (for instance), then those articles will be added to articlesToMarkRead and deleted from articlesToMarkUnread. The system will then know that it has pending status changes to the send to the server.

Then, periodically, the app will attempt to contact the server. Most of the time this should happen within a minute or so.

Part of the idea is using this system to coalesce calls: the app shouldn’t call the server every time you select an article (which marks it read). Better to update the database and call the server very-soon-ish, but not necessarily this exact second, to allow for multiple articles to queue up.

The downside to this particular layout, which may make me change it, is that supporting additional status types means adding more tables. But the alternative is to add more columns to a single table, which is better but not necessarily that much better.

Another downside is that there’s no automatic guarantee that an article ID won’t exist in, say, both articlesToMarkRead and articlesToMarkUnread at the same time. But the answer here is simple, careful coding: a single bottleneck function that never adds to one without deleting from the other.

Evergreen Diary #6: Proposed Sync Design

Most of the time, a user will do a thing — mark some articles as read, for instance — and then Evergreen will tell the server (Feedbin, Feedly, etc.) right away. That’s good and easy.

The hard part is this: what happens when the server is not reachable for some reason?

Continuing the example: Evergreen could, in the case of the unreachable server, refuse to mark those articles as read.

It could insist that syncing works only when the server is reachable — which is not that weird, especially given that it can’t pull new data from the server unless it’s reachable.

But it’s weird enough. Consider that selecting an article marks it as read. This policy would make it impossible to even just read what you’ve already downloaded.

Offline Syncing

It’s clear that syncing requires some sort of offline ability.

Rather than set a policy for each possible user action (marking articles read, deleting a feed, etc.), I want to set a single policy and single structure for handling these actions.

(Note: there will be some things that do actually require a reachable server: downloading new articles and adding a feed come to mind. This design refers to actions where it would be okay to defer notifying the server.)

Proposed Policy

When the user performs a syncable and deferrable action, that action and the relevant parameters will be remembered.

Evergreen will attempt to notify the server periodically. Once it succeeds, it will forget that action and its parameters.

(Update 18 Jan: 2018: the original version of this design had a time limit, which has been removed. The concern was that this offline queue would grow unbounded — but it can’t, since if you can’t reach the server you can’t download new articles, which means the queue is limited to your current local data set.)

Proposed Implementation

When the user performs a deferrable action, the app tries immediately to notify the server.

If this first attempt fails, then the action is recorded in a SQLite database with a timestamp and with the parameters it needs to be able to make that call again later. Those parameters will be minimal — article IDs instead of entire Article objects, for instance.

The app will periodically attempt to empty this database: starting with the oldest action, it will notify the server of each. On success, a given action is removed from the database.

Undo

Evergreen supports undo as much as possible. For instance, if you mark a number of articles as read, you can undo it.

In this case, Evergreen will notify the server of the mark-read action, and then, upon undo, notify the server of a mark-unread action, thereby undoing the effect of the first action.

In the case where the mark-read action is in the offline action queue, that will be found and deleted, and there will be no queued action to reverse the effect.

Non-Deferrable Actions

Actions that require an immediate response from a reachable server include actions such as adding a feed. In the case where a user attempts one of these and the server can’t be reached, an error sheet will explain the problem. Such actions will not be queued.

There will not, however, be an error sheet for performing a refresh when the server is unreachable. Instead, in that case, a warning icon is placed next to it in the sidebar, a la Mail and similar apps. Clicking the icon will attempt a connection to the server, and an error sheet explaining the problem will appear if the server still can’t be reached.

Other Notes

Evergreen works like Mail and similar apps in that it has a set of accounts for services such as Feedbin, Feedly, and others, along with an On My Mac account that is not synced.

Each account has its own section in the main window’s source list; each account has its own set of folders and feeds.

Accounts will be added to Evergreen in Preferences, in an Accounts pane.

The policies of these accounts will be respected. For example, if the On My Mac account automatically marks articles as read after n days, it’s possible that some service does this after some other number of days, or never does that at all, and Evergreen won’t override those policies for that service.

Another example: the On My Mac account doesn’t support folders-within-folders — but if a feed service does support that, then the app will support it too, for that service.

(Current plan: Evergreen 1.0 will ship with Feedbin support, and Evergreen 2.0 will add support for more services.)

Let me know if any of the above sounds weird or if I missed anything. (There is contact info at the bottom of Evergreen’s main page.)

The Omni Show #6 with Liz Marley

This episode features Liz Marley, OmniGraffle engineer, formerly a tester and PM.

She talks about switching from testing to writing code — and about how she came to do technical talks using Swift playgrounds. Playgrounds with ducks, that is, because every good playground has ducks.

Evergreen Diary #5: Send to MarsEdit

The latest build of Evergreen, fresh from the lab, now adds MarsEdit to the sharing menu in the toolbar.

When you choose the command, it sends the current article to MarsEdit — which opens it in a new window, and then you can edit and add your own commentary before posting to your blog.

* * *

See SendToBlogEditorApp.m for the code that packages up the article and sends an Apple event.

The code is not MarsEdit-specific — other apps in the past have supported this same Apple event, though I don’t know if any other current apps do. If I find some, I’ll add them to the sharing menu alongside MarsEdit.

* * *

The Apple event code is written in Objective-C, even though I almost always write new code in Swift. But, with Apple events code, Objective-C is easier.

It’s easy to call from Swift: see SendToMarsEditCommand.swift.

(Reminder: this is all MIT-licensed. You can use this code.)

Evergreen Diary #4: Send to Micro.blog

The latest build of Evergreen adds Micro.blog to the Share menu in the toolbar, if you have the Micro.blog Mac app.

It sends the title and link of whatever you’re reading over to the Micro.blog Mac app, and you can edit it before actually posting.

This is hugely important. RSS readers exist not to just make reading easy but to make the web a conversation.

The next release will (probably) include MarsEdit, for the exact same reason. If you have ideas for other apps to include, let me know (see the bottom of the Evergreen site for contact info).

Ideally, every app like Micro.blog and MarsEdit would have a sharing extension — but, where they don’t, I can add them as long as they support some kind of way to do so.

Technical Details

Micro.blog supports a URL scheme for sending it a post: see See SendToMicroBlogCommand.​sendObject for the implementation in Evergreen. (And of course feel free to use any Evergreen code in your app.)

History

It’s not that we realized just today that posting from an RSS reader is important. Since posting was always an absolute requirement, way back in 2003 I released NetNewsWire 1.0 with an integrated blog editor.

I got that idea from Radio UserLand, which was an RSS reader and blog editor by the company where I worked before I wrote NetNewsWire. It’s Dave’s idea, not mine.

By the time I was working on NetNewsWire 2.0, it became apparent that smooshing a blog editor into a reader app meant that the blog editor suffers. The blog editor in NetNewsWire 1.x was pretty darn bad. So I had the idea of splitting out the blog editor to a separate app — inducing mitosis — which you can read about in my MarsEdit report of late 2004.

Mitosis was a success.

One of the keys of that success was that the API for contacting MarsEdit had to be open, and NetNewsWire had to support any blog editor, or any other app, that supported that API — because otherwise I would have been unfairly leveraging success with NetNewsWire on behalf of my new blog editor, and that would have been wrong.

MarsEdit still supports that interface, and my next step is to write the client side in Evergreen, so it can send to MarsEdit and any other app that supports that interface.

I am, by the way, delighted to be writing code for a 14-year-old API that still works.

Mentions: Proof-of-Concept

Ben Curtis posted feed_searcher to GitHub — it creates custom search feeds, and even has a handy Deploy to Heroku button. Cool.

(This is in reference to App Idea: Mentions).

App Idea: Inside Story

I had this idea around the time Apple came out with Bonjour (née Rendezvous), and all these years later I realize I’m never going to get around to it.

Here’s the scoop:

The idea is microblog posts (tweet-like) but that live inside a specific network only.

You run a Mac app that:

  • Lets you post short messages.
  • Shows short messages from people inside your network.

The app runs a small webserver that’s discoverable via Bonjour, that has a specific service name. That webserver publishes a feed of your posts.

The app also downloads a feed from every other app it finds (and caches those feeds, for when other apps are offline).

In other words, it’s instant office microblogging with no centralized server. Completely peer-to-peer.

The design is what you expect — a reverse-chronological list of posts with usernames and avatars. You can reply, mention people, mute people, etc. (I think you’d follow everybody on the network automatically — so muting becomes the thing, not following.)

* * *

Important thing: your posts are network-specific. So if you go to a coffee shop or the airport with your Mac, you won’t be publishing your at-the-office posts.

* * *

I haven’t done research into whether this is do-able on iOS. Is it possible to run a little webserver, in a backgrounded app, that other people on the local network could connect to? Seems unlikely — but I don’t know. Cool if you could.

* * *

I have no idea how you’d make money with this one. If you charge for it, not enough people would use it to make it worthwhile. (The standard dilemma of social networking apps.)

Maybe, though, you could add some IAP features? One might be to let people add RSS feeds, so it could also function as a lightweight RSS reader. A person might want posts from Daring Fireball and wherever else added to their timeline.

Or just add that feature anyway, and do the whole thing for fun.

PS “Inside Story” is a long name. Should be one word. Scoops? Bulletins? Officecast? Gossip? (“Gossip” was, if memory serves, the name of an app Chuck Shotton was working on a long time ago. Probably reusable at this point.)

(I mean, c’mon, office Gossip. It’s so perfect.)

App Idea: Mentions

“Hold on — I need to check my Mentions.”

Ten years ago or more we had several blog-specific search engines and services: Technorati, BlogBridge, and others.

One of the great things about these services was not just being able to search for something but being able to set up persistent searches: that is, you’d get a search as an RSS feed, and in your feed reader you’d get results from all over the place on the thing you’re searching for.

In the obvious and common cases, you’d set up searches for people linking to your blog, writing about the apps you work on, mentioning the place where you work, and mentioning you.

I’d like to see something like this for today, but where the scope is just the Apple/Mac/iOS community. It would crawl the obvious sites (such as Daring Fireball and Loop Insight) and it would crawl the many blogs and microblogs that make up the community.

I think this is best as a web service, though it could have Mac and/or iOS client apps. It needs to provide feeds for people using feed readers.

* * *

With that reduced scope, and with the better tools and cheaper cloud computing these days, I don’t think it would be terribly expensive. Not like it was 15 years ago.

You might want to make money with this, though, and there are a few ways to do it: charge extra for things like notifications and email alerts. Charge money for client apps. Make the first three search feeds free, and charge money for a ten-pack of searches. Etc.

* * *

An alternate version of this idea isn’t a web service — it’s a Mac and/or iOS app that does the crawling. A kind of specialized Mac/iOS feed reader.

The list of feeds-to-crawl would probably just be an OPML file on the web, and the app would periodically grab that list.

(If you did this, you could use RSParser as starter code, since it parses OPML, RSS, Atom, JSON Feed, and RSS-in-JSON.)

This would make a great open source project, but it could just as well be free or for-pay.

* * *

One challenge is handling spam and abusive or hateful content. You’d most likely want to have a suggest-a-site form, so you don’t have to go out and find every single site there is.

But you have to be able to say no, and you have to be able to update the list of sites-to-crawl quickly if a site turns bad or inappropriate somehow.

You’d also need a way for users to report an issue.

(Again, the reduced scope — Apple-related blogs — makes this somewhat easier than if you tried to do the entire blogosphere.)

* * *

I want this! I’d use it every day. I know other people who would too.

But I’ve got enough on my plate that there’s no way I can do it myself — though I’d be happy to answer questions and provide feedback to anyone who does want to do it.

Evergreen Diary #3: On Punting

I have a vision for 1.0, and then, as time goes on, I have to cut it back beyond where it hurts. I have to keep punting. I hate this part of shipping software.

But I also love it because it reminds me that I have the stomach for it. Shipping software is an emotional skill.

After shipping — no matter what — there will be people who absolutely cannot believe that feature X wasn’t included. In fact, it’s the one thing they totally need.

And they’re right. Not wrong. And I would have loved to have included feature X, loved I not quality (the app maker’s honor) more.

* * *

I wondered if Evergreen as open source instead of as a for-pay app would affect what and how much I punt. I hope the answer is: not at all; the decisions would have been the same.

I think that’s right because I’m trying to make as good an app as I can, which would be true regardless.

But it is nice not to have to consider money as I make decisions. I do think about how I want it to have as many users as possible, and I want people to love the app. And I have to think about the economy of my time. But I don’t have to consider money.

Another difference between open source and commercial is that I can be utterly transparent about what’s been punted. Check out Evergreen’s 2.0 milestone. It will keep growing. Every single one of those things was originally supposed to go in 1.0.

* * *

Some things I keep reminding myself as I make progress toward 1.0…

This isn’t the last release, it’s the first. There will be many more. (I hope to work on this app for 20 years. It’s been only three years so far.)

Any feature I ship, I probably have to support forever. So it’s wise to be cautious.

Quality — design, stability, performance, lack of bugs — is way more important than any collection of features.

And, perhaps most importantly: shipping 1.0 means learning about your users, how they use your app, and what they need — and those lessons will change future plans, as they should. It’s best to learn those lessons early, before doing too much.

The Omni Show #5 with Mark Boszko

In this episode I finally get around to talking to Mark Boszko, the show’s intrepid producer and The Omni Group’s Video Producer.

I’ve known Mark for more than ten years — we met at a SXSW conference many years ago, long before either of us came to Omni. Mark didn’t even live in Seattle in those days. Now he does, and now we have the pleasure of working together on a podcast.

PS If you like movies, definitely check out the podcast Mark hosts: The Optical.

Archive