Evergreen Diary #6: Proposed Sync Design

Most of the time, a user will do a thing — mark some articles as read, for instance — and then Evergreen will tell the server (Feedbin, Feedly, etc.) right away. That’s good and easy.

The hard part is this: what happens when the server is not reachable for some reason?

Continuing the example: Evergreen could, in the case of the unreachable server, refuse to mark those articles as read.

It could insist that syncing works only when the server is reachable — which is not that weird, especially given that it can’t pull new data from the server unless it’s reachable.

But it’s weird enough. Consider that selecting an article marks it as read. This policy would make it impossible to even just read what you’ve already downloaded.

Offline Syncing

It’s clear that syncing requires some sort of offline ability.

Rather than set a policy for each possible user action (marking articles read, deleting a feed, etc.), I want to set a single policy and single structure for handling these actions.

(Note: there will be some things that do actually require a reachable server: downloading new articles and adding a feed come to mind. This design refers to actions where it would be okay to defer notifying the server.)

Proposed Policy

When the user performs a syncable and deferrable action, that action and the relevant parameters will be remembered for 30 days.

Evergreen will attempt to notify the server periodically during that period. If it fails, it will forget about the action/parameters.

There has to be some kind of limit: I don’t want to just grow a queue forever. Thirty days is an arbitrary but reasonable length of time.

Once that time is up, there’s a good chance that the actions have been superseded by activity elsewhere, and they’re redundant — or they’d be surprising, because it was long enough ago that a user has forgotten about them.

Proposed Implementation

When the user performs a deferrable action, the app tries immediately to notify the server.

If this first attempt fails, then the action is recorded in a SQLite database with a timestamp and with the parameters it needs to be able to make that call again later. Those parameters will be minimal — article IDs instead of entire Article objects, for instance.

The app will periodically attempt to empty this database: starting with the oldest, non-expired action, it will notify the server of each. On success, a given action is removed from the database.

Expired actions will be deleted from the database.

Undo

Evergreen supports undo as much as possible. For instance, if you mark a number of articles as read, you can undo it.

In this case, Evergreen will notify the server of the mark-read action, and then, upon undo, notify the server of a mark-unread action, thereby undoing the effect of the first action.

In the case where the mark-read action is in the offline action queue, that will be found and deleted, and there will be no queued action to reverse the effect.

Non-Deferrable Actions

Actions that require an immediate response from a reachable server include actions such as adding a feed. In the case where a user attempts one of these and the server can’t be reached, an error sheet will explain the problem. Such actions will not be queued.

There will not, however, be an error sheet for performing a refresh when the server is unreachable. Instead, in that case, a warning icon is placed next to it in the sidebar, a la Mail and similar apps. Clicking the icon will attempt a connection to the server, and an error sheet explaining the problem will appear if the server still can’t be reached.

Other Notes

Evergreen works like Mail and similar apps in that it has a set of accounts for services such as Feedbin, Feedly, and others, along with an On My Mac account that is not synced.

Each account has its own section in the main window’s source list; each account has its own set of folders and feeds.

Accounts will be added to Evergreen in Preferences, in an Accounts pane.

The policies of these accounts will be respected. For example, if the On My Mac account automatically marks articles as read after n days, it’s possible that some service does this after some other number of days, or never does that at all, and Evergreen won’t override those policies for that service.

Another example: the On My Mac account doesn’t support folders-within-folders — but if a feed service does support that, then the app will support it too, for that service.

(Current plan: Evergreen 1.0 will ship with Feedbin support, and Evergreen 2.0 will add support for more services.)

Let me know if any of the above sounds weird or if I missed anything. (There is contact info at the bottom of Evergreen’s main page.)

The Omni Show #6 with Liz Marley

This episode features Liz Marley, OmniGraffle engineer, formerly a tester and PM.

She talks about switching from testing to writing code — and about how she came to do technical talks using Swift playgrounds. Playgrounds with ducks, that is, because every good playground has ducks.

Evergreen Diary #5: Send to MarsEdit

The latest build of Evergreen, fresh from the lab, now adds MarsEdit to the sharing menu in the toolbar.

When you choose the command, it sends the current article to MarsEdit — which opens it in a new window, and then you can edit and add your own commentary before posting to your blog.

* * *

See SendToBlogEditorApp.m for the code that packages up the article and sends an Apple event.

The code is not MarsEdit-specific — other apps in the past have supported this same Apple event, though I don’t know if any other current apps do. If I find some, I’ll add them to the sharing menu alongside MarsEdit.

* * *

The Apple event code is written in Objective-C, even though I almost always write new code in Swift. But, with Apple events code, Objective-C is easier.

It’s easy to call from Swift: see SendToMarsEditCommand.swift.

(Reminder: this is all MIT-licensed. You can use this code.)

Evergreen Diary #4: Send to Micro.blog

The latest build of Evergreen adds Micro.blog to the Share menu in the toolbar, if you have the Micro.blog Mac app.

It sends the title and link of whatever you’re reading over to the Micro.blog Mac app, and you can edit it before actually posting.

This is hugely important. RSS readers exist not to just make reading easy but to make the web a conversation.

The next release will (probably) include MarsEdit, for the exact same reason. If you have ideas for other apps to include, let me know (see the bottom of the Evergreen site for contact info).

Ideally, every app like Micro.blog and MarsEdit would have a sharing extension — but, where they don’t, I can add them as long as they support some kind of way to do so.

Technical Details

Micro.blog supports a URL scheme for sending it a post: see See SendToMicroBlogCommand.​sendObject for the implementation in Evergreen. (And of course feel free to use any Evergreen code in your app.)

History

It’s not that we realized just today that posting from an RSS reader is important. Since posting was always an absolute requirement, way back in 2003 I released NetNewsWire 1.0 with an integrated blog editor.

I got that idea from Radio UserLand, which was an RSS reader and blog editor by the company where I worked before I wrote NetNewsWire. It’s Dave’s idea, not mine.

By the time I was working on NetNewsWire 2.0, it became apparent that smooshing a blog editor into a reader app meant that the blog editor suffers. The blog editor in NetNewsWire 1.x was pretty darn bad. So I had the idea of splitting out the blog editor to a separate app — inducing mitosis — which you can read about in my MarsEdit report of late 2004.

Mitosis was a success.

One of the keys of that success was that the API for contacting MarsEdit had to be open, and NetNewsWire had to support any blog editor, or any other app, that supported that API — because otherwise I would have been unfairly leveraging success with NetNewsWire on behalf of my new blog editor, and that would have been wrong.

MarsEdit still supports that interface, and my next step is to write the client side in Evergreen, so it can send to MarsEdit and any other app that supports that interface.

I am, by the way, delighted to be writing code for a 14-year-old API that still works.

Mentions: Proof-of-Concept

Ben Curtis posted feed_searcher to GitHub — it creates custom search feeds, and even has a handy Deploy to Heroku button. Cool.

(This is in reference to App Idea: Mentions).

App Idea: Inside Story

I had this idea around the time Apple came out with Bonjour (née Rendezvous), and all these years later I realize I’m never going to get around to it.

Here’s the scoop:

The idea is microblog posts (tweet-like) but that live inside a specific network only.

You run a Mac app that:

  • Lets you post short messages.
  • Shows short messages from people inside your network.

The app runs a small webserver that’s discoverable via Bonjour, that has a specific service name. That webserver publishes a feed of your posts.

The app also downloads a feed from every other app it finds (and caches those feeds, for when other apps are offline).

In other words, it’s instant office microblogging with no centralized server. Completely peer-to-peer.

The design is what you expect — a reverse-chronological list of posts with usernames and avatars. You can reply, mention people, mute people, etc. (I think you’d follow everybody on the network automatically — so muting becomes the thing, not following.)

* * *

Important thing: your posts are network-specific. So if you go to a coffee shop or the airport with your Mac, you won’t be publishing your at-the-office posts.

* * *

I haven’t done research into whether this is do-able on iOS. Is it possible to run a little webserver, in a backgrounded app, that other people on the local network could connect to? Seems unlikely — but I don’t know. Cool if you could.

* * *

I have no idea how you’d make money with this one. If you charge for it, not enough people would use it to make it worthwhile. (The standard dilemma of social networking apps.)

Maybe, though, you could add some IAP features? One might be to let people add RSS feeds, so it could also function as a lightweight RSS reader. A person might want posts from Daring Fireball and wherever else added to their timeline.

Or just add that feature anyway, and do the whole thing for fun.

PS “Inside Story” is a long name. Should be one word. Scoops? Bulletins? Officecast? Gossip? (“Gossip” was, if memory serves, the name of an app Chuck Shotton was working on a long time ago. Probably reusable at this point.)

(I mean, c’mon, office Gossip. It’s so perfect.)

App Idea: Mentions

“Hold on — I need to check my Mentions.”

Ten years ago or more we had several blog-specific search engines and services: Technorati, BlogBridge, and others.

One of the great things about these services was not just being able to search for something but being able to set up persistent searches: that is, you’d get a search as an RSS feed, and in your feed reader you’d get results from all over the place on the thing you’re searching for.

In the obvious and common cases, you’d set up searches for people linking to your blog, writing about the apps you work on, mentioning the place where you work, and mentioning you.

I’d like to see something like this for today, but where the scope is just the Apple/Mac/iOS community. It would crawl the obvious sites (such as Daring Fireball and Loop Insight) and it would crawl the many blogs and microblogs that make up the community.

I think this is best as a web service, though it could have Mac and/or iOS client apps. It needs to provide feeds for people using feed readers.

* * *

With that reduced scope, and with the better tools and cheaper cloud computing these days, I don’t think it would be terribly expensive. Not like it was 15 years ago.

You might want to make money with this, though, and there are a few ways to do it: charge extra for things like notifications and email alerts. Charge money for client apps. Make the first three search feeds free, and charge money for a ten-pack of searches. Etc.

* * *

An alternate version of this idea isn’t a web service — it’s a Mac and/or iOS app that does the crawling. A kind of specialized Mac/iOS feed reader.

The list of feeds-to-crawl would probably just be an OPML file on the web, and the app would periodically grab that list.

(If you did this, you could use RSParser as starter code, since it parses OPML, RSS, Atom, JSON Feed, and RSS-in-JSON.)

This would make a great open source project, but it could just as well be free or for-pay.

* * *

One challenge is handling spam and abusive or hateful content. You’d most likely want to have a suggest-a-site form, so you don’t have to go out and find every single site there is.

But you have to be able to say no, and you have to be able to update the list of sites-to-crawl quickly if a site turns bad or inappropriate somehow.

You’d also need a way for users to report an issue.

(Again, the reduced scope — Apple-related blogs — makes this somewhat easier than if you tried to do the entire blogosphere.)

* * *

I want this! I’d use it every day. I know other people who would too.

But I’ve got enough on my plate that there’s no way I can do it myself — though I’d be happy to answer questions and provide feedback to anyone who does want to do it.

Evergreen Diary #3: On Punting

I have a vision for 1.0, and then, as time goes on, I have to cut it back beyond where it hurts. I have to keep punting. I hate this part of shipping software.

But I also love it because it reminds me that I have the stomach for it. Shipping software is an emotional skill.

After shipping — no matter what — there will be people who absolutely cannot believe that feature X wasn’t included. In fact, it’s the one thing they totally need.

And they’re right. Not wrong. And I would have loved to have included feature X, loved I not quality (the app maker’s honor) more.

* * *

I wondered if Evergreen as open source instead of as a for-pay app would affect what and how much I punt. I hope the answer is: not at all; the decisions would have been the same.

I think that’s right because I’m trying to make as good an app as I can, which would be true regardless.

But it is nice not to have to consider money as I make decisions. I do think about how I want it to have as many users as possible, and I want people to love the app. And I have to think about the economy of my time. But I don’t have to consider money.

Another difference between open source and commercial is that I can be utterly transparent about what’s been punted. Check out Evergreen’s 2.0 milestone. It will keep growing. Every single one of those things was originally supposed to go in 1.0.

* * *

Some things I keep reminding myself as I make progress toward 1.0…

This isn’t the last release, it’s the first. There will be many more. (I hope to work on this app for 20 years. It’s been only three years so far.)

Any feature I ship, I probably have to support forever. So it’s wise to be cautious.

Quality — design, stability, performance, lack of bugs — is way more important than any collection of features.

And, perhaps most importantly: shipping 1.0 means learning about your users, how they use your app, and what they need — and those lessons will change future plans, as they should. It’s best to learn those lessons early, before doing too much.

The Omni Show #5 with Mark Boszko

In this episode I finally get around to talking to Mark Boszko, the show’s intrepid producer and The Omni Group’s Video Producer.

I’ve known Mark for more than ten years — we met at a SXSW conference many years ago, long before either of us came to Omni. Mark didn’t even live in Seattle in those days. Now he does, and now we have the pleasure of working together on a podcast.

PS If you like movies, definitely check out the podcast Mark hosts: The Optical.

2018: Some Hope

Mike Monteiro writes of Twitter CEO Jack Dorsey:

Jack let it happen. He watched as a once-entertaining, once-illuminating, once-vital network to global communication became a garbage fire of hate. He did nothing to stop it. Or curb it. He didn’t see a problem.

Our current crises of democracy and good faith did not just blow in with the wind and transform the air without our knowledge or consent.

These crises were made by people, and we knew what they were doing, and we agreed to this.

Jack Dorsey is one of those many people. Just one. But one with a kind of power that nobody in the world should have: the power to directly control a vast amount of the world’s communication.

It’s not that Dorsey failed to consider the good of the world. Or, really, it’s not just that. It’s that this kind of power should not exist at all.

But we agreed to it. We’re still agreeing to it.

Twitter — and Facebook, and the power of tech companies — is not our only problem.

But I have no doubt that had Twitter not become a loving home for hate, Trump would not be President now. In that universe we’d still have big problems, yes, but not like this.

How we can stop agreeing to this

The great social network is, or ought to be, the web itself.

The unruly web — unregulated and uncontrolled — is, perhaps paradoxically, the easiest place to limit hate. Not because we can stop people from publishing, but because we don’t have to live by Dorsey’s and Zuckerberg’s rules and designs.

I don’t know all the details of how we get there, or what it will be like once we do. That’s fine: that’s part of what makes the journey fun.

Software

Consider a few apps.

Overcast and Castro and others help ensure that podcasting is not just a vital and exciting medium of independent publishing but is also open and built on standards. Anybody can write any kind of podcasting software they want to — but nobody can control podcasting.

And nobody can force you to listen to hate. You pick the shows you want to hear.

MarsEdit lets you write whatever you want to write and publish it on the web. You’re limited only by the law and whatever terms of service your hosting provider may have.

All the words you read in MarsEdit are your own. And nobody can make you read what other MarsEdit users write.

Evergreen (which I’m working on), NetNewsWire, Reeder, Unread and other RSS readers work like Overcast and Castro but for written words. You choose what to read, and if a blogger you like suddenly turns hateful, you hit the Delete key.

Then there’s Manton’s new service, which needs its own section…

Micro.blog

It’s a publishing platform and a social network, based on standards.

You don’t even have to use Manton’s Mac or iOS apps: you can write posts in MarsEdit or other blog editor, or read your timeline using an RSS reader.

People could, though, sign up for it and flood your mentions with hate. In theory. So I asked Manton about that, and he wrote:

Micro.blog is similar to MarsEdit in a way in that it can be used to write hateful posts, etc. What we have to do as a social network is limit the damage. So, by default, if someone writes something terrible… No one sees it. It doesn’t automatically show up in trends (because we don’t have them, for this reason) and it doesn’t show up in Discover (because that’s curated by a human). Replies are where it’s an issue, and that’s where good tools and automatic flagging and reporting are needed.

In other words: the rules are different, and the easy exploits on Twitter are not so easy on Micro.blog. And there’s an actual committment to fighting this.

Manton also writes on his blog:

Imagine instead a service based on blogs, where the internal posts on the platform were the same format as the external posts. The curators of the platform would have more freedom to block harassing posts and ban nazis because those problematic users could always retreat to their own web site and leave everyone else in the community alone.

That’s how the web is supposed to work. It’s a core principle of Micro.blog.

(I’m @brentsimmons on Micro.blog, by the way. Here’s my microblog. I plan to post there more often than on Twitter in 2018.)

I should also mention…

Slack

I’m not sure where to put Slack in all this — except to say that admins control who’s in their groups, and I’ve never seen hate and harrassment there. I run a few groups, and I would have zero tolerance for this, and so would everybody I know who runs groups.

Well. One more thing about Slack: it meets some of the needs that Twitter used to meet. The talking-with-friends-and-family needs are very well covered there.

And the more we find ways outside of Twitter and Facebook to meet those needs, the less we’ll use Twitter and Facebook.

Maybe, back in 2012, Twitter did five important things for you that only Twitter did. I bet, in 2018, that that’s down to one or two.

Fun

The period from 1995-2008 (roughly speaking) was fun. It seemed like everybody was coming up with new things, and people were experimenting, and we were finding new joys in new connections, both human and technological.

Then, as Facebook and Twitter (and Google Reader; can’t forget that thing) grew, it’s as if we froze.

And those things were fun for a while, but they’re not now, and it’s obvious we made a mistake in allowing that much power to concentrate.

It’s time for the thaw: it’s time to get back to having fun. You’re free to make whatever you want.

What I’m Not Saying

Rebuilding the social open web is not the one cure that we need for all our ills. I’m fully skeptical of technological solutions to problems of culture and politics.

But it is an important thing we can and should do.

My small hope for 2018 is the knowledge that I’m not the only person thinking that way.

Related Reading

Me in 2011: What we talk about when we talk about RSS

Me in 2013: Why I love RSS and You Do Too

Anil Dash in 2012: The Web We Lost and Rebuilding the Web We Lost

IndieWeb is a thing I need to learn more about.

Tantek Çelik (video) - The once and future IndieWeb

Archive