Responding to yesterday’s post, a number of people pointed out to me that it sounded weird if one minute I suggest checking out Azure Mobile Services and the next minute tell developers they should take control of their app’s web services.
I meant no contradiction, but I could have explained better.
When creating web services, you should consider high-level systems, low-level systems, and everything in between, and figure out what makes sense for you.
Here are some — not necessarily all — of the things to consider when choosing:
Does it support iOS, Android, and browser-based apps? (Knowing that you may — may — want to move beyond just iOS.)
Can you create a social component?
Can you add additional services — push notifications, feed-polling, sending email, whatever — to the system?
Can you get aggregate data and learn how people use your app?
Perhaps most importantly: is it possible to migrate to something else (even if takes some work)?
The answer is yes to all of these for Mobile Services. (While the answer is no to all of them for iCloud syncing.)
The answer is also yes if you want to work at a low level — it’s yes if you get a virtual server on Linode and run Ruby on Rails and MySQL.
At some point you have to outsource some things, right? You’re not going to build your own server machine that runs your own operating system in a data center you constructed with hammers and saws that you made.
So you choose what makes sense. And if it can be a high-level system, that’s cool — it will probably save you time and be easier to maintain. But you might have good reasons to choose something medium or low level, and that’s cool too.
The key is this: you need control of the data.
iCloud Core Data syncing is, once again, completely opaque and outside your control. It fits none of the criteria listed above.
Yesterday the Verge posted Apple’s broken promise: why doesn’t iCloud ‘just work’?
Here’s the thing: if you’re a developer, you shouldn’t use iCloud syncing anyway. I’ll explain.
Android and the web
You may think you’ll never want an Android or browser-based version of your app. But are you sure? Really, really sure?
You hope your app will be a hit. (If not, then quit writing it and choose something else.) If it’s a hit on iOS, it could be a hit on Android too — and you can bet that customers will ask for a web app version.
You don’t want to limit the success of your app just because you didn’t want to write your own server.
And even if you’re sure that you’ll never want an Android or web version, is it possible you’d want a Mac version that isn’t sold on the App Store? (Only App Store builds are allowed to do iCloud syncing.)
We’ve been living in a social world for years. But iCloud syncing is not social: it’s per-user syncing.
If you write your own server, you can write the social bits, so your users can share recipes, weather forecasts (look, Mom, it’s going to be sunny on Thursday!), favorite articles, or whatever-it-is your app does.
People expect social.
If you’re writing an RSS reader, you can’t ask iCloud to download feeds.
There are all kinds of services that make sense on the server side. You could do some of them on a client, but at the expense of timeliness and battery life. If it’s a good idea, and you don’t do it on a server, your competition just has to write a server that does it, and your app is finished.
Learning how people use your app
You shouldn’t look at private data.
With Glassboard we made the decision — there was no debate — to encrypt messages in the database, so that we couldn’t see private data.
But we could still look at aggregate data. It was interesting to know how many boards were created each day, how often people used invitation codes, and so on.
There’s no substitute for learning about how people use your app. You can guess how people use your app. You can — and should — get all the feedback you can via email, Twitter, and App Store reviews.
But seeing actual data makes a real difference, because it helps you figure out where your resources need to go.
It used to be expensive to develop, run, and maintain your own server. You’d buy a machine or a few machines, get them installed in a data center, figure out how Apache works, install MySQL, and write a ton of scripts. Perl or PHP scripts, most likely. (Ugh.)
You’d use Subversion (if you were lucky) or cvs. You’d write your own testing system from scratch. You’d write a bash script that copied the files up to the server.
And you’d spend a bunch of money.
Everything has gotten easier and cheaper. These days you’d run services on Amazon, Azure, Engine Yard, or Heroku. Or get a virtual server on Linode. You’d choose from one of the many excellent systems like Ruby on Rails, Sinatra, Node.js, and Django. You’d deploy via git or Mercurial.
If you can learn Cocoa, you can learn this stuff. (And so much of it is wonderful — you’ll enjoy learning it.)
Tim Wood, CTO of The Omni Group, tweeted the very wise words: Own the Wheel.
Here’s the thing: half the mobile revolution is about designing and building apps for smartphones and tablets.
The other half is about writing the web services that power those apps.
How comfortable are you with outsourcing half your app to another company? The answer should be: not at all comfortable.
Do it yourself.
My Mom has the scoop on goings-on in Newfield, a small town in south Jersey where my family comes from, where my Mom lives, where my grandfather was chief of the volunteer fire department.
The city council wanted — for stupid reasons — to shut down the fire department, which had been serving the town for more than 100 years. The council voted to shut it down before changing its mind. Narrowly.
All I can think about is that this mess marks the passing of the two important groups - those responsible for the original Borough bargain of lower taxes thru volunteerism and the Great Generation that Ever Was. Political/social decision making has passed into the hands of the baby boomers and we are not prioritizing the welfare of the town, state, nation, world above our own petty egos. This is not the legacy past generations wanted us to embrace.
Black Pixel writes, in The Return of NetNewsWire, that future versions will have syncing, despite the end of Google Reader.
They’ve been working on new versions, including modern designs and new features.
Sync has been delayed because they were trying to get iCloud Core Data sync working, and it’s just too buggy.
Since Brian worked on the server side, his list of Stupid Feed Tricks didn’t include my very favorite feed screw-up.
A lot of hotels and similar offer wi-fi. When you open a page in your browser, it redirects you to their login page.
Those systems don’t differentiate between an http request made by a feed reader and an http request made by a browser. (Nor should they.)
But what happens is that you launch your reader and it gets a redirect for every single feed, to some kind of URL like
Normally that would be fine. It’s just a redirect, and once you have actually logged in you can do a refresh in your reader and all’s well.
A bunch of these dumb systems redirect using a permanent redirect: they use 301 instead of 302.
When a feed reader gets a permanent redirect, it’s supposed to take that to mean: “Hey, the feed moved. It’s over here now. Save the new URL and use the new one from now on.”
And if you don’t do that in your reader, and your feed reader is popular enough, smart people who quite rightly care about proper behavior will call you out. You have to do that.
So you write your feed reader to do the right thing — and then one of your customers goes to a dumb hotel, opens their laptop, and their subscription list is wiped out. Every feed URL is replaced with
http://dumbhotel.com/register.aspx. And now they can’t get their news, and they don’t know how to get it back.
At NewsGator and Sepia Labs I worked with Brian Reischl, one of the server-side guys. Among other things, he worked on NewsGator’s RSS content service, which reads n million feeds once an hour.
(I don’t know if I can say what n is. It surprised me when I heard it. The system is still running, by the way.)
Brian is intimately acquainted with the different ways feeds can be screwed up. So he posted Stupid Feed Tricks on Google Docs.
I quote the entire thing below for people like me who don’t have Google accounts. The below is all by Brian:
Stupid HTTP Tricks
- When the feed is gone/errored, publisher may still return a 200 OK but send an HTML page instead.
- Using permanent redirects for temporary errors. In one instance, all the Microsoft blogs had a temporary system error. All the feeds did a permanent redirect to the same system error page, and we updated all 40,000 feeds to point to that one URL. Whoops.
- Using very slow or overloaded servers. It might take 60 seconds just to connect and send the request, another 60 seconds to first response byte, and so on. This can bog down your content retrieval.
- Very slow responses, or responses that never actually complete (ie, you hang trying to read data essentially forever)
- Infinitely long responses. eg, feed server has an error and prints an error message in a infinite loop until something stops it. Hopefully it’s stopped by a check in your system, rather than consuming all the memory on your server.
- Sending back things that are not XML (eg, videos). It can help to check Content-Type and Content-Length headers, but sometimes they misidentify RSS as something else (eg, text/plain).
- Infinite redirect loops.
- Long (but non-infinite) chains of redirects.
- Responding with 304 Not Modified if you send any If-None-Match/If-Modified-Since header, even if the feed has changed.
- Throttling your IP address. Some don’t tell you they’re throttling. Some provide Retry-After headers, but the HTTP status code can vary. eg, Twitter used to use their cutesy “420 Enhance Your Calm” response, then switched to “500 Internal Server Error”. Some use “503 Unavailable”. You’re mostly covered if you look for Retry-After headers in every non-success response..
- Redirecting (perhaps permanently) to a URL that’s already in your system. So now you either have a duplicate feed, or you have to update clients somehow. Note this can sometimes be legitimate, eg consolidating multiple feeds into one.
Stupid XML Tricks
- Any sort of XML well-formedness error you can think of. Missing closing tags, mismatched tags, bad escaping, not quoting attributes, missing root elements.
- Including unescaped HTML content inside a tag - which sort of works, except that most HTML isn’t XML-compliant.
- Putting in characters that are illegal in XML documents (eg, some non-printable characters that should be escaped, but aren’t)
- Declaring the document as ISO-8859-1 encoding, but actually using UTF-8, with some Arabic characters in it.
Stupid RSS/Atom Tricks
- Missing any element you can think of.
- Adding custom elements without namespaces.
- Using common extension elements without defining the namespaces (eg, using the common “mrss” namespace prefix for MediaRSS elements, without actually specifying that namespace anywhere)
- Not providing a GUID.
- Providing the same GUID for every post in the feed (eg, using the feed URL as the GUID)
- Providing the same GUID for every post, but changing each time you request the feed (eg, using the current date/time)
- Using a different GUID for each post, changing each time you request the feed (eg, generating an actual GUID each time the feed is requested)
- Not giving a PubDate
- Changing the PubDate on every retrieval.
- Changing the PubDate when a post is edited, rather than using a lastUpdated tag.
- Putting a tiny number of posts in the feed (sometimes just one). These types then usually publish 10 articles in the space of two minutes, and wonder why you’re missing 9 of them.
- Putting only one post in the feed, with a GUID that never changes. When there are new posts, just the title and description change. (I believe this was a bunch of Japanese newspaper sites.)
- Updating post content without changing the lastUpdated date (or not having one)
- Updating post metadata (eg, enclosures, MediaRSS extensions, etc) with or without changing the lastUpdated date.
- Treating their feed as append-only, so over time the feed grows without bound. eg, each request might pull back 10,000 posts covering the entire 8 year history of the feed.
- Specifying dates in whatever their language’s “Date.ToString()” spits out. eg, “Tuesday, March 31st, Year Of Our Lord Two Thousand And Twelve, 4:59 PM”
- Not specifying timezones for dates (very common. It’s easy to just assume UTC, but note that can yield pubdates in the future).
- Specifying dates that are far in the past or future (anything up to thousands of years)
- Having the Link element point to another site. This is actually pretty common (eg, DaringFireball). This can be a problem depending on how you’re identifying individual posts, or if you’re trying to detect duplicates across feeds.
Other Stupid Tricks
- Updating posts very frequently. Newspapers are very fond of this. In 4 hours they might change a post 12 times, by the end it might have nothing in common with the original article (completely different title, completely different body). Sometimes combined with not using lastUpdated, or just not changing lastUpdate.
- Publishing updated posts as new posts, so you have 12 versions of the same post in the feed.
- Occasionally giving you an two-week-old version of the feed for one or two requests. It looked like one server in a cluster had cached an old version and wasn’t updating it. (This was the New York Times back in ~2009. They might’ve fixed it by now.)
- Adding posts very quickly. This is very common with feeds like the StockTwits stream, Twitter feeds (when that was allowed), the “all news” feeds from news organizations, etc. If you only check the feed every 60 minutes, you could easily miss something.
- Changing content literally every time you get the feed. eg, a feed that returns the current time in all the timezones, or the current weather for 20 different cities.
- Putting out private data without requiring authorization of any sort. eg, a feed of all your GMail. This isn’t a problem until you provide search or other feed discoverability, and then people’s private data starts showing up. Then they get very angry.
- Some places will publish a feed and then get angry that you use it, especially if you have ads in your reader. (name redacted before I get sued) got very bent out of shape over that back in ~2007.
- Providing feeds, but then also using robots.txt to say you can’t crawl it. So now do you violate the robots.txt, or not let your users subscribe to feeds because the publisher is a dipshit?
- Providing valid, but limited interest feeds. eg, search feeds (couches for sale in Portland on Craigslist!). Also lots of custom things like combinations from Yahoo Pipes (or whatever equivalent people come up with), bookmark/favorite feeds, etc. Can lead to lots of duplicate (or near duplicate) posts, and lots of feed retrievals that very few people care about.
- Publishers will routinely have 2-4 copies of the exact same feed. eg, one sourced from their site, and another republished through FeedBurner. Note: FeedBurner includes extension elements that tell you what the source feed and post were.
- You should think hard about canonicalization of URLs. Some parts of the URL can be case-sensitive (path and query) other parts can’t (protocol, host and post). Users (and webmasters) will absolutely use different upper/lower casing in different places.
- If you build a database index on FeedUrl, consider that 99% of them start with “http://”, which makes for a shitty index. Consider separating the protocol into its own column, and then indexing on the remainder of the URL. Alternatively, you could index on a hashed value of the URL. Theoretically you could have collisions, but in practice there are not that many feeds.
I created a new mailing list for people who want to talk about the technical side of RSS syncing.
Ideally there would be a single standard for RSS syncing, and clients could choose among systems. But I don’t insist on that — at least not yet. I think it’s ambitious enough just to get some working sync systems up and running.
While the demise of Google Reader got all the attention, there are some other things going away in A second spring of cleaning. It’s worth reading.
Google certainly has the right to shut things down — and I think they’re right to do so, also. The company’s been all over the place and should continue to work on focus.
But I don’t like when they turn off existing support for an open standard like CalDAV in favor of a Google-only API. Sure, they can — it’s within their rights, obviously. But I don’t like it.
The main thing we learned is that subscription needs to be centralized to make the process as simple as possible for the user. That’s one of the main reasons Twitter was such an effective competitor.
There isn’t really any such thing as Google Reader API documentation.
But I do have a PDF document that Nick Bradbury wrote up when we were switching FeedDemon and NetNewsWire from NewsGator syncing to Google Reader.
It’s from 2009. Parts of it are surely out-of-date (authentication, for sure), and it was written with the expectation that the reader is familiar with NewsGator syncing.
Nevertheless, in the spirit of Marco’s baby steps toward replacing Google Reader, I present it (with Nick’s permission) in case it’s useful to anyone:
Here’s something website owners should realize:
You might look at your logs and find that Google Reader reports 100,000 subscribers. (For instance.)
And then you look and find that clients apps X, Y, and Z don’t seem to hit your feed all that often.
The easy conclusion is that Google Reader has thousands more users than X, Y, and Z. But that wouldn’t necessarily be correct.
If X uses Google Reader for syncing, then X doesn’t read your feed directly. It gets it from Google Reader, and that person is reported to you under the Google Reader subscription count.
So it’s possible that half — or whatever — of those Google Reader subscribers are using X. (Or Y or Z.)
Or not. Or more than half.
You just have no way of knowing. But it’s a safe bet that more people are using client apps than your logs report.
Even if you don’t use an RSS reader, you still use RSS.
If you subscribe to any podcasts, you use RSS. Flipboard and Twitter are RSS readers, even if it’s not obvious and they do other things besides.
Lots of apps on the various app stores use RSS in at least some way. They just don’t tell you — because why should they?
RSS is used for mundane things too, like Mac app updates (for non-App-Store apps) and Xcode documentation.
And those people you follow on Twitter who post interesting links? They often get those links from their RSS reader.
One way or another, directly or indirectly, you use RSS. Without RSS all we’d have is pictures of cats and breakfast.
RSS is plumbing. It’s used all over the place but you don’t notice it. Which is cool.
But here’s why it’s great plumbing:
There are many millions of feeds, from the smallest blog to the many feeds at the New York Times. Just about everything that gets published on the web is available via RSS. (Outside of Twitter and Facebook.)
There are no user caps. No company can tell your favorite app how many users it can have. (Twitter does this.)
Nobody can tell you how to display an article from an RSS feed. (Twitter does this with tweets.)
The formats are stable. Code I wrote five years ago to parse feeds would work today and will work in five years. (The formats are simple, too.) Other services have APIs that change and break existing apps.
RSS can’t be shut down. Any number of companies can go out of business, but nobody can stop anybody from publishing and reading RSS feeds.
Nobody can force ads on you. A given RSS reader could add ads, but you can switch — because another RSS reader can read the same feeds. A given publisher could put ads in their own feeds, but you can unsubscribe. There is no company that can force ads on everyone, as Twitter and Facebook are working on for their systems.
Nobody can force you to be tracked. If you’re not using a syncing system, then nobody knows what you subscribe to and what you read.
You don’t need to register anywhere to write an RSS app. (You do need to register to write Facebook and Twitter apps.)
In the general case there are no security issues with feed reading. (Unless you’re using a sync service or reading authenticated feeds.)
This is elegance. It derives from the design of the internet and the web and its many open standards — designed so that no entity can control it, so that it survives stupidity and greed when it appears.
Lots of things work like this. Not just RSS.
A naive reading of the above makes it sound like RSS is anti-business. That’s not true at all. (I did well with my RSS business.)
Instead, it’s anti-monopolist. By design it creates a level playing field. Anybody can write RSS apps, and anybody can use RSS however they want to.
This means that competition and innovation are permitted to thrive.
But it’s not a guarantee. In the past several years it seems to have slowed way down.
Prague 1948 Forever
When Eastern Europe opened up, following the fall of the Berlin Wall and the collapse of the Soviet Union, Prague looked like it had been sealed up in a bubble since 1948.
Google Reader isn’t communist Russia, obviously, duh — but it’s a similar pattern. There was one gigantic player and a bunch of satellites, and RSS readers more-or-less looked like it was still 2006.
Not that there wasn’t any innovation — there was some — but it’s been pretty quiet, especially compared to the several years before 2006.
RSS the format has remained as useful and cool as ever, but RSS readers haven’t done so well.
My hope — my expectation, even — is that a few things will turn this around:
The end of Google Reader takes away that one dominant player. The market for RSS readers is no longer frozen — and it will interest more developers than it has in recent years.
Over-reach by Twitter and its diminishing user experience makes people interested in other ways of finding good stuff to read.
The lower costs of server-side development and deployment brings creating RSS services within reach of smaller companies.
The challenge — as ever, with everything — is to make useful and delightful apps that people love.
But now, if I’m right, we’ll have more people working on that challenge.
In the meantime, the loss of Google Reader syncing is going to be tough. That’s a big hurdle. Marco proposes some baby steps. I don’t like Google Reader’s (undocumented) API, but I like the pragmatic approach.
At any rate — these are interesting times! I know that’s a curse, but I take it as a blessing, because it’s way more fun that way.
My friend and former co-worker Nick Bradbury writes of The End of FeedDemon:
That was the nail in the coffin for me. I hate to say goodbye to FeedDemon after a decade of working on it, but it’s time to say goodbye. When Google Reader shuts down on July 1, FeedDemon will also disappear.
Before Nick and I were co-workers, we were both indie developers working on RSS readers. (Check out this Economist article from 2004 that mentions us both.) We weren’t competitors, since FeedDemon ran on Windows and NetNewsWire ran on Macs.
NetNewsWire shipped first — but FeedDemon shipped features faster than I did, and I shamelessly stole some good bits. Which Nick didn’t mind. (He encouraged me, even.) We became friends, and later became co-workers.
It makes me sad to read of the end of FeedDemon. It mattered to me. But, more than that, I’m sad for my friend.
By implementing a suitable syncing API for RSS, and implementing a reasonably useful web interface, Black Pixel could establish NetNewsWire Cloud as the de facto replacement for Google Reader. Charging a reasonable fee for this service would likely inoculate it from the risk of sudden termination, and it would doubly serve to provide the very service that NetNewsWire needs to thrive on the desktop and on iOS.
Here’s the announcement on the official Google Reader blog.
I called it in my post last month: RSS Sync Apocalypse Preview.
And I’ve written about Google Reader in Google Reader and Mac/iOS RSS readers that sync in late 2011.
I think this is a good thing for RSS and RSS readers in general — but it’s sure going to hurt RSS reader customers unless people come up with a sync solution quickly.
My preference: somebody would write a sync service — and charge a monthly subscription — and allow any RSS reader to use it.
Other things I’ve written about RSS syncing:
- Idea for alternative RSS syncing system
- Why “Just Store the App Data on Dropbox” won’t work for RSS readers
- How about IMAP for RSS syncing?
- On the benefits of thin-server RSS syncing
Part of my reason for talking about this so often over the years was to encourage people to build a new sync system. Now you don’t need my encouragement. :) But if I can advise in any way, get in touch with me.
In the latest episode of Identical Cousins, Michael explains how he changed his mind about putting a date badge on the Fantastical for iPhone icon.
Then we talk about a bunch of other things. It’s a grab-bag, a potpourri, une mélange.
My favorite part of the Letterpress 1.4 change notes:
App binary is even smaller. If the world were just, the market would reward my relentless drive for efficiency fueled by a visceral hatred of bloated software. #gimmemoney #jk #no4real
(Letterpress is the one game I play. I play mostly with my family. We all love word games.)
Yesterday I tried using Cocoa auto layout in my latest project. I’d tried it before — and gotten quickly frustrated.
The difference this time is that I didn’t use Interface Builder. I set up the constraints in code, using ASCII art layout strings.
And it worked quite nicely.
I’m an accomplished CGRect-slinger. I’ve been doing layout in
layoutSubviews and the Mac equivalent for more than a decade.
I can do it in my sleep. Which is good, because writing layout code makes me sleepy. It’s one of the most tedious bits of programming — not at all interesting or fun.
The thing about CGRect-slinging is that you have two steps:
- Figure out the rules.
- Translate the rules into code that sets the frames of your views.
I like auto layout because it’s more like one-and-a-half steps: figure out the rules and then type up the rules.
You still have to translate, but it’s a shorter distance. Like going from English to French — instead of going from English to the tentacle-speech of Alpha Centaurans. (Those cats have more than 49,000 words for methane snow.)
Punching and Cursing
The problem with IB and auto layout is that IB adds constraints as you’re working. It won’t let you have ambiguous constraints.
The thing is, they’re bound to be ambiguous as you’re working on them. So IB will get in your way, and you end up fighting it and calling it dirty names you didn’t know you knew. Or at least that’s what I do.
Keeping the Dream of the ’90s Alive
I always thought that Interface Builder was the product of a beautiful idea: coders would write code, and designers would create interfaces using a drag-and-drop design app.
But — in my my experience, and based on everything I’ve heard — designers use Photoshop, Acorn, Keynote, OmniGraffle, and so on instead. IB shows you at best an approximation of what the app will look like — while designers want to see exactly what the app will look like.
Sure, a designer could work in an image editor and then work in Interface Builder. But it’s a rare designer who does. Instead they create mockups and movies and let the programmer do the Xcode work. (Yes, there are designers who code, even. I know. But this is still generally true.)
I had thought, when Apple moved IB into Xcode, that it was tacitly acknowledging this. But then storyboards made me wonder if Apple is still keeping the beautiful dream alive.
Life in Programmer-Land
Despite the dream, IB is very much a part of programmer-land. It’s not designer-land.
So the question becomes: as a programmer, do you find it useful?
I’ve found that I use it less and less. I use it almost solely for very static screens.
It may make it easier to set things up at first — but that’s like using protomatter in the initial matrix, which history tells us is a poor call. (It’s a shortcut that leads invariably to pain.)
Here are some of the problems I run into when using IB:
• Navigating back and forth between code and a xib file is a pain, even on my big screen. (To go through all that just to change a background color feels ridiculous.)
• Diffing versions of a .m file is nice. Diffing versions of a xib file is terrible.
• Pointing-and-clicking and dragging-and-dropping is not how a programmer gets through their day. It’s an unwelcome context switch.
Any one of those would be enough: all three thoroughly convince me to stay away from IB for all but the rarest cases.
And the fourth reason I added just yesterday is that dealing with auto layout in IB is not worth the hassle — not when it’s so much easier done in code.
But more code is more code, right?
Eschewing IB does mean writing code to create and configure your views. And that is, no doubt about it, more code.
But I look at it this way: it’s fewer entities. For each line of code there would have been a corresponding thing in IB.
And fixing a thing in code is simpler than fixing it in IB — because as programmers we’re optimized for code-writing.
Want to fix a color or change the position of a view? I like doing it the way I solve every other problem: I code it.
I recorded some videos for Azure Mobile Services.
I got paid for doing the videos, of course — that’s how these things work. But I wouldn’t have done them if I didn’t think Mobile Services is cool and useful. It is.
(I’ve been filmed while on stage before — but this is the first time I’ve talked directly to the camera. Cameras are cold. It was weird! But fun. I’d do videos again.)