inessential by Brent Simmons

April 2014

Manton on Decentralization

Manton Reece, Write locally, mirror globally:

Decentralization is the internet’s greatest strength and weakness. There shouldn’t be one service to hold all of blogging; each writer should have his or her own domain and web site. But web sites also die all the time from neglect. We need centralized services to index and syndicate content so that it’s preserved and accessible to more people.

What the Plans Actually Were

This was a long time ago, and it wasn’t necessarily the best plan — but it was the plan.

The last version of NetNewsWire I shipped was NetNewsWire Lite 4.0 for Macintosh. The next step would have been the full (non-Lite) 4.0 for Macintosh, but we sold NetNewsWire to Black Pixel. And Black Pixel’s plans are most likely different from my plans, which is expected and totally a-okay.

Aside from general modernization and performance and memory use enhancements, my big plan was plugins.

The idea was pretty simple: make NetNewsWire extendible by other developers. There were already some things developers could do:

  • Script the app. (It supports AppleScript.)
  • Create custom themes (with HTML, CSS, and JavaScript).
  • Create scripted feeds (using AppleScript, Ruby, Python, etc.).

Those were cool features. Not everybody used them, but the people who did use them really liked them. Since NetNewsWire was a productivity-style RSS reader rather than a casual reader, it made sense to add these kinds of features. They were out of the way of the average user, and added a bunch of value for power users.

Along the way I discovered that there was a marketing benefit too. People liked to publish their scripts and themes. And then I’d link to those from ranchero.com. This meant more publicity for the app and it meant people could easily find these extensions that helped them get more out of the app.

This was especially true with themes — because they’re fun to make and fun to try out.

For NetNewsWire 4.0 I wanted to take this to the next level and provide additional ways to extend the app. I got pretty far along in defining some of the plugin types, to the extent that the app was using the plugin API under-the-hood.

Below are the types I was considering. They would all have had Objective-C APIs. And they’d use protocols, so the plugins wouldn’t have had to link to anything.

Sharing Plugins

With the explosion of new places to send your content (Twitter, Pinboard, Pocket, and on and on) I realized it would be difficult to keep up and cover everything.

A sharing plugin allowed you to create a user interface (if needed) and to send a URL and text wherever you wanted.

It also would have allowed me to ship plugins outside the normal release process. I could ship a sharing plugin that wouldn’t be included in the standard release, or ship one that could be included later.

These were super-easy to write. Enough was given to you for free that you didn’t have to do much more than configuration plus an http call. (For most cases. And if you needed to do more, you could.)

Account Type Plugins

The app would have handled Google Reader and stand-alone feeds itself — but a developer would have been able to add additional types of accounts.

I should describe the planned source list change. Taking a cue from Mail, it would have had zero or more top-level “Google Reader (account name)” folders and one “On My Mac” folder.

Remember that this was 2011, and Google Reader was all most people thought of when they thought of RSS readers. (It would also have had separate sections for smart feeds and scripted feeds. It would have been a more structured source list than in previous versions.)

The idea is that you might have some synced feeds and some non-synced (“On My Mac”), for whatever reason. You might even have multiple Google Reader accounts. (Work and personal, for instance.)

And it was likely that other types of accounts could exist. They might not even be RSS accounts. If a developer wanted to add a way to read Twitter or Facebook, or anything else, that would have been do-able.

While the API for sharing plugins was all the way worked-out, this API was only partly worked out at the time of the sale. But I didn’t foresee any stumbling blocks.

The idea was to make NetNewsWire an app for all the various streams of new items.

The native account types — Google Reader and On My Mac — would have used the API. In addition I was planning to support OPML reading lists, and that would have used this API too. (OPML reading lists are cool. Instead of subscribing to a feed, you subscribe to an OPML file on the web, and then your reader reads all the feeds in that file.)

JSTalk

Though the above were Objective-C APIs, I wanted to make it possible to use JSTalk for those APIs. I also wanted to support JSTalk in NetNewsWire’s scripts menu.

I considered adding a plugin API for adding other embedded languages. A developer could have added Lua support, for instance. I was on the fence about this one, as it could have meant me going down a rabbit hole.

Non-plugin thing

Besides plugins, the other big feature for NetNewsWire 4.0 was the return of the Sites Drawer.

The Sites Drawer, you may recall, was a big outline of feeds you could subscribe to. I had removed it on the theory that it had become pretty easy to find RSS feeds.

Big mistake. Discovering feeds was still hard, and there’s a definite value to making that easy inside the app.

It wouldn’t have been a drawer this time. It would have been a window (or a view that swaps in to the main view) with a cool UI for finding feeds based on categories. A view with artwork and everything, something great to look at and fun to use.

Unlike the original Sites Drawer — which was built from a plist that shipped with the app — it would have talked to a web service which would have been updated with new feeds regularly.

There would have been a way to suggest a feed or upload your own feeds (anonymously) to the system so that I could look at them and pick out the ones that might interest NetNewsWire users.

Think of something like an App Store for RSS, only designed to fit into NetNewsWire.

(There’s an additional marketing benefit, by the way — we found in the past that when we added someone’s blog, they blogged about it. More publicity for the app. That’s not the reason to do this, but it didn’t hurt.)

Special Note About Google Reader

At the time — 2011, remember — I was already concerned that Google Reader might go away. That was part of what made me want to have separate account types and make that explicit in the source list — it would have made it easier to adopt other syncing services.

Rather than make a user switch all their feeds from Google Reader to whatever, they could pick and choose, and it would have been clear which feeds were in which account.

I didn’t have plans to write my own syncing service (I was at NewsGator, and NewsGator had shut down its syncing service). But I did want to support any that came up, and make it so users didn’t face a steep switching cost — they could try out a new service, or many new services, with just a few feeds, and move more feeds over when they felt like it.

Anyway — those were my plans. Very three-years-ago.

Thanks to Omni

Eight of nine episodes of season one of The Record were recorded in a three-day span at The Omni Group offices last May.

They didn’t ask for any thanks or anything — they’re just plain generous. They even fed us. (Good food, too!)

Their generosity extends to the local Xcoders group. For months now the monthly meeting has been held in the Omni auditorium.

I’m a big fan of their apps, and I bet you are too. And I’m a big fan of the example they set as developers: create super-high-quality apps and charge a sustainable price.

Dear Omni: thank you.

Layout Code and Screen Sizes

Damon Stephenson writes, on Twitter:

@inessential I personally think that iOS 7 was a trojan horse for wider spread Auto Layout use if they change the resolution in future.

Here’s the thing. Auto layout is absolutely not needed in order to support different screen sizes and resolutions.

Remember the Mac.

We’ve been using autoresizing masks and old-fashioned layout code for years, and Macs have to deal with dynamic layout in ways that iOS apps don’t have to.

Windows can change size. Splitviews can change widths. There are more variations of screen size — and even multiple monitors.

The point of auto layout isn’t to make supporting different-sized devices possible, it’s to make it easier.

I don’t think it’s actually making it easier yet, not when you factor in the learning curve. I think we need better documentation and examples and better tools for debugging. Once we get there — then, yes, auto layout will make it easier. But it was never actually necessary for that particular job.

Update 7:15 pm: A case can be made that auto layout is needed for non-integral resolutions: 1.5x, for instance. But I think the chances of seeing this, even on a Mac, are slim. It appears to be a lesson learned. (I could always be wrong, however.)

[Sponsor] NSScreencast

NSScreencast features bite-sized screencasts for iOS Developers.

Want to level-up your iOS knowledge while enjoying your morning coffee? With an NSScreencast subscription you’ll enjoy a new screencast each week.

Why will you love NSScreencast?

  • Each episode is short and focused (10-20m). I don’t want to waste your time.
  • Trickle-learning. You’d be amazed how much you can accomplish in small chunks.
  • You can subscribe in iTunes to have the videos automatically downloaded, watch on your iPhone or iPad.
  • Subscribers get immediate access to the entire catalog of over 115 episodes.

Subscriptions are $9 per month or $100 per year. Team plans are available.

If you’d like to give NSScreencast a shot, here’s a coupon for 50% off the first month.</p>

The Cost of iOS 7

Jared Sinclair, iOS 7 Squandered a Year of Third-Party Development on Superficial Changes:

It’s been almost a year since version 7.0 was announced, yet as a developer I feel like a year’s worth of work has brought about only superficial changes to the apps I work on and the apps I use.

Jared makes good points. iOS 7 meant a bunch of unplanned work for developers — Vesper-with-syncing would have shipped by now, had we not spent last summer updating the app for iOS 7. (And we were a lucky case, since our app was more 7-like than most before iOS 7 shipped.)

And, meanwhile, there are other more urgent concerns (for developers) that have gone unhandled.

Jared argues that iOS 7 wasn’t urgent, that evolution rather than revolution would have been fine, since customer satisfaction was extremely high with iOS 6. In retrospect I agree, but were I at Apple I would have argued that the situation is like tech debt — UI debt — and it’s best to deal with it quickly, completely, and early.

Well. I look forward to WWDC.

Tvvitter

An NBC executive says that Twitter isn’t bringing higher ratings to its TV shows.

I hope this experiment ends. I can’t stand when TV news shows show Twitter messages. I can’t ignore them, because they’re words that appear on screen, and I have absolutely no defense against that.

They’ve never done anything but annoy me — they don’t contribute in any way.

Also: if Twitter does deliver higher ratings, then every TV show will figure out how to use Twitter. Can they all have higher ratings? Maybe if we all watch more television.

Indie Web

Dan Gillmor, Why the Indie Web movement is so important:

We’re in danger of losing what’s made the Internet the most important medium in history – a decentralized platform where the people at the edges of the networks – that would be you and me – don’t need permission to communicate, create and innovate.

I should probably pay attention to the Indie Web movement.

Green Rocks from Space

I like auto layout. A ton. It totally makes sense to me to declare layout rather than to code it.

I’m all on board.

I’ve worked with it enough to understand the APIs. I have no trouble visualizing what’s going on, and no trouble translating my intentions to the right declarations.

And yet I do have trouble. Of course I do. Because whenever I use it everything goes wrong. It’s like Kryptonite to me.

I don’t tend to get errors or warnings about my constraints — I just get things like a table header view sitting on top of the cells. Weird things like that that I can’t explain.

This isn’t like Core Data is to me — there are some things about Core Data that don’t fit how I like to do things. Which is fine. Auto layout, on the other hand, fits exactly.

What surprises me is how much I’ve worked with it without mastering it. It’s not that I don’t like learning new things — I learn new things all the time. (JavaScript and Node.js, most recently.)

The premise is great: save time and get better results. What is it about me that I can whip up layout code practically in my sleep and yet I can’t get the easier thing to work?

There’s no excuse for giving up. Except for the big, obvious one: I have to ship software, not play endlessly.

That’s okay as long as the giving-up isn’t permanent. I think my best bet is to adopt it really, really slowly, with near-leaf views and simple cases to start. But not till after I finish what’s in front of me right now.

Update 5:40 pm: I just realized that I don’t use the old-fashioned struts-and-springs stuff much either. I write layout code for everything. Struts-and-springs was often frustrating, and things didn’t work as I expected. (I’m not saying there were bugs, just that this gave me gas too. Clearly it’s my problem.)

The thing about auto layout, though, is that when things go wrong they go wildly wrong. I think that’s the issue for me. With old-fashioned layout code, things go wrong by just a little bit — and I can figure out, and fix, the problem pretty quickly.

But with auto layout, when things go wrong, it looks like a bomb went off.

Autolayout and Table View Header

What’s the best way to handle this situation?

I have a view that contains a 1) UIImageView and 2) UILabel (multi-line).

I set constraints appropriately. I do not specify a width or height for the image view or label, because intrinsicContentSize + constraints should combine to do the right things.

I create a UITableViewHeaderFooterView and I add this view to its contentView.

And… it’s all messed up.

It appears that I need to know the height of the view and sets its frame before adding it to the contentView.

How do I found out the best height for my view? Given the width (width of the superview), I need to know the best-fit height for my view.

Update 3:25 pm: I created a gist that shows my code.

Update 3:40 pm: Here’s what I expected. Create a view with subviews. Set constraints. Height is calculated from constraints. Everything works.

This reminds me of the old 45-minute rule for CSS-based layout. If it takes more than 45 minutes, punt and just use a table. Since it’s been about two hours, I’m overdue.

The old-fashioned way isn’t that hard, after all, especially given that I’ve done it a zillion times. Override layoutSubviews. Override sizeThatFits. Measure the text. Not rocket science. I would have been done two hours ago. (And it would be less code.)

But it still bugs me. I think that’s part of my anxiety with UI programming — I can master it, but then there are new, better ways of doing things, and those take time to master. Sometimes a lot of time. But I’ve got software to ship.

I don’t feel good about punting. But at least I learned some things, which is progress.

UIConundrum

Here’s one.

Storyboard. Static table view, non-scrolling. (It’s beautiful. Everything works.)

Except: I need a button at the bottom of the screen. It can’t be a table footer because it needs to be anchored to the bottom of the screen — 20 pts from the bottom. (No matter what the screen size or table size.)

I thought I could just add it as a subview of the UITableView and give it a couple autolayout constraints.

If I do, I get an error:

Auto Layout still required after executing -layoutSubviews. UITableView's implementation of -layoutSubviews needs to call super.

Well, I shouldn’t be surprised. Adding a subview to a table view willy-nilly is a hack.

But — how do I get that button in there, then?

One option is to use a container view, so that the table view is a subview of the view controller’s view.

But!

If I do that I can’t use that great storyboard feature of laying out static table views, since that works only when the view controller is a UITableViewController, and UITableViewController’s view needs to be a table view. (Correct?)

But I can do this, I think, if I use view controller containment. Make the static table view a separate view controller, and have the parent view controller add it. Then I still get storyboards and static table editing.

Okay. That’s one way.

The other option is to use a container view and do the static table view in code. Which means throwing away work I’ve already done and doing things the “hard” way. (Which is not actually all that hard.)

I can’t help but note that if I’d started out in code, I’d be done already, since adding a container view in code is an easy change to make.

So right now I have to evaluate trade-offs:

  • Using view controller containment increases the complexity. Another element in the storyboard. Code for embedding the table view controller.

  • Doing everything in code means more code. Nice that it’s all together instead of spread out, but it’s still more code.

Also: this applies to not just one but four screens.

Would switching to code allow for a more elegant solution? If I stick with storyboards, is there something else that might come up that would make me wish I had used code?

I don’t know the answers. It’s “maybe” to both.

There’s one thing I’ve learned, though: for me, it’s best to start out with code. I’m most productive that way.

But the question remains of what to do right now in this specific situation.

Mike Lee on The Record

We saved Mike Lee’s episode for the end — it’s the closing episode of season 1. And it’s super-good.

When Apple’s new Powerful ad came out, I was already thinking of Mike because we were working on this episode. But the “You’re more powerful than you think” message is perfect for Mike: he’s learned that lesson so well and now he teaches it.

The choice of the song “Gigantic” also fits Mike. Not because he’s a big guy, but because of his big, big love for making software and for the people in the community.

So: this is end of season 1. Chris and I are both immensely pleased with the podcast and the reaction to it. Thanks go to our sponsors — and especially to all our listeners. Making the show is a pleasure.

The Record will return.

Lesson on Clarity

It was back in my UserLand days, back in the ’90s, but I’ll be damned if I can remember the exact context.

I might have been working on some code — trying to take two similar things and create one generic solution for both.

Or it might have been some UI work (more likely) where I was trying to do something similar: make two similar things appear to the user as the same kind of thing.

This is what developers do, right? Collapse the differences, find the essence.

But Dave saw that I was struggling and had some advice for me.

The issue was clarity, he said, which is not helped by making different things the same. In fact, things that are truly different should be treated as different, and the differences themselves should be made clear. If you’re struggling, it’s a good sign that the things are truly different.

Some Mobile Services Tips

Here are a few tips, in no particular order, on using Mobile Services. I have months of experience, but not years, so treat as such.

Custom APIs Only

I don’t use the scripts attached to the tables. Instead I create all custom APIs. This lets me define exactly what callers can do.

Create SQL Tables Using Command Line

If you create a table using the portal, you get an id column that’s a string, plus some extra columns that you don’t need if you’re using custom APIs only.

Make sure you have the azure command line tool installed. (It’s a Node package.) Do azure mobile table create --help to see your options.

Note the --integerId flag — that’s what you want. If you use that flag when creating a table, you get a table where the id column is a bigint, and no extra columns.

Database queries and updates

It’s pretty easy to use table.where, table.insert, table.update — but I use request.service.mssql.query for almost everything, for reasons of efficiency and control.

Advanced Database Admin

If you have to do more than the portal will let you, use the Azure SQL Server command line.

Hopefully you won’t need to — but it’s not scary. (It’s much like using the sqlite3 command line.)

You May not Need the iOS Framework

If you’re using Facebook or Twitter authentication, you need the framework.

But since we’re not, we don’t. Here’s the thing: connecting to the service is really easy without the framework.

You call your methods as you’d think, with URLs like https://yourappname.azure-mobile.net/custom/api — and you need to do just two extra things:

  1. Add an x-zumo-application header with the app’s public access key.

  2. For authenticated calls, add an x-zumo-auth header with the JWT that Mobile Services returns.

After that it works like any other REST + JSON interface.

Create Two Mobile Services

One for production, one for staging.

Set up Git

You don’t want to paste your scripts in the browser. Set up SCM (Git) instead and deploy that way.

(I’ve asked for Mercurial support. I realize that doesn’t matter to most people.)

Use Shared Scripts

You can have shared scripts that your API scripts call. When you set up Git, you’ll note that there’s a shared folder. Include those (via require) from your API files, as in:

var something = require('../shared/something.js');

If you have a function somefunction you want made available, in something.js add a line like this:

exports.somefunction = somefunction;

This will make it available from the outside as something.somefunction.

Don’t Store Binaries in the SQL Database

Use blob storage instead.

Before Deploying

Make sure your files compile. (You should have installed Node.js on your system.) Type node filename. It’ll tell you if there are errors.

To Read

Josh Twist has a bunch of blog posts worth reading: The Twelve Days of Zumo.

If you need an associated website

You can create an Azure Node.js site. You don’t get all the nice parts of Mobile Services, but it’s still Node. That associated website can make API calls to your API server.

Mark on DHH on TDD

Mark Bernstein, Driven:

Methodological fundamentalists descend on technology from time to time. Hell, TDD and agile started as a rebellion against an older tech fundamentalism. As a profession, we’re suckers for The Word and want to know The True Way. Sometimes, the methodologies do help: structured programming, literate programming, strong types, weak types, objects, patterns, functional, agile: it’s all been good for us.

But, perversely, these helpful little ideas often become modes of punishment.

Surviving UI Programming

I’ve often wondered why UI programming is less fun than everything else.

My theory is that it’s because so much of it is arbitrary, single-use programming — I’m making a specific screen or view work the way it needs to, and there isn’t much that can be re-used. There are only rarely generic problems to solve. (Those generic problems are mostly solved — that’s what UIKit is for.)

Luckily, UI programming has gotten easier. These days we have view controller containment, custom view controller transitions, auto-layout, and, especially, appearance controls.

One of the nicer additions is editing static table views in storyboards. I’m doing that right now.

To make UI programming as smooth as possible I try to use all the things designed to make it easier. Stick with standard controls as much as possible. Use storyboards.

But it seems inevitable that there will be some point where I get frustrated. Right now it’s this: I have a grouped table view, a static table, and I want to add a section footer. (Not a table footer.) There’s nothing built-in to storyboards for this. (Please correct me if I’m wrong.)

To make matters worse: I don’t know how big the footer will be. It should be a container view that holds a UILabel with text that uses a custom font. The UILabel, and the container view, should be the exact height needed and no taller or shorter.

I think that the modern way is to add auto-layout margin constraints to the UILabel and the container view will tell me its proper size. But I worry that I’ll spend half a day on this, not get it to work (even if there is a way to get it work), and I’ll end up punting: doing a view that draws text in drawRect, and getting its size by measuring the text. I don’t want to do that, but I’ve done it plenty of times before, and I know it works.

So I think my problem is really anxiety. I expect to get frustrated.

And I have reason. Just last night I was attempting to add a button to a navigation bar. I created the button, then made a UIBarButtonItem with the button as the custom view, then added it as leftBarButtonItem.

The button worked — you could tap it and the right thing would happen — but the text of the button was never actually visible. I spent an hour, and I couldn’t explain it. (I finally just gave up.) (I even tried embedding the button in a container view and adding that instead. I gave the container view a background color. So I ended up with a red rectangle in the navbar but still no button text. Yes, the button was a subview of the container view.)

I’m not new to this. I had an app on day one of the App Store. And if you look at Vesper I don’t think you’ll think, “Oh, this app was clearly made by someone who gets anxious when he writes UI code.” There’s a ton of custom UI in Vesper, and I’m very proud of it.

But still: here I am, today, looking at something that should be simple — some text as the footer of table section — and wondering how long it’s going to take me, and wondering if I’ll have to give up on the right way and do it the old-fashioned way.

It’s possible that this just means I should spend more time doing UI programming.

Update 3:50 pm: John Siracusa on Twitter:

@brentsimmons The problem with UI prog. is you’re building on top of the deepest, most opaque stack of code you didn’t write and can’t see.

Exactly right. I think the only remedy is experience. But, the thing is, I have a bunch of experience.

Storyboard Advice

It’s a couple years old, but UIStoryboard Best Practices still has good advice.

I’m trying to use storyboards for the first time. At first I had one big storyboard that was really hard to work with, even on a 27” screen. Now I’m breaking it up into separate storyboards.

App Store Improvements

David Smith has good ideas Towards a Better App Store.

I’m worried about a possible future where the only way to make money (outside of the super-rare mega-hit) is with 1) websites with recurring revenue and 2) Mac apps sold outside the Mac App Store.

The iOS app, in this future, would be a checklist item. Something you have to make but that doesn’t directly make money.

The problem with that future is that making a good iOS app is expensive and takes time. If quality can’t be justified with revenue, then quality will suffer.

(There are some people who’d argue that this future is already here.)

A better App Store would mean (by definition) that quality apps do better. And the future I worry about could be avoided.

Mobile Services and Blob Storage

This is a quick write-up of how I got this working.

Here’s the situation:

  • I want to store images and other binaries in Azure blob storage (which is like Amazon S3) rather than in a SQL database or NoSQL table.

  • I want the client to upload directly to blob storage — I don’t want the image data to pass through the API server.

I struggled with this. For a while I was looking at the REST API and building a SharedKey. It wasn’t until I found Chris Risner’s page on this from last year that I found what I needed. (Thanks tons to Chris for writing this up.)

So this tutorial exists in case you’re in the same boat.

I’ll assume you already have a Mobile Service and know how to set up custom APIs. I’ll talk about the server side mostly, since you already know how to use NSURLSession and friends.

Get a blobService

Set up a storage service if you haven’t already. (New > Data Services > Storage.) A storage service is NoSQL table storage, queues, and blobs.

Add the storage service’s name and primary access key to your Mobile Service’s key storage (see the Configure pane, under app settings). Key names such as BLOB_ACCOUNT and BLOB_ACCESS_KEY are fine. Whatever you want.

In your code:

var azure = require('azure');

The azure module is available without you having to do anything. Just require it to use it.

The azure module gets you a blobService:

var blobService = azure.​createBlobService​(accountName, key, host);

accountName and key should come from your Mobile Service’s key storage — request.​service.​config.​appSettings.​BLOB_ACCOUNT, for instance.

The host is accountName + '.blob.core.windows.net'.

Whenever you’re going to do something with blob storage, get a blobService.

Creating a container

blobService.​createContainerIfNotExists​(containerName, function(err)…

This creates a private container. (You can create public containers too, but that’s outside of the scope here. It’s easy, though.)

Listing files

blobService.​listBlobs​(containerName, function(err, results)…

The results are in JSON. You could return those to the client, or filter them to just filenames or whatever the client needs.

Deleting a file

blobService.​deleteBlob​(containerName, filename, function(err)…

Uploading

Here’s the fun part. You don’t want the image binary to go through the API server. Instead, the API server generates a temporary shared access URL that will allow the client to upload directly.

First use createContainerIfNotExists. (I assume you have some way of deciding what the container names should be. They could be based on a person’s user ID, for instance. Depending on the security needs of your app, it may be a good idea to hash the container names, so they’re not personally identifiable.)

You need a sharedAccessPolicy.

var sharedAccessPolicy = {
  AccessPolicy: {
    Permissions: 'w',
    Expiry: azure.date.​minutesFromNow(5)
  }
}

Then you generate the URL to return to the client:

var sasURL = blobService.​generateSharedAccessSignature​(containerName, filename, sharedAccessPolicy);
var accountName = request.​service.​config.​appSettings.​BLOB_ACCOUNT;
var urlForUploading = 'https://' + accountName + '.blob.core.windows.net' + sasURL.path + '?' + qs.stringify(sasURL.queryString);

(Somewhere above this you need var qs = require('querystring').)

Return this URL to the client.

The client can use an NSURLSession​UploadTask. Use the PUT verb. Add Content-Type and Content-Length headers. (It’s a good idea to add Content-MD5 too, but not required.)

Downloading

It’s almost exactly the same as uploading, except that sharedAccessPolicy​.AccessPolicy.​Permissions is ‘r’ instead of ‘w’.

I have the server respond with a redirect to the URL.

That’s it. Pretty easy.

Glassboard Design Review

Jared Sinclair looks at Glassboard’s design:

When starting a new design project, the hardest part is finding the project’s voice. There are no easy solutions, and more than one right solution. You discover the voice through a repeating cycle of experimentation and refinement, one following the other, over and over. You try lots of things that turn out to be dead ends. With time and effort you accumulate enough familiarity with your project until one of the possible paths seems obvious.

Atom XHTML Content Considered Jerky

Dr. Drang writes about varying support for Atom’s content type="xhtml".

I strongly disliked this part of the Atom spec.

The thing about this feature is that the HTML tags are included as naked tags in the XML. That is, they’re not escaped or placed in a CDATA section.

From the good doctor’s example:

<content type="xhtml" xml:lang="en-US" xml:base="​http://lancemannion.typepad.com/​lance_mannion/">
<div xmlns="​http://www.w3.org/​1999/xhtml"><p><em>Barnes & Noble. Thursday. April 17, 2014. Six forty five p.m.</em></p>…
</content>

Just look at it. This feature is a giant invitation to screwed-up feeds. The HTML — which is probably a blog post, typed by a human — has to be valid XML. People writing scripts to generate these feeds have to make sure they can turn that HTML into valid XML.

Feeds are already screwed-up often enough. This could only make it worse.

The second issue: how can a parser handle this? What an RSS/Atom parser really wants is everything between <content> and </content> as a string.

I remember writing this code for NetNewsWire, which still supports this feature. (I checked. I have no idea if they’ve touched my code or not.)

I’m going from memory, but I’m pretty sure this is what I did.

NetNewsWire used libxml2’s SAX parser, which meant it gets called when the XML parser encounters the beginning and end of a tag and when it encounters a range of characters. There was no easy way — that I know of; correct me if I’m wrong — to tell the parser to treat everything inside a tag (<content> and </content>) as a string when there are XML tags inside that tag.

So I wrote the Atom parser to rebuild the HTML. It maintained a string and pushed stuff on it. If it encountered a tag and (optionally) attributes, it would create a string version and push it. When it encountered characters it would push those. When it encountered the end of a tag it would create a string version and push that. Once the </content> was reached then it had the entire string.

The end result was equivalent HTML, but not necessarily character-for-character the same, since little things like quote characters could change.

It worked. But boy was I coding angry that day.

PS Luckily I didn’t see this feature used very often. Still had to write the code, though.

PPS I still wonder if there’s an easier way. (Using a SAX parser. No way would I give up SAX. Performance and memory use considerations require SAX.)

PPPS For a great list of other ways feeds get screwed up, see Brian’s Stupid Feed Tricks.

Azure SQL Server Command Line

Database stuff is easy using Mobile Services — it’s a piece of cake to do almost everything you need via the browser or via the Azure command line.

Almost everything. It’s great until you hit the things that aren’t supported. Then, well, then there’s this weird browser-based SQL Server portal you could use.

But that’s not true anymore. Now there’s sql-cli, a Node module that provides a command-line interface to SQL Server. It’s so much like using the sqlite3 command-line interface that I’m at home right away. (It’s written by Hasan Khan, who’s been a huge help. Just about daily, lately.)

[Sponsor] Microsoft Azure Mobile Services

Mobile Services is a great way to provide backend services — syncing and other things — for your iPhone, iPad, and Mac apps.

If you’ve been to the website already, you’ve seen the tutorials where you input code into a browser window. And that’s an easy way to get started.

But don’t be fooled: Mobile Services is deep, too. A mobile service is a Node.js site, and you can write JavaScript in your favorite text editor. Deploy via Git. Write unit tests using Mocha. Administer via the command line. All on your Mac.

You can get started today with a free trial.

Vesper Sync Diary #15 - Server Testing

I don’t write many tests as I go along, because I know I’ll change the design and do a bunch of refactoring. Tests would have to be rewritten and rewritten again.

I like the idea of test-driven development, but in practice it means more wasted time than I’d like. (For me. I’m sure it works great for other developers, but not every developer’s the same.)

I just ran cloc on the API server. It’s 1,545 lines of code. It’s small, which I take as a good thing. By the time I’ve finished writing tests it should be closer to 3,000 lines of code.

(If writing a Mac app is like writing a novel, and writing an iOS app is like writing a short story, then writing an API server is like writing a poem. It has to be compact, and every line has to have a reason to be there and has to do its job perfectly. In an iOS app you can slightly screw up an animation somewhere and just fix it later. On the server nothing can be slightly screwed up.)

My process

First step is to go through every single file and add all the easily-testable functions to a to-test list. That’s most of them.

As I’m going along I find some small refactorings that can be done that will make more things testable, so I jot those down and then do those after going through all the files. And update the list of functions to test.

Next step is to write the tests — using Mocha — from the ground up. Do the low-level simple utility routines first.

(I used Josh Twist’s post on unit testing Mobile Services scripts to help me get started.)

Live tests

Unit tests are indispensable, but there are two issues:

  • They don’t actually hit the server itself.

  • There are some things that are very difficult to test — because they need all the context of the server.

For instance, it’s easy to test the logic for merging notes, but it’s hard to test end-to-end — sending some notes to the server and making sure they really end up merged and stored in the database.

I’ve been thinking about what to do here and I don’t see any existing frameworks that fit the bill. (Tell me if I’m wrong.)

So here’s my thinking. I’ll write code in the client app that calls the server and runs tests with some test data. Ideally these would be XCTestCase tests — with special junk for doing async tests — but if that doesn’t work, if it’s a special configuration of the app instead, I won’t stress about it. (Well. Weeeeell. I might stress about it. And bang on it until it works. Because it really ought to work. It bugs me so much that XCTestCase doesn’t have built-in support for async tests.)

This is made just a little more complex by the fact that I want to run tests against both the production and staging servers. That’s a small complexity, though. The bigger thing is the code to set up the environment on the server so that it can run the tests.

Here’s what I like, though: every line of code is testable. All the logic is testable. This is so different from client apps, where it’s hard-to-impossible to test everything.

Given the complete lack of margin for error, given my need to not constantly worry about the server, this is a wonderful thing.

PS I counted. First step is to write unit tests for 53 functions.

Scaling and Performance

In the recent Accidental Tech Podcast John Siracusa pointed out that I had talked about scaling when I was really talking about performance.

John’s right, and I know better.

In the vernacular sense of scaling the two things are related, but I should use more precise language.

So I’ll be more precise: my main goal is to maximize performance and minimize the amount of resources used. My second goal is to design so that I have scaling options if needed.

(Performance is related to scaling only in the sense that it lessens the need to scale, but it doesn’t solve the problem of scaling itself.)

Here’s what I’m doing for actual scaling:

  • Using a system that allows for multiple web servers to be automatically created as needed.

  • Using a SQL Server database with a maximum of 150 GB.

  • Using blob storage for binaries (images), which can scale forever (presumably).

The weakest link here is, I think, the database. If I have to, I can split up the data into separate databases. There are just four tables — accounts, deletednotes, notes, and tags — and each of those could be moved into a separate database. (There are no joins and no foreign key constraints.)

I don’t expect to ever come anywhere near doing that, so I’m not actually planning the migration steps. But it’s in the back of my head that there’s a non-zero probability.

And if I have to go even further, I can. The biggest of the tables, by far, is the notes table. I can break that table out into separate databases, separated by userID. (Which is an integer. Each database would store notes for a range of userIDs.)

And, if that’s not enough, the remaining tables (accounts, deletednotes, and tags) could also be broken out the same way.

I don’t think I’ll ever have to do any of this — but I can, if I need to, with only small code changes.

The web server may also be a weak link. The way to solve this one — not that I think I’ll need to, because I can run a bunch of instances of the server — is to create separate API servers. Each endpoint could be a separate server with its own set of instances. This would require a small code change on the client, but I’d see this coming and make sure it gets done in plenty of time.

IAC on iOS

Check out GCDWebServer. (Via iOS Dev Weekly, which you should subscribe to.)

GCDWebServer is an embeddable and lightweight http server for Mac and iOS. On iOS it runs as a background task — it keeps running even when your app isn’t in the foreground.

Picture this: app X wants to send some data to app Y.

App Y is running GCDWebServer, so app X wraps up the data as JSON and talks to app Y via http. Just as if it were talking to some server on the web, only it’s a local app.

Update 4 pm: Or not. I’m not sure it’s possible, or maybe just not allowed, to keep a network server running when the app is in the background.

Node + MongoDB + iOS

A two-part tutorial from Michael Katz is a good place to get started writing services. Node makes a great API server. And it’s fun.

How To Write A Simple Node.js/MongoDB Web Service for an iOS App

How to Write An iOS App that Uses a Node.js/MongoDB Web Service

The tutorials use MongoDB. I haven’t had a good reason for a NoSQL database myself lately — but my early career, back in the ’90s, was all about schema-less databases, and I have a major soft spot for them.

(I’m digressing now.)

Frontier’s database was a hierarchy of tables. Each table could contain anything, including other tables — including even your scripts.

To run a script named myScript inside the bar table which was inside the foo table, you’d write foo.bar.myScript(params).

If that script took a string as a parameter, say, you could use a local variable or reference any string anywhere in the database: myApp.data.settings.username, for example. This was all presented with a user interface, navigable and editable.

I haven’t seen a database like that anywhere else since then. So easy and intuitive. Great for productivity. (It was within this laboratory that such things as templated and scripted websites, blogs, RSS, OPML, and XML-RPC were invented and/or fleshed-out.)

Web, Money; App Store, No Money

Subvert: Why we chose to build our core software business on the open web instead of on a closed app store platform:

More so, unless you have huge sales volumes, it’s near impossible for a company selling $2.99 apps (our lowest cost product and a fee which, in the app store world, is considered wildly exorbitant) to make enough money to support their team, pay for office space, keep up to taxes, cover fees and look after all of the financial requirements that it takes to run a real business.

What is making money for them? A web app.

We’ve done next to zero promotion of this software-as-a-service (SaaS) product since we started building, using and supporting it two years ago. Instead, we’ve been slowly adding to the system, improving the platform and signing up paying customers on a regular basis. As a result, the product has been making money — real money — and long ago surpassed the combined revenues of our other commercial software applications in a big way.

I’m not sure that the choice is between web apps and app store apps — I think it’s actually about standalone apps versus apps-plus-services. But, still, read the post.

Vesper Sync Diary #14 - Keys

Here’s the initial design: the text of notes is encrypted in the database. The key is not stored in the source code. (The source code could get out and you wouldn’t be able to decrypt notes.)

That design is a no-brainer, and I thought I was finished at that point. But then I did a design review with some security folks, and they suggested I revise it like this:

The text of notes is encrypted in the database. They key is not stored in the source code. The key should change from time to time. To make this work:

  • Always encrypt using the latest key.

  • Add some token to the text that lets me know if it’s been successfully decrypted.

  • Try decrypting using the latest key first. If the token doesn’t appear in the right place, use the previous key. Repeat if necessary until decrypted.

  • Add a new key regularly. (Twice a year, for example.)

  • Be prepared with a script that re-encrypts all note text with the latest key, in case there are any security concerns at all.

This is sensible. I can’t think of any reason not to do it this way. Is there anything I’m missing?

Clangalyzer

Keith Harrison shows how to run custom Clang analyzer builds:

The open source build is updated frequently with bug fixes and extra checks. This makes it more likely to spot an error in your code that would go undetected with the Xcode bundled version.

Got @inessential

I went through Twitter’s impersonation process and was able to get the @inessential username.

The way it worked is that Twitter renamed @inessential_com, which I created a few days ago, to @inessential.

So if you followed @inessential_com, you’re now following @inessential, and there’s nothing you need to do.

Justin Refactors

Justin Williams: Refactoring in the Cloud:

Given that I’m not a fan of the C# threading model currently implemented, the amount of code used to power something that’s relatively simple, and the homegrown nature of many of the push wrappers we’re using, I made the decision to rewrite this portion of Glassboard’s architecture using Node.

Justin’s use of Azure is quite different from mine. Glassboard is more complex than Vesper syncing, which should be no surprise.

(If you think about it, you realize that Glassboard does syncing, but it’s silly to call it that because it’s a natural part of what’s expected of a group messaging app.)

Vesper’s architecture is smaller. There are two main components: an API server (Node.js Mobile Services app) and a reset-password site (Azure web site, also Node.js). There are no architecture-level refactorings for us to do at this point.

We’re at about 2,000 lines of server-side code. Small.

Premature Optimization and Servers

More than one person has suggested I’m guilty of violating the law of premature optimization when it comes to my server work.

Here’s the thing, though: when it comes to database schema, I really, really want to get it right before shipping.

Making code changes in a client app is normal. Making database schema changes in a client app is a pain, but not the worst thing.

Making code changes on the server is normal too, though a little hairy. But the hairiest of all is database schema changes on the server. I’m designing so that I don’t ever need to do that. (I may not reach that goal. Time will tell.)

Even though Brian Reischl wrote up how to do data migration, and so I have a good plan if I ever need to go there, I just don’t ever want to go there.

In other words: getting the server-side database schema right right now isn’t premature — it’s exactly the right time.

More on UUIDs and Clustered Indexes

My SQL Server genius pointed me to this article by Kimberly Tripp about the problem with GUIDs as primary and/or clustering key. (GUIDs and UUIDs are the same thing in this context. Microsoft folks often call them GUIDs.)

Another article suggests this isn’t a big deal with Azure SQL because a network write is slower than a page split anyway.

But the advice still seems to be that UUIDs don’t make the best clustering key. You want something narrow to keep down index size.

So I’ve slept on it. Do I still like this layout for the notes table?

id - auto-incrementing integer clustering primary key
noteID - UUID, unique
userID - int

Yes, I like it.

It’s the API?

Cesare Rocchi argues that the Heartbleed problem isn’t C — it’s the API.

My counter-argument: people make mistakes. People make dumb APIs. With C, combine a dumb API and a mistake and you get Heartbleed.

That’s far less likely with another language.

Here’s the thing: we will always have dumb ideas and mistakes. We can and should do our best to eliminate them, but we’ll never succeed entirely. Because we know that, we’re negligent if we don’t do our best to minimize their consequences.

Vesper Sync Diary #13 part 3 - Thinking Too Much

There are two pieces of advice I’ve been getting:

One is that I’m thinking too much about this. It’ll be fine if I have a properly normalized schema and I use the appropriate indexes. After that, don’t worry.

(I admit that I’m prone to going down every performance rabbit-hole I can find.)

The other — partly related — is that the right way to deal with the notes table is to do create a clustered primary key as userID + noteID. This way all notes by a given user will be together.

And this is the default behavior. It’s good. Smarter people than I am have thought about this.

And I can drop the integer identity column.

Update a few hours later: No. Wait. The best database guy I know tells me to do it the way I was thinking: surrogate key integer identity column as clustering key.

He also suggests I don’t need to add a unique constraint for noteID + userID, since noteID is a UUID. A unique constraint on noteID is all that’s needed.

SettleApp

I helped Ondřej with some of the wording on the home page for SettleApp.

Some advice for developers in general, off the top of my head:

  • Use words that humans use. (No “algorithms.”)

  • Use verbs. Verbs are vivid. Verbs trigger people’s imaginations — they picture themselves using the app.

  • Use curly quotes.

  • Don’t use comma splices.

  • Don’t talk about how you rewrote any part of your app. Nobody cares.

  • Don’t talk about how it’s native or about how it’s ready for the latest version of the OS. These things are assumed.

  • Don’t be repetitive.

  • Simplify.

  • Cut, cut, cut, cut, cut, cut.

  • Cut.

Vesper Sync Diary #13 part 2 - Maybe It’ll Be UUIDs After All

I slept on the thing about using 53-bit integers as note IDs. I woke up and it felt weird — slightly but detectably. I listen to those feelings.

As previously mentioned, I’m still learning my new environment. Any new environment takes time. The thing to do, when things seem weird, is more research.

Though the initial problem was a JavaScript problem, the thing I’m trying to optimize is a SQL Server database. So I’m doing some more research on the database.

UUID data type

SQL Server has a UNIQUEIDENTIFIER type. (Sorry for the shouting, but database people are shout-y.)

I assume this saves space over just using a 36-character UUID string — I figure it strips the dashes, at least.

That’s wider than a 64-bit integer, and I don’t love it, but it’s not the worst thing ever to happen. The great thing about UUIDs is that they can be created on the client and guaranteed (for practical, real-world values of “guaranteed”) not to collide.

(Remember that, in Vesper’s case, a note ID need be unique only for a given user.)

I really wanted 64-bit integers. But I’m willing to accept UUIDs because it’s not weird and is a standard practice.

Clustered indexes

Azure SQL Server insists that each table has a clustered index — that is, an index which tells the database how to physically lay out the data.

The primary key is by default the clustered index. The primary key for the notes table is noteID + userID.

Recall that UUIDs look like this:

4C958F48-A332-44D2-A0AD-3E91EF172C6D
0ACA46EE-86F8-4443-BF51-A52A4506FD6C

There’s no order to them. So, to insert a new note could mean inserting it anywhere in the table rather than at the end.

That sucks.

(Yes, I know about NEWSEQUENTIALID(), but I’m generating IDs on the client.)

But here’s the thing: I had this exact same problem even before using UUIDs, because I was using random integers.

What I want instead is a way to add a row at the end, like adding a page to a book. For that I’d need to create a separate identity column with a monotically-increasing integer — like adding a page number — and then make that the clustering key. (The primary key and clustering key do not have to be the same.)

Alternate approach

Having primary key as noteID + userID, and a separate identity clustering key, is just about the same thing as this layout:

Primary key: identity integer (page number style)
Constraint: noteID + userID is unique

This alternate approach feels more right.

My goal is to make noteID + userID unique, and to enable fast lookups. Adding a constraint (which adds an index) makes that work. I don’t also have to make it the primary key.

Do I still have the problem of JavaScript and big integers?

Does my code ever need to see the identity integer? The clients don’t — it’s a server-side detail only.

But if my server-side JavaScript code needs to reference it, then once it passes 9007199254740992 notes then we’re screwed.

You know what? I’m fine with that. To hit that number, every human on Earth would have to create over a million notes. I won’t worry about it. I’d be a billionaire long before then, and my blog posts will be me figuring out which island I should buy.

(With that much money this problem would be solvable, in other words. Solvable by other people who I pay very, very well.)

Summary of changes

I think it’s a plan.

Client will have to make these changes:

  • Generate note IDs as UUIDs.
  • Store note IDs as UUIDs. (Change database schema and data model objects.)
  • Fix code that expects note.uniqueID to be a 64-bit integer.

Server will have to make these changes:

  • Update notes and deletednotes tables to have an integer identity clustered primary key.
  • Update notes and deletednotes tables so that noteID is a UUID.
  • Updates notes and deletednotes tables with a unique constraint on noteID + userID.

The good news: it’s possible that none of the server-side JavaScript will have to change. If there are any changes, they’re few and small.

The bad news: there’s more client-side code to change than I’d like. Normally a schema change in a development version isn’t a rough thing, but it does mean going into the database migration code again, which I don’t like doing.

The other good news: but the database migration will actually be simpler, since I was using UUIDs for note IDs in Vesper 1.0. I can preserve those UUIDs across the migration, where until just now I was changing note ID to 64-bit integers. So that’s cool. Simpler data migration is better.

PS The only thing that gives me pause — and something always gives me pause — is the idea that I could make userID + noteID the primary clustered key. (userID first.) This would mean that notes don’t get added at the end of the table — but all notes for a given user would be stored together. I think. I don’t have an easy way to test this, but my gut says that adding notes at the end is the better way to do this, and I should just rely on that unique constraint index to make lookups fast.

This is where the value of knowing your environment comes in. If I had five years, say, of SQL Server experience, I’d probably just know the best answer.

Well, the only way to learn an environment is to work in it, and that’s what I’m doing. Later I may look back and laugh. (Wouldn’t be the first time. Not even close.)

PPS If it seems like I’m working things out by writing them up here, it’s because I am.

Also, if it seems like I sometimes have to try everything before I get to an answer I like — well, that’s often also true.

PPPS Why did 53-bit integers seem weird enough to bug me? I have to plan for all options. There’s always the chance that another developer helps with this code — web app, for instance. Or we could make the API open. (Doubtful, but I have to plan for craziness.) Would the 53-bit limit be observed? It’s a data corruption bug waiting to happen. Even though it’s one little thing — an upper limit on note IDs — it’s too much complexity. Sync needs to be as simple as possible.

Drew’s Core Data Sync

Ensembles 1.0 ships:

Today, Ensembles is a framework worthy of consideration if you want to add sync to a Core Data app. As you would expect from a 1.0 release, it is robust and stable, and is already shipping in a handful of apps.

[Sponsor] Briefs

If you’re reading this, you’re likely a developer of impeccable taste who cares and sweats the details. Don’t you want to work with designers who care as much about those same details? If so, ask them to check out Briefs.

It will let them create professional quality prototypes from their mockups and wireframes using a powerful Mac application. A brief is easily shared and can be deployed to iOS and Android devices. They’re great for quick demos. Briefs lets you worry about the final product, while your designer is busy playing with ideas that might or might not work out.

Recommend Briefs to every designer you work with and make your life easier. Send them to http://giveabrief.com and tell them they can try out Briefs for free.

Vesper Sync Diary #13: Unlucky Numbers

I’ve been writing Cocoa apps for so long that I’m rarely surprised. I don’t know everything — surprises happen, and of course there are things that I know I don’t know — but, in the main, I know the intricacies well enough that I don’t even have to think about them.

But writing syncing means learning a new environment. For me it’s JavaScript, Node.js, and Azure. And I just ran across a big surprise in JavaScript.

If you have Node.js installed on your system, you can open Terminal and do this.

Launch Node. (Just type node and hit return.)

Copy-and-paste the following number and hit return: 9223372036854775807.

Node evaluates the expression. What do you think you get?

> 9223372036854775807
9223372036854776000

That’s right. It’s not the same number.

Some of you will note that the number I chose is the maximum value of a 64-bit integer. The problem occurs with sufficiently large integers. Here’s the deal: JavaScript numbers are 64-bit floating point numbers, which means they don’t have enough precision for very large integers.

I had no idea.

I should have known. Well, it’s a thing I needed to learn, and I just learned it.

Why Care?

When does a note-taking app need to deal with very large numbers? After all, if someone types a big number into a note, it doesn’t matter, because the entire note is treated as a string.

Here’s the thing: unique IDs for notes and users are 64-bit integers in my system.

So when the iOS app sends a note to the server, it sends a 64-bit integer along with note text and other properties of the note.

JavaScript code takes that data and then inserts the data in the database.

The “takes that data” part is where JavaScript turns the note’s unique ID from 9223372036854775807 (for instance) to 9223372036854776000 (for instance). We can’t have that.

Options

I’m writing this up as soon as I learned about it. I have to figure out what to do.

The first thing: user IDs might as well be 32-bit integers. (JavaScript can handle 32-bit integers accurately.)

Before we get anywhere near 2147483647 (max for 32-bit integers), we will have hired a team of the brightest server-side programmers in the world, and they’ll figure out what to do.

But that still leaves note IDs.

Option 1: stringify

On the server side, treat the unique ID as a string. JavaScript won’t mess with it if it’s a string. Strings are tough and opaque.

Problem: database bloat. Storing strings takes up way more space than storing 64-bit integers.

Plus it’s inelegant.

Option 2: 32-bit integers

Collisions are more likely on assigning a random note ID when using 32-bit integers instead of 64-bit integers.

Now, the client app does check for collisions by looking at the note IDs it knows about.

But there may be a few note IDs it doesn’t know about (that haven’t been synced from another client, for instance).

That’s a very, very small risk. Most of the time there won’t be unknown note IDs, and when there are it will tend to be very few.

But still: ugh.

(Remember that global note ID collisions aren’t an issue. A given note ID has to be unique for a given user only.)

Option 3: 53-bit integers

The largest integer JavaScript can handle is 2-to-the-53rd power.

> Math.pow(2, 53)
9007199254740992

The downside is still that there’s a chance of collisions — but, really, that’s a sufficiently large number size that I’m not at all concerned. (Remember that the client app checks, so the chance of a collision is really with a new note ID and any note IDs the client doesn’t know about yet, which will usually be zero anyway.)

And — it just occurred to me that I can add a second collision check: if a note comes from the server that matches the unique ID of an existing note, and the creation dates of the notes are not identical, then I can deal with that. (By giving one of the two notes a new ID and syncing.)

This is what I’ll do.

Mark on What’s Really Dangerous

Mark Bernstein: It’s Not C.

Any computer language capable of doing real work is also capable of being confusing, capable of being misused, capable of being subverted.

Mark’s right. Free computing is free speech.

I certainly don’t recommend banning C, as if anyone could. I don’t use it much, but when performance is an issue, I do. (Note that I need to update that code: needs more braces.)

However, it’s worth thinking about less-dangerous alternatives. Programmers will always make mistakes, and that’s why we do automated testing, static analysis, code reviews, and so on. That same reason — the inevitability of mistakes — is a reason to use higher-level languages when possible.

Unfortunately, “when possible” is just not very often when it comes to portable libraries such as OpenSSL. What would you write it in, if not C? I don’t know, but I think it makes sense to have an alternative language for this where you don’t have to be so damn careful with memory, where things like buffer over-runs are impossible.

I suppose that static analysis tools could get good enough to catch these errors. Maybe. C is so flexible that I just don’t know if they’ll ever be good enough. But if they get that good, then the culture and the tools have to make it so that no developer would ever fail to run and pay attention to static analysis results. That seems like a longshot to me, but I’d be pleased to be wrong.

An even bigger longshot is counting on developers to get better at testing. That’s like counting on people not to drink and drive. The culture has done a good job at stigmatizing this particular bad idea — but it still happens. That’s not to say we shouldn’t try — we should, definitely, for sure — but alone it’s not enough to fix the problem.

Though the OpenSSL bug affected servers, I note that most people don’t write their web services and sites in C. It’s do-able. You could, and surely some people are, somewhere. But to most of us it sounds crazy, and for good reason.

As computing evolves, the domains where C “sounds crazy” will continue to expand. Consider that people are already treating JavaScript as assembler (see CoffeeScript and TypeScript). As an industry we continue to move toward higher-level languages, and that’s a good thing.

Big Nerd Dependency Injection

Graham Lee: Dependency Injection, iOS and You:

The reason brittle object graphs are bad is that you cannot easily replace parts of the application. If an object expects to ask its environment for a load of other objects around it, then you cannot simply tell it that it should be using another object. Dependency injection fixes that. It tells the object, “Hey, these are the objects you should work with,” so that if we want to change the collaborators we just inject different things.

I’d link to everything published by Big Nerd Ranch. You should subscribe to its RSS feed.

Feature Request for Text Editors

(Maybe there’s a text editor that does this that I don’t know about. But I haven’t found one.)

I have my Node.js project open in my text editor. Folders and files are in the left-hand sidebar. Click on one to show that file. You know the picture.

(Wouldn’t have to be Node.js. Could be any programming project.)

Now I’m looking at a file that references a function that I want to look up. As an Xcode user, I expect to be able to cmd-click on the name and go right there, even if it’s in another file. And then I expect to be able to go right back to where I was with a keyboard shortcut. (Or with a swipe on my mouse.)

But it doesn’t work. The text editor is smart enough to be able to give me a list of the symbols in each file, but it’s not putting this all together.

I’ve made this work before — at least the navigate-to-symbol part — by maintaining a Ctags file. But this is a pain. It needs to be updated when things change.

What I want is the obvious thing: the text editor — which knows what language each file is, which knows where the root folder is — should handle creating and updating the Ctags file behind the scenes. It should do this for me.

And the second part of this — going back easily — is just as critical.

My point: a significant chunk of programming is just navigating, and no text editors (that I know of) get this completely.

It should work like a browser. Forward and back.

(For bonus points: it’s not file-based but based on position in file. Say I’m at line 300, and I go to the top of the file to include another file. I want to jump back to line 300 the same way I’d go back to a separate file.)

If there is a text editor that does this — all of this — let me know what it is because I want it right now.

That Pretty Much Wraps it Up for C

I love C. So damn much. I enjoy writing straight C.

I love having some memory and some abstractions and the flexibility to do whatever I want. I love that it imposes a discipline that prevents me from using that flexibility in stupid ways.

But between goto fail and the Heartbleed bug I have to wonder if my beloved language should be retired — at least for everything that could be a security issue. (Which is potentially everything, I suppose.)

There are two things that make other languages better for these things: 1) in many languages these particular bugs are impossible, and 2) there are often better static analysis tools that can prove that those particular flaws don’t exist in a chunk of code.

The Heartbleed bug is a major hassle. What worries me is that the next time something else might happen — maybe the power grid goes down or ATMs stop working. (Or worse.)

If we’re serious about protecting ourselves from the NSA and other malevolent entities, maybe we have to move away from C.

(I suggest this with great reluctance. And a realization that the cost of this would be huge.)

“Security”

Were the National Security Agency an agency charged with the security of our nation, it would have reported the Heartbleed bug immediately instead of exploiting it.

Analytics

Justin Williams writes about data analytics that matter. Glassboard didn’t have much when he took it over.

A number of things contributed to that. Partly it was just time — we were busy developing the features of the app.

But another is that analytics (and data, and statistics) is a blind spot for me. Part of that comes from an intense respect for privacy, and part comes from preferring words and stories to numbers.

I’m learning. But I start from the belief that to use analytics at all is to lose your soul — which is immature, emotional, and incorrect.

It is possible — I keep telling myself — to use analytics to make better software (and blogs and podcasts) and stay on the good side, far away from the creepy line.

Constants and Reading and Writing

It’s common wisdom — and almost always correct — that it’s better to use constants than literals. As in return VSSidebarWidthMaximum instead of return 256.0f. I agree completely with that wisdom.

But there are two particular (and similar) cases where I disagree. Which is better?

response.send(200, someStuff);

…or…

response.send(statusCodes.OK, someStuff);

For me the first one is far better. By now the HTTP response codes are wired deep into my brain, and so the 200 literal is much easier to write and read than any constant.

Another example:

[request setHTTPMethod:@"POST"];

…or…

[request setHTTPMethod:VSHTTPMethodPost];

The first one wins. Yes, there is the possibility that I’ll mis-type it, and of course the compiler won’t tell me. But that trade-off is totally worth it, because now I can see instantly that it’s a POST method request. My brain doesn’t need to translate. (The odds of my mis-typing @"POST" and not noticing it are ridiculously small.)

Cocoa for Web Services

Consider Marco Arment’s software. You know him from Tumblr, Instapaper, The Magazine, and (coming soon) Overcast.

You may think of Marco as an iOS developer — but every single one of those apps is a web service.

And, furthermore, those web services do more than just provide syncing between iOS devices, though syncing is a component.

Here’s the lesson to learn: we’re no longer in the apps business. We’re in the apps-and-services business.

The problem with generic syncing

Generic syncing with Mac and iOS devices will never be as efficient as something you build yourself. It’s a super-hard problem to solve, but there are people and teams who’ve made progress on it.

It will get solved to the point where lots of apps can adopt it without too much trouble.

But it’s not enough. If it’s easy, everyone does it, and it’s a baseline. Competitive apps will have better and more efficient sync — and they will have more than just syncing with Mac and iOS devices.

They’ll have additional services — which might mean support for other platforms (web, Android, etc.), publishing (as Day One recently added), social networking, content services, and so on. (It depends on the app.)

The cloud is more than just a file system. It’s data plus code.

Progress

Though there has been amazing progress in the web world in the last 10 years (while I was mostly ignoring it), we don’t have the equivalent of Cocoa for web services. I think it’s coming, though.

I don’t mean that we’ll write in Objective-C and host our services on Macs (though we might, and I’m aware that there was once a thing called WebObjects) — I mean that eventually we’ll have something that matches Cocoa in spirit.

Whatever it is will have these things in common with modern Cocoa:

  • It will be very high level.

  • It will solve all the common problems easily, via configuration.

  • It will allow developers to concentrate on the parts that make their app unique — while still using rational APIs and design patterns encouraged by the framework.

  • It will be customizable at every level.

  • It will be fast and efficient — built for small teams making consumer apps rather than for enterprise.

An Example

An accounts system — with password-reset pages and everything else — is an obvious example. Let’s talk about something more complex.

A common issue for lots of apps is storing images and videos in the cloud and optionally publishing them. This means generating and storing images at different resolutions and transcoding video for different platforms.

It means serving the right size and format to the device or computer that asks. It means managing permissions — is this private to me or to a group, or is it public?

This is something you can solve using S3, Azure blob storage, or a file system somewhere. But to do so is to reinvent the wheel. This is exactly the kind of thing that should be solved once and made a part of a high-level web services framework.

Second Example

How many services are based on polling? Reading RSS and Twitter feeds, web pages, some APIs somewhere — and then parsing the results somehow and updating a database.

It’s another super-common pattern — and, again, the kind of thing that could be solved once. Give me an API that makes it easy to configure a download queue and add things to it. I’ll write a delegate method that parses the results of a download. Maybe there’s a second queue to update the database. And observers that determine who should get push notifications.

Sending a push would be as simple as providing some user IDs and the message to send — the service would use the right mechanisms for the right platforms for those users.

Seems obvious, right? Imagine writing web services the same way you write apps.

Who Will Get There First?

What I saw in Azure Mobile Services, which I use, is that they’re moving in that direction. It’s not there yet, and I think it will be a while before anybody has the whole thing. But the idea that there’s a higher level than deploying Node.js sites is absolutely correct.

I have not checked out everything else. Many people like Parse (Facebook liked them enough to acquire them last year).

Another one people like, Helios, is open source.

I don’t know who’s going to get there first. It could be a framework we haven’t heard of yet. But I do think it will come from a company that understands mobile app development and has experience supporting native app developers.

(Apple? I doubt Apple’s interested. In my dream world Apple designs the services and APIs and gets another company to run it. But Apple, understandably, has zero apparent interest in doing anything that would make it easier to create services for Android, Windows Mobile, and web apps. It’s easy to understand why — it doesn’t sell Apple hardware.)

But — and here’s the important point — we app developers can’t wait. Though I think we’ll have this eventually, it will arrive incrementally, and we need to use what exists today.

Update 12:50 pm: People on Twitter reminded me of another one: Objective-Cloud.

Just Don’t Listen

In 1997 I went to InternetWorld in Los Angeles — it may have been my first tech conference.

The prevailing wisdom was that in a few years the web would be consolidated. There’d be seven or eight large sites like PathFinder, and that’s it. Everybody knew it.

And then blogging happened.

Fast-forward to just ten years ago. In 2004 everybody knew that to write native apps was to stick to a hokey religion and ancient frameworks. No match for web apps. Time was up for native apps. Everybody knew it.

And then iPhone and iPad happened.

Now, today, there’s a report about how the web is doomed and native apps are eating up everything. Everybody knows it’s pretty much over for the web.

Do not believe it.

Cosmopolitanism

Small thing. When you create a new service in Azure Mobile Services, it gives you some scaffolding — directory structure and a few files. Those files include read-me files. And they’re Markdown files. (Screenshot.)

I happen to know the guy who came up with Markdown. Total Mac nerd.

It shouldn’t be surprising. I’ll stop being surprised.

Build 2014

Q Branch was at Microsoft’s Build 2014 conference in San Francisco. Build is very much a bizarro-world WWDC — it’s in Moscone West, and it’s full of programmers, but everything’s different.

I’ve never attended anything but Apple events in Moscone West, and I couldn’t help but think of it as an Apple outpost. But it’s not.

Other companies can do whatever they want. Who knew? They can have a DJ. They can rotate the keynote stage and seating 90 degrees — instead of a deep and narrow room, it’s a wide and shallow room where everybody’s closer to the stage.

Like Apple, Microsoft featured plenty of iPhones and iPads in their keynote. (At least the day two keynote, which was all about Azure and the cloud.)

While seeing iOS devices on a big screen in Moscone West was normal to us, we knew you’d never see Apple feature Android or Windows Mobile devices in their keynotes.

Nor should they. That’s not a criticism — that’s just not Apple’s thing. It’s the new Microsoft’s thing to be cosmopolitan.

I talked to a number of Microsoft employees — on the Azure side — and got the same sense from all of them. They’re excited. They know they’re underdogs; they know that Amazon Web Services is dominant.

They also know that the kind of dominance Microsoft once had — where just about everything that computed ran Windows — is gone and will never come back.

This new Microsoft didn’t launch from thin air the day Satya Nadella was made CEO. They’ve been working on their cloud services for years. (Example: Mobile Services, which I use, runs on Node.js and includes an open source iOS SDK on GitHub. This comes from the Ballmer era.)

But where the new CEO makes a difference is that leadership has caught up to where Microsoft employees already were. They can be honest, with themselves and others, about the company’s role in the world. They can stop wasting time trying to recapture those days of monopolistic dominance and instead concentrate on building great things for the future, for the many-platforms future.

That future, by the way, runs on http.

Consider the opportunity: many billions of smartphones and tablets and many apps on each device. Those apps need syncing and web services.

Though our tools and frameworks for creating apps for those devices are good, tools and frameworks for creating associated web services are still in the stone age.

They’re progressing quickly, but we’re nowhere near a web services framework that matches the spirit of Cocoa.

I don’t know anything about future Azure plans, or about how they talk about it internally — but that’s how I’d think about it. Make something as good as Cocoa is for creating iOS and Mac apps, build the services to support it, and then you’ll be the cloud supplier for all those billions of devices and apps.

Microsoft, more than Google and Amazon, knows what it means to support developers of native apps. Apple is never going to work on this — it’s not their thing, which is fine.

Microsoft has a shot at this. It’s not guaranteed. It means, though, that the company can go from protecting to building again. And that’s exciting and interesting.

PS I should also mention that, at a recent Seattle Xcoders after-meeting-at-the-Cyclops, my friend Olof Hellman was showing off Office for iPad. He’s proud of his team. They shipped — and it’s good work. iPad apps.

Justin on Build

Justin Williams attended Build 2014:

Overall though, Microsoft seems to be embracing open source in new and interesting ways that the old Microsoft never seemed to care about. Previously when they open sourced a piece of technology it’s because they were no longer interested in it. Now, key pieces of functionality that the future of the company is based on are out in the open.

Brian on Data Migration

Brian Reischl, my former co-worker at Sepia Labs — and cyborg in charge of web services — emailed me after I mentioned that to do a migration I’d “stop the world” first.

(Brian also wrote Stupid Feed Tricks, which, if you’re a fan of the horror genre, is worth reading.)

Brian’s Words Are Below

Pretty much any migration can be done without stopping the world. You migrate in steps, with double writes and double reads at some points. Here’s the general outline:

  1. Build your new data store. This could be an entirely new technology (NoSQL, different RDBMS), a new server, or even a new table structure in your existing database.
  2. Change your code to write to both the old and the new data store, but continue to read from the old store only. Be sure to capture errors from the new system so that they don’t affect your users, and log them out so you can fix them.
  3. Copy your data from the old to new store. But note that the copy process must take into account that some data will have already been written, so it can’t expect an empty store to write to. It also has to account for ongoing changes from your production system. So, for example, if you took a SQL backup of the old store and applied it to your new store, any changes that happened during the backup/restore might be lost.
  4. At this point your old and new stores should be kept in sync automatically. You want to run this way for a while to debug the new store, and ensure that the data stores remain in sync. You may want to run checks to ensure they’re kept in sync. One option is to read from both stores, compare the results and log errors, but then discard results from the new store and only use the results from the old store.
  5. Switch to reading from the new store. You can make the change all at once, or slowly (eg, 25% reads go to new store, 75% old, then 50/50, and so on). Making the change slowly will be more time consuming, but it may reveal any performance or scalability issues before they become a major problem.
  6. Eventually you should be reading only from the new store, and have developed confidence that it’s working properly. Now you can change your code to remove all references to the old store.
  7. Delete the old store.

Obviously this is far more time consuming than a “stop the world” migration. In return, it gives you a chance to develop confidence in your new system before depending on it fully, and allows you to make the change with zero downtime.

The Record Sponsorships Available

We have two spots for season 1 of The Record still available — this Friday (April 11) and April 25.

April 25 is the closing episode of season 1, and is an especially strong episode.

If you’re interested in either or both, let me know.

The Record averages over 10,000 downloads per episode. Its audience is developers and power users, mostly Mac and iOS. Price is $1,000 for one episode or $1,600 for both.

[Sponsor] Learn More iOS with Treehouse

If you’re an iOS developer, Treehouse is the perfect place to sharpen your skills.

Whether you’re just getting started or you’re currently working in iOS development, Treehouse offers an affordable and flexible education suited to you. Learn when and how you want and develop skills that can land you freelance work or your dream job.

Just getting started with iOS? Tackle the basics of C and Objective-C and then build a simple iPhone app in no time.

Not a beginner? Brush up on the latest features to iOS SDK has to offer, including collection views, view controller transitisions, UIKit dynamics and asynchronous downloading of data.

Treehouse also offers courses on 11 subjects, including lessons on PHP, Android, design and starting a business.

Sign up for a free trial and discover what you can learn.

Vesper Sync Diary #12 - Let’s Make This Change No Matter How Late

For a couple hours yesterday I was convinced I was going to change how user account info is stored — I was going to switch from SQL to table storage. (Azure table storage, a variety of NoSQL.)

Reason: I don’t know how the data model for accounts will change. I’m sure we’ll add to it, but I can’t guess what we’ll need. At the moment it’s nothing more than username and the little needed to make authentication work.

It’s not that I suspect we’ll want to store more personal information. It’s likely we’ll never even store your name, even. There’s nothing in it for us — storing things like that would just make the database larger for no good reason.

But you could imagine there might be some preferences and settings that should get synced. I just don’t know what they will be.

Table storage is a great answer for that because it’s schema-less. A single row can have an arbitrary collection of key/value pairs. (Subject to some data size limits.) Sounds perfect.

But it’s too late! Don’t do it!

Making this change would delay shipping. And it’s really, really hard to justify any delays.

But here’s the thing about web services: it’s way easier to change things before shipping than after. Once the app ships, any changes to the backend have to be made with extreme care.

And a change as large as switching account info from SQL to table storage means stopping the world, migrating the data, then restarting the world. (And testing beforehand. And after. And rehearsing. Multiple times.)

That’s not something I ever want to do. It may be inevitable some day, but I sure hope to avoid it.

So that’s how I justified making the change. A little pain now saves bigger pain later.

But then I didn’t do it anyway

As I started mapping out the implementation, I ran into a snag.

I want user IDs to be a 64-bit integer, because those user IDs are stored in the notes, deletedNotes, and tags SQL tables. SQL provides an auto-incrementing row ID which is perfect.

If I used the actual username as user ID instead, the databases would be much larger. For performance reasons we want to keep the database as small as possible. (For cost reasons too — but our attitude is that better performance pays for itself. Luckily the two values are aligned in this case.)

So I thought some more about the future. What I’m really worried about is adding to the data model, and that’s actually not that hard. We could add SQL columns and tables to store the additional data — or even use NoSQL table storage to add associated data. (Whatever is the smartest move.)

This is not nearly as bad as doing a full migration of data from one type of storage to another. If I had thought it through initially I wouldn’t have looked at NoSQL table storage at all.

But I’m glad I did, because I’ve learned some things about my environment, and that’s always a good thing.