April 2019

I wrote up three new tech notes:

Why Articles and Statuses Are Separate Tables

Possible Gotchas To Be Aware Of When Working on Syncing

Syncing Implementation Roadmap

The last one is the important one — and it shows (but you knew already) that Syncing Is Not a Small Feature.

The good news is that a lot of it is infrastructure that needs to be written just once. Adding systems later will be much easier than adding the first one.

(I might be done with tech notes for now.)

I’m writing some documents about NetNewsWire’s code and architecture. In part because I believe this is a good practice for any software project — but even more because I want NetNewsWire to be completely knowable by anybody who’s interested.

It’s one thing to have an open source app, and quite another to have one that anybody — even, or especially, new programmers — can read about and come to understand.

I don’t claim that every decision I’ve made is brilliant. In fact, some are just okay, and some might be bad. That’s fine! But it’s a real, working app, and I like the idea of having explanations of how it works and why it works that way.

I’ll be posting links on this blog as I write them. Note that the tech notes are part of the repo, and they appear in the workspace tree, so everybody who checks out the code has a local copy always at hand.

Two new ones today:

Accounts — notes about the accounts system. (The “On My Mac” thing is an account; later you’ll be able to add accounts for Feedbin and other systems.)

How NetNewsWire Avoids Parsing Feeds — talks about using Conditional GET and other methods to avoid parsing feeds.

PS My favorite, though written quite a while ago, is the Coding Guidelines.

Picture Jane in her office. She gets an email from Bob every month with the latest WidgetX numbers. With that email in front of her, she double-clicks a script (or chooses one from a scripts menu), which:

1. Grabs the text of that email
2. Extracts the actual numbers (minus Bob’s signature, etc.)
3. Tells FileMaker Pro to open a specific database, and then adds those numbers to the database
4. Creates an HTML page from a template, including the new numbers
5. Uploads that HTML page to the company’s internal CMS
6. Creates and sends an email (via Mail), with the new numbers, that goes to the CEO
7. Updates and saves (on a shared folder) a Keynote presentation with the new numbers

This used to take hours, and it was prone to errors. Now it takes a minute or less — and it’s error-free.

What’s going on here? It’s AppleScript, sure — but, under the hood, it’s Apple events.

How to save a computer company

Apple events (lowercase “e” is correct) arguably saved Apple in the ’90s. Desktop publishing was huge and very Mac-centric, and that was in part because the various tools were automatable. When you can automate, you can save time and money.

This was before Mac OS X: there was no UNIX automation. No shell scripts, no pipes, no Terminal.

Instead, there was AppleScript and UserLand Frontier and an underlying bit of technology called Apple events. And apps like QuarkXPress which were scriptable.

You might say that Steve Jobs — or the iMac or the iPod or NeXT technology — saved Apple. But it’s very possible that Apple, without automation, might not have lasted long enough to get to Steve Jobs.

What Apple events are

Roughly put — they’re a way of calling a function, with parameters, in an app and getting back a result.

There are a few key things to know:

• The other app is an already-running instance: Apple events do not result in a new instance of an app (exception: if an app isn’t running, you can, conveniently, launch it via an Apple event)
• The function parameters are not limited to strings — you can send binary data (image data, for instance), arrays, etc. (This is key: the API is much richer than just a URL scheme with some text, which is what we’re limited to on iOS)
• Apple events are an underpinning of AppleScript, but they’re not limited to AppleScript — they can be sent and received programatically by any app

Mac developers will often use Apple events without even realizing it. For instance, there are methods in NSWorkspace that hide underlying Apple events: opening a URL in the browser, or opening a Finder window rooted at a given folder.

These work because the browser and the Finder respond to specific Apple events, and NSWorkspace knows how to send them.

Another key thing to know: this was early ’90s technology. It survived the transition to Mac OS X because people relied on automation — particularly, as I mentioned, for desktop publishing, but for other purposes too — and, without that automation, they would have left the Mac.

When I say “relied on,” I mean that there were companies whose profits relied, at least in part, on being able to automate away hundreds of hours of manual tasks.

And there are plenty of people today who rely on Apple events to get their work done. It’s quite common.

Apple events are an example of the genius of the Mac

The Mac, the “computer for the rest of us” — for those of us who didn’t want to deal with a DOS prompt and arcane stuff — always promised to be easy for novice users. The UI was consistent, which helped a ton. Designers were encouraged to make commonly-used features the most visible and easiest to use.

And that’s great — but the absolute genius was combining that with power-user features by way of progressive disclosure.

This meant that more power was there, but it wasn’t required in order to use the app well. But, once you needed it, you could find it. And that extra power was as well-designed as the rest of the app, so you could get the most bang for your click.

But what do you do about features that shouldn’t actually be baked into the app? What about the power to do what Jane does?

That’s where automation — Apple events — comes in. It doesn’t get in the way of the UI, but if you can find your way to Script Editor (or a similar app: there are others), you can learn how to write any feature or workflow you can dream of (as long as it’s technically possible).

Part of the genius of this is that you’re scripting the apps you already use. You’re scripting these great GUI apps that you know and love. No command line, no piping/launching/closing. Just pulling information from apps and telling them to do things.

The Mac OS X Accelerator Pedal

I have heard — I don’t know if it’s true — that some people at Apple wanted to ditch Apple events (and AppleScript) in the transition to Mac OS X. It was already a several-year-old technology, and I can see how people thought it might not fit in with OS X.

But the technology was preserved, as I mentioned earlier, because people and companies needed it. So it’s old stuff, but it’s stuck around for very good reasons.

But here’s where the Mac gets even better: OS X is UNIX, and now we can mix-and-match traditional Mac scripting with Python and Ruby and shell scripts and all the power of UNIX tools. (Note: you can call an AppleScript script from a shell script, and vice versa.)

I didn’t mention it, but Jane, in the example at the top, is using AWK to extract the numbers from Bob’s email text.

An outside observer might think Mac users just use pretty — and pretty simple — apps, and that’s the whole story. But that completely misses the power and genius of Macs.

I can’t think of another platform with the sheer level of automation power that OS X (now macOS) has.

So, after all that, a question

What happens to Jane if Mail is a Marzipan app that doesn’t respond to Apple events?

When my parents bought me an Apple II Plus — in 1980, when I was 12 — I fell in love with this entire new world that was mine. Anything that was technically possible, I could do.

I played games and I learned how to write programs in BASIC and, later, using an assembler.

It was marvelous. There were no chains on me other than the limits of the machine.

* * *

At the same time, grown-ups were sneaking in Apple II Pluses to work so they could run VisiCalc. A little later it was IBM PCs.

And the reason there was similar: the computing power they had access to was tightly controlled by the IT priesthood. Then the personal computer came along, and they had the freedom to solve problems on their own — when they wanted to, the way they wanted to.

This was a revolution.

And then, a little while later, the Mac came out. The PC was great and it democratized computers to a certain extent — but the Mac, and the ideas behind the Mac, took that much, much farther. People who were put off by a DOS prompt on a green-screen monitor had a computer they wanted to use. This extended that revolution, and Microsoft later joined in with Windows.

* * *

After a while, of course, people realized that work computers weren’t really their own. And IT departments did their jobs — as they should — and they networked these computers and worked out administration and so on.

But still, it was much better than what had been.

And — very importantly — outside work, your computer was yours. You had the freedom to use all its power.

And Macs and PCs became ever-more-powerful, with faster CPUs, bigger hard drives, more accessories, and — perhaps most importantly — better software. Things like Photoshop and QuarkXpress and Lotus 1-2-3 and Word.

And (on Macs, which I know best): HyperCard, AppleScript, and UserLand Frontier. ResEdit. INITs. All this crazy powerful stuff, which I loved.

* * *

Maybe because I lived through this — maybe because I’m a certain age — I believe that that freedom to use my computer exactly how I want to, to make it do any crazy thing I can think of — is the thing about computers.

That’s not the thing about iOS devices. They’re great for a whole bunch of other reasons: convenience, mobility, ease-of-use.

You can do some surface-level automation, but you can’t dig deep and cobble together stuff — crossing all kinds of boundaries — with some scripts the way you can on a Mac. They’re just not made for that. And that’s fine — it’s a whole different thing.

In a way, it feels like iOS devices are rented, not owned. This is not a criticism: I’m totally fine with that. It’s appropriate for something so very mass-market and so very much built for a networked world.

* * *

But what about Macs?

Macs carry the flame for the revolution. They’re the computers we own, right? They’re the astounding, powerful machines that we get to master.

Except that lately, it feels more and more like we’re just renting Macs too, and they’re really Apple’s machines, not ours.

With every tightened screw we have less power than we had. And doing the things — unsanctioned, unplanned-for, often unwieldy and even unwise — that computers are so wonderful for becomes ever-harder.

And now comes Marzipan, and I can’t help but worry that it’s another tightened screw. Will sandboxing be a requirement? Probably. Will they be able to send or receive Apple events? Will they support AppleScript? Seems unlikely. Will they be available only on the App Store? Good chance.

And you could say that’s fine — developers will use AppKit. But if AppKit is deprecated — or, effectively the same thing, perceived as deprecated by developers, or just perceived as uncool — then you get only the less-powerful Marzipan apps.

Meanwhile, the iOS triumphalists are saying that we should welcome the end of the revolution.

People will probably tell me it’s generational. And maybe it is. But if we don’t have this power that is ours, then I don’t actually care about computers at all. It meant everything.

Maurice Parker took his proof-of-concept code and moved it into the main NetNewsWire repo — and now we have a version of NetNewsWire that builds and runs on iOS.

This morning, during my bus commute, for the first time I was able to read my feeds using NetNewsWire on my iPhone. So. Damn. Cool.

* * *

A TestFlight build is still quite a way away, I think. There’s still a lot to do. You could build it and run it on your own phone right now, but I wouldn’t recommend it yet.

* * *

I’ve been working on the app for five years. Most of the work is under-the-hood stuff — the UI is always the tip of the iceberg. UI is super-important, obviously, and takes a while to write too, but it’s not the bulk of the code.

Along the way I’ve had many moments where a thing I’d written years before — because I knew I would need it — suddenly becomes useful. For instance, I wrote the OPML parser early on (one of the very first things), and it was only years later when I added OPML import to the app. (There wasn’t even an app at all when I wrote the OPML parser.)

Those moments are great. The pieces start to click together, and you realize you planned well.

And this particular moment is one of the greatest of all — because it means that all of that under-the-hood code, written over so long, was ready to run in iOS with just the barest amount of rejiggering. (We needed to deal with NSImage vs. UIImage, for instance. We needed to restructure the workspace tree to make it easier to work on the two apps.)

So: I’m continuing to work on wireframes. We’ll iterate over appearance and behavior using a running app. I’ll get back to working on syncing pretty soon (because it won’t ship without syncing).

* * *

If you’re interested in helping — testing, coding, giving feedback, helping me think things through, etc. — I’d be happy to invite you to the NetNewsWire Slack group. Just send me email asking for an invitation.

Several days ago Joe Groff posted Improving the UI of generics on the Swift forums.

This proposal felt important — but it was, I hate to admit, really slow going for me to figure it out. For one thing, I had no idea what an “existential” is.

This is no criticism: this stuff, like a scientific paper, has to be written precisely and with agreed-upon terminology. It’s just that I don’t know all the terminology.

So, on another discussion forum, some of my friends were talking about it, and two people really helped with everyone’s understanding of the proposal: Greg Titus and Tim Ekl.

And then Tim went on to write a blog post which explains all of this in a way that regular Swift users — like me! — can understand.

Go read it! Swift Generics Evolution by Tim Ekl.

I’ve used generics a little, but I didn’t like what it did to the readability of my code. Now that I understand this new proposal, I like it. Very much.

I can dream about how I’d build one of these. (I’m not going to! This is way outside my expertise, and I have other things to do.)

Instead of having it crawl blogs, I’d have it download and index RSS feeds. This should be cheaper than crawling pages, and it ensures that it skips indexing page junk (navigation and so on).

To get feeds into the system, I’d add an accounts system to the site. A registered user can do two things: 1) add individual feeds and 2) upload an OPML file of feeds (which they’d probably get from their RSS reader).

Registration (with an email confirmation loop) would be required for feed-suggesting.

And: a feed gets added to the crawl-and-index list once it’s been suggested by at least two users. This should help cut down on spam.

Accounts that are suggesting spam would be just shut down. And suggestion counts for all the feeds they suggested would be appropriately decremented. (And of course all spam feeds should be removed from the index.)

There would also have to be a way for users to report spam. And report hate speech and other things that shouldn’t be there.

* * *

Anybody should be allowed to use the system: it doesn’t require registration. The main page is like Google or DuckDuckGo — a big search field.

Registered accounts can login and see their saved searches.

Searches should be able to look for incoming links to a given site as well as search terms.

It should also provide search results via RSS — to all people, registered or not — via an easily-constructed URL, as in https://blogsearch.example.com/search.rss?q=some+search+term

Since a search results feed includes items from different feeds, it should use the RSS source element.

* * *

I don’t have any solid ideas about making this a business. I’m sure that it would be way easier to build than the search engines we had in 2005. And it should be way cheaper to maintain.

It could display ads on the website. Maybe it would offer a subscription that gets rid of the ads and perhaps offers some kind of extra features.

Years ago you could probably get VC funding for something like this. I consider it a blessing that we’re way past VC interest in RSS and blogs — you don’t need that amount of funding to get it built and running, and you wouldn’t want it anyway.

Could a person or small team run it as a labor of love, like I do with NetNewsWire? I’m not sure, because I don’t know enough about the costs involved (other than that they’ve gone down). Maybe?

One of the key would be to keep it simple. It’s just one component in an ecosystem of tools. Do search, and do it well, and that’s it.

One thing I wish we had that we used to have: blog-only search engines.

You could go and search for a hash tag. Or for links to your blog or elsewhere. Or for keywords. Etc.

It should have an API that returns RSS, so RSS reader users could set up persistent, updated searches.

There used to be a bunch of these, and now there are none that I know of.

* * *

Sure, it’s easy to search on Twitter. But you only get things posted on Twitter, and it doesn’t search the content of linked-to articles. So you’ll miss all kinds of things.

I can’t do this work myself — partly because I’m too busy with work and with other apps, and partly because I’m no expert at web-based stuff like this. I wish I could.

Jeffrey Zeldman, Nothing Fails Like Success:

On an individual and small collective basis, the IndieWeb already works. But does an IndieWeb approach scale to the general public? If it doesn’t scale yet, can we, who envision and design and build, create a new generation of tools that will help give birth to a flourishing, independent web? One that is as accessible to ordinary internet users as Twitter and Facebook and Instagram?

I think so. I hope so. My part is to write a free RSS reader — and make it open source so that other people can easily use RSS in their apps.

RSS isn’t the only part of the solution, but writing an RSS reader is in my wheelhouse. So this is what I choose.

Do I claim it’s as accessible to ordinary internet users as Twitter (for instance)? I do not. But it’s the step forward that I know how to take.

My point is: don’t give in to despair. Take a step, even if it’s not the step that will solve everything. Maybe your step is just to start a blog or open a Micro.blog account. Whatever it is — do it! :) #LetsFixThis

* * *

See also: Why Micro.blog is Not Another App.net

I noticed, starting in MacOS 10.14.4, that switching between articles in NetNewsWire was way less smooth than it had been.

NetNewsWire uses a WKWebView to display HTML. Before 10.14.4, there was no perceptible delay when switching to a new article.

With 10.14.4, there is. It’s quite noticeable, enough to be unacceptable.

I did some more detective work, and I’ve narrowed down the problem a little bit.

I’m using loadHTMLString(String, baseURL: URL?). The HTML is generated locally, and I set the baseURL because I want relative paths (especially for images) to get resolved properly.

What I found:

• If I make baseURL nil, then the latency is gone.
• If baseURL is the same as in the previously-loaded article, then the latency is gone.

Probable workaround

Here’s what I’m exploring: instead of using loadHTMLString for each article, I will:

• Call loadHTMLString with a nil baseURL exactly once, to load a template.
• To make relative paths resolve, I’ll parse and rewrite article HTML with resolved relative paths.
• To load a new article, I’ll pass data to a JavaScript function inside the template that will swap-in new HTML in the right places (title, body, byline, avatar, etc.).

This is not work I was planning, and it means taking a break from syncing to get this done — because, right now, with 10.14.4, the app is not nearly as nice to use as it was.

PS While I’m rewriting the HTML, I’ll also remove ping attributes.

On the latest episode of The Omni Show, Annette Fuller, Support Human, joins the show to talk about writing, storytelling, and helping people. And the Marvel Universe. And Harry Potter. And more.

I love doing this show — it’s now about a year-and-a-half old. This is the 37th episode. (We publish every other Wednesday.)

I love that it serves as a kind of documentation for a specific company with specific people at a specific place and time — and it’s also a good look away from the celebrities of the podcast world.

What are the people like who — sensibly! — have hobbies other than recording podcasts? (Well, they’re pretty cool!)

In an in-progress build of NetNewsWire, I turned off embedding the Swift libraries — which brings the app size down from 18.4MB to 6.9MB. Which is a huge saving, and I’m so glad we can do this now.

It reminds me of a thing I’ve been thinking about. I don’t have anything super well-put-together — just some provisional thoughts.

It seems to me that software uses electricity, and electricity use should be minimized for the health of the climate. Right? The larger the download, the more electricity it takes to download it.

And, of course, in general, the more efficient an app is, the less electricity it uses. It’s not like making NetNewsWire smaller and more efficient will change the planet — but if every app maker were to do better, it might help?

I’ll put it another way: I used to work on performance because people like apps that feel fast. (I sure do. I’m a speed junkie.) I still do that — but now I also think in terms of efficiency (which helps with performance), because I also think about the environmental cost of my software.

But, were I to bet, I’d put money on websites being the biggest-by-far abusers of electricity. Page sizes, and the amount of JavaScript that goes into them, has ballooned. It’s absolutely nuts.

(This is one of the reasons I favor static rendering. Pages are rendered once instead of on-demand — it seems obvious that this is most likely way more efficient. But if you’re still using a ton of JavaScript, maybe not.)

Should we be thinking green as we’re programming? Is it pointless? I hope it’s not — and I think we should be thinking green with everything we do, including programming.

And, along the way, we might find we make software and websites (especially) that people like better, since they’re faster and lighter.