inessential by Brent Simmons

April 2017

Frontier Diary #7: Pretty Much Everything Throws

A script can throw an error, either intentionally (via the scriptError verb) or by doing something, such as referencing an undefined object, that generates an error.

OrigFrontier was written in C, which has no error-throwing mechanism, and so it worked like this: most runtime functions returned a boolean (for success or failure), and the return value was passed in by reference. If there was an error, the function would set a global error variable and return false. The caller would then have to check that global to see if there was an error, and then do the right thing.

This was not unreasonable, given the language and the times (early ’90s) and also given the need to be very careful about unwinding memory allocations.

But, these days, it seems to me that Swift’s error system is the way to go. There’s just one downside to that, and it’s that I have to do that do/try/catch dance all over the place, since pretty much any runtime function can throw an error.

Even the coercions can throw, so last night I changed the Value protocol so that asInt and so on are now functions, since properties can’t throw (at least not yet).

The extra housekeeping — the do/try/catch stuff — kind of bugs me, but it’s honest. I considered making script errors just another type of Value — but that meant that all those callers have to check the returned Value to see if it’s an error, and then do the right thing. Better to just use Swift’s error system, because it makes for more consistent code, and it makes sure I’m catching errors in every case.

It also means I’m not multiplying entities. A Swift error is a script error, and vice versa.

* * *

Working on this code is like applying the last 25 years of programming history all at once.

A completely different type of error is a bug, and I’m certain to write a bunch of them, because that’s how programming goes.

That’s where unit tests come in. Frontier has long had a stress-test suite of scripts — you’d launch the app, run that suite, wait a while, and see if there are any errors. This was critically helpful.

But OrigFontier didn’t have unit tests at the C code level. The new version does. (Well, I’ve started them anyway.) This means I can more easily follow Rule 1 — the no-breakage rule — and can also more easily follow Rule 1b — the don’t-break-Dave rule.

PS I’ve added a collection page for the Frontier Diary, as I did with earlier diaries. There’s a link to it in the footer of every page on the blog.

My Microblog

I’m on Manton‘s cool new microblogs system. Here’s where you can follow me, once you’re on the system: http://micro.blog/brentsimmons.

And here’s my microblog: http://brent.micro.blog/. (Which you can read using RSS, whether you’re on the system or not.)

I wrote about three-quarters of my own single-user microblog system — and then stopped because I didn’t feel like running a server and because Manton’s service is so good.

Frontier Diary #5: Values and Progress on the Language

I put the Frontier repository up on GitHub.

(The build is currently broken. This is bad discipline, but since it’s still just me, I forgive myself. Sometimes I run out of time and I just commit what I have.)

The repo has my new code and it also contains FrontierOrigFork, which is the original Frontier source with a bunch of deletions and some changes. The point is to give me 1) code to read and 2) a project that builds and runs on my 10.6.8 virtual machine.

The original code is in C, and the port is, at least so far, all in Swift. In the end it should be almost all in Swift, but I anticipate a couple places where I may need to use Objective-C.

Here’s one of the Swift wins:

Values

Since Frontier contains a database and scripting language, there’s a need for some kind of value object that could be a boolean, integer, string, date, and so on.

Original Frontier used a tyvaluedata union, with fields for the various types of values.

This is a perfectly reasonably approach in C. It’s great because you can pass the same type of value object everywhere.

Were I writing this in Objective-C, however, I’d create a Value protocol, and then create new value objects for some types and also extend existing objects (NSNumber, NSString, etc.) to conform to the Value protocol. This would still give me the upside — passing a Value type everywhere — while reducing the amount of boxing.

But: this still means I have an NSNumber when I really want a BOOL. Luckily, in Swift I can go one better: I can extend types such as Bool and Int to conform to a Value protocol.

This means passing around an actual Bool rather than a boxed boolean. I like this a ton. It feels totally right.

Other topic:

Language Progress

I’m still in architectural mode, where I’m writing just enough code to validate and refine my decisions. A couple days ago I started on the language evaluator — the thing that actually runs scripts.

It works as you expect: it takes a compiled code tree and recursively evaluates it. It’s not difficult — it’s just that it’s going to end up being a fair amount of code.

I’ve done just enough to know that I’m on the right path. (The Swift code looks a lot like the C code in OrigFrontier’s langevaluate.c. See evaluateList, for instance.)

The next step is for me to build the parser. I thought about writing a parser by hand, because it sounds like fun, and it would give me some extra control — but, really, it would slow me way down, so forget it.

OrigFrontier generated its parser by passing a grammar file — langparser.y — to MacYacc (there was such a thing!), which generated langparser.c.

I’ll do a similar thing, except using Bison (which is compatible with Yacc). Or, possibly, using the Lemon parser generator instead. Either way, I’ll want the generated code to be Objective-C. (Well, mostly C, but with Objective-C objects instead of structs.) (I don’t know of a generator that would create Swift code.)

This is completely new territory for me, and is exciting.

(Almost forgot to mention: I’ll need to write a tokenizer. This means porting langscan.c. I’ll need to do this first, since the parser generator needs it. So this is the real next step.)

Frontier Diary #4: The QuickDraw Problem and Where It Led Me

In my fork of Frontier there are still over 600 deprecation warnings. A whole bunch of these are due to QuickDraw calls.

For those who don’t know: QuickDraw was how, in the old days, you drew things to the Mac’s screen. It was amazing for its time and pretty easy to work with. Functions included things like MoveTo, LineTo, DrawLine, FrameOval, and so on. All pretty straightforward.

These days we have Core Graphics instead, and we have higher-level things like NSBezierPath. QuickDraw was simpler — though yes, sure, that was partly because it did less.

* * *

I was looking at all these deprecation warnings for QuickDraw functions and wondering how I’m ever going to get through them.

I could, after all, convert all or most of them to the equivalent Core Graphics thing. But sheesh, what a bunch of work.

And, in the end, it would still be a Carbon app, but with modern drawing.

* * *

So I thought about it from another angle. The goal is to get to the point where it’s a 64-bit Cocoa app. All these QuickDraw calls are in the service of UI — so why not just start over with a Cocoa UI?

The app has some outlines (database browser, script editor, etc.), a basic text editor, and a handful of small dialogs. And all of that is super-easy in Cocoa.

Use an NSOutlineView, NSTextView, and some xibs for the dialogs, and we’re done. (Well, after some work, but not nearly the same amount of work as actually writing an outliner from scratch.)

In other words, instead of going from the bottom up — porting the existing source code — I decided to start from the top down.

I started a new workspace and started a new Frontier project: a Cocoa app with Swift as the default language.

Then I looked at the existing source and thought about how to organize things. I came up with this:

  • Frontier — App UI
  • UserTalk.framework — the language
  • FrontierVerbs.framework - the standard library
  • FrontierDB.framework — the object database
  • FrontierCore.framework — common utility functions and extensions

I like using frameworks, because it helps enforce separation, and it helps in doing unit testing. And frameworks are so easy with Swift these days.

Hardly any of this is filled-in yet. I’ve got the barest start on FrontierVerbs. Ted Howard, my partner in all this, is taking UserTalk.framework and FrontierDB.framework.

In the end, it’s possible that no code from the original code base survives. Which is totally fine. But it also means that this is no quick project.

At this point I should probably put it up on GitHub, since it’s easier to write about it if I can link to the code. I’ll do that soon, possibly on the weekend.

Frontier Diary #3: Built-in Verbs Configuration

Frontier’s standard library is known as its built-in verbs. There are a number of different tables: file, clock, xml, and so on. Each contains a number of verbs: file.readWholeFile, clock.now, and so on.

Most of these verbs are implemented in C, in the kernel, rather than as scripts. At the moment, to add one of these kernel verbs, you have to jump through a few hoops: edit a resource, add an integer ID, add to a switch statement, etc. It’s a pain and is error-prone.

So I want to re-do this in Swift, because I’m all about Swift. And I want adding verbs to be fool-proof: I don’t want to remember how to configure this every single time I add a verb. Adding a verb needs to be easy.

My thinking:

  • Give each table its own class: ClockVerbs, FileVerbs, etc.
  • Have each class report the names of the verbs it supports. These need to be strings, because we get a string at runtime.
  • Run a verb simply by looking up the selector, performing it, and returning the result.

To make things easy and obvious, I think it should work like this: the selector for a given verb is its name plus a parameter. Then there’s not even a lookup step.

Each verb will take a VerbParameters object and return a VerbResult object.

dynamic func readWholeFile(_ params: VerbParameters) -> VerbResult

The flow goes like this:

  1. We have the string file.readWholeFile.
  2. We see the file suffix and so we know we need a FileVerbs object.
  3. We check fileVerbs.supportedVerbs (an array) to see if readWholeFile is in the list. It is.
  4. We construct a selector using the readWholeFile part of the string and we add a : character: NSSelectorFromString(verbName + ":")

This is great! We’re almost home free. Then we run the verb:

if let result = perform(selector, with: params) as? VerbResult {
	return result
}

That doesn’t work. We get:

Cast from 'Unmanaged<AnyObject>! to unrelated type 'VerbResult' always fails

Nuts.

* * *

It was so close.

In Objective-C this would have worked. And obviously, apparently, I still think in Objective-C.

I investigated some other options. At one point enums were abused, because there’s always, in Swift, an enum-abuse step. But everything I tried was more code and was more error-prone, and my goal here is to improve the situation.

I think, in the end, I’m going to do something that looks kind of ugly: a switch statement where the cases are string literals.

switch(verbName) {
case "readWholeFile":
	return readWholeFile(params)
…
}

“Nooooo!” you cry. I hear ya.

My experience as an object-oriented programmer tells me this: if I write a switch statement, I blew it.

And my experience as a programmer tells me that string literals are a bad idea.

But the above may actually be the easiest to configure and maintain. Each string literal appears only in that one switch statement and nowhere else in the code. And the mapping between a verb name and its function couldn’t be more clear — it’s right there.

(Yes, instead of using a string literal, I could create a String enum and switch on that. But that’s actually more code and more room for error. I’m going to have to type those string literals somewhere, so why not right where they’re used?)

It does mean that readWholeFile appears three times in the code (the string literal, the call, and the function itself), and in an Objective-C version it would appear only twice (in a supportedVerbs array and the method itself).

But. Well.

I’m torn between shuddering in abject and complete horror at this solution and thinking, “Hey, that’s pretty straightforward. Anybody could read it. Anybody could edit it.” Which was the plan all along.

And I get to stick with Swift, so there’s that.

But, sure as shootin’, some day someone’s going to come across this code and say, “Brent, dude, are ya new?” And I’ll send them the link to this page.

* * *

Update the next day: well, the performSelector thing would work, if only I’d known about Swift Unmanaged objects.

Joe Groff told me how this works.

Here’s the gist: the Unmanaged<AnyObject> just needs to be unwrapped by calling takeRetainedValue or takeUnretainedValue. Once unwrapped, it can be cast to VerbResult.

All this means that I can use my original design, which is great news.

* * *

Update April 25, 2017: I ended up using enums after all. See MathVerbs.swift for an example.

Frontier Diary #2: Two Good Ideas that Aren’t Good Anymore

Strings in Frontier are usually either Pascal strings or Handles.

You probably don’t know what I’m talking about. I’ll explain.

Pascal Strings

Frontier is a Mac Toolbox app that’s been Carbonized just enough to run on OS X. You may recall that the Mac Toolbox was written so long ago that the original API was in Pascal. That Pascal heritage lived on in many ways, even after everyone switched to C — and one of those ways was Pascal strings.

A Pascal string is n bytes long, and the first byte specifies the length of the string, which leaves the rest of the bytes for the actual string. Str255 was probably most common, and certainly is most common in Frontier, but there are also smaller sizes: Str63 and Str31, for instance.

Unlike C strings, they’re not zero-terminated, since there’s no need to calculate the length: you always know it from that first byte.

You create a literal Pascal string like this…

Str255 s = "\pThis is a string";

…and the compiler turns the \p into the correct length (16 in this case).

Now, I bet you’re saying to yourself, “Self, those Pascal strings are too small to be useful.”

But consider this: every menu item name can fit into a Pascal string. You can fit a window title or a file name into a Pascal string (in fact, memory suggests that file names were even shorter, were Str31 Pascal strings). Any label or message on any bit of UI is probably short enough to fit into a Pascal string. (Especially if you assume English.)

So for GUI apps these were terrifically useful, and the 255-byte limit was no problem. (You can fit a tweet in a Pascal string, after all, with a bunch of room left over. [Well, depending on the size of the characters.])

Frontier still uses them internally a ton. (For some reason, in the Frontier code, Str255 strings are called bigstring, which sounds ironic, since they’re so small, but I think it was to differentiate them from even smaller Pascal strings such as Str31.)

You might ask what the text encoding was for these strings.

“Text whatzit?” I’d reply. “Oh, I see. Just regular.” (MacRoman.)

It was a good idea, but its time has come and gone. We have better strings these days.

Handles

Frontier includes a scripting language and a database, which means it certainly has a need for strings much larger than 255 bytes.

It also needs heap storage for other things — binary data, structs, etc. — that could be much larger than 255 bytes.

Enter the Handle. A Handle points to a pointer that might move: the memory you access via a Handle is relocatable.

Which sounds awful, I know, but it was a smart optimization in the days when your Mac’s memory would be a single-digit number of megabytes, or even less than that.

Here’s the problem: your application’s heap space can become fragmented. It could have a whole bunch of gaps in it after a while. So, to regain that memory, the system could compact the heap — it would remove those gaps, which means relocating the memory pointed to via a Handle.

This is better than running out of memory, obviously. But it means that you have to be careful when dereferencing a Handle: you have to actually lock it first — HLock(h) — so that it can’t be moved while you’re using it. (And then you unlock it — HUnlock(h) — when finished.)

Handles are also resizable — SetHandleSize(h, size) — and resizing a Handle can result in it needing to move, if there’s not enough space where it is. Or other Handles might move. You don’t ever know, and don’t care, and you think this is elegant because the system handles it all for you.

All you have to deal with is an additional level of indirection (**h instead of *p), locking and unlocking it when needed, and disposing of it — DisposeHandle(h) — when finished. (No, there’s no reference counting, slacker.)

Nowadays, on OS X, Handles don’t ever move and there’s no heap compaction. So there’s no reason for them whatsoever. And they are, as expected, deprecated.

Nevertheless, Frontier, a Mac Toolbox app written in C, uses Handles everywhere.

(I remember being shocked, when I first started learning Cocoa 15 years ago, that there were no Handles. It seemed incredibly daring that objects were just pointers. It made me nervous!)

The Size of the Job

Almost all the Mac APIs that Frontier uses are deprecated. That’s one thing.

But it’s worse than just that: the ways Frontier handles strings and pretty much every single thing it stores on the heap are also deprecated.

So: what to do?

The end goal is a Cocoa app, which means I’ll be able to use Foundation, CoreFoundation, and Swift data types: NSString and Swift String, for instance. There are a number of different structs in the code, and those will be turned into Objective-C and Swift objects and Swift structs.

The tricky part, though, is getting from here to there. I think the first step is to start with Objective-C and Foundation types and use them where possible. I can do that without actually turning it into a Cocoa app (the app will still have its own WaitNextEvent event loop and Carbon windows) — which means I’ll have to bracket all Objective-C code in autorelease pools, and I’ll have to use manual retains and releases. I’m not sure how far that will get me, but it will get me closer.

PS Here are a couple articles by Gwynne Raskind on the Mac Toolbox you might enjoy: Friday Q&A 2012-01-13: The Mac Toolbox and The Mac Toolbox: Followup.

CocoaConf Near WWDC

There are a bunch of things happening near WWDC this year. Me, I’ll be at CocoaConf Next Door. I’m not preparing a talk, but I’ll probably be on a panel. And hanging out.

Check out the speakers list, which includes Omni’s own Liz Marley. And a bunch of other people you totally want to see — Manton Reece, Jean MacDonald, Laura Savino, and plenty more.

Also… AltConf and Layers will be near WWDC. If you could be in three places at once, you would. Well, four, including WWDC itself, I suppose. :)

OmniOutliner 5.0 for Mac

I’ve been on the OmniOutliner team for over a year now. Though we don’t have positions like junior and senior developer, I enjoy calling myself the junior developer on the Outliner team, since I’m newest.

I may be a new developer, but I’m not a new user — I’ve been using the app since the days when OmniOutliner 3 came installed on every Mac.

Every time I start a talk, I outline it first. I organize the work I need to do in my side-project apps in OmniOutliner. And — don’t tell the OmniFocus guys, who are literally right here — sometimes I even use it for to-do management in general. I’d be lost without a great outliner.

Anyway… there’s a new version: OmniOutliner 5.0. It’s my first dot-oh release at Omni, and I’m proud of it and proud of the team.

As is common with our apps, we have two levels: a regular level and a Pro level. The regular level is called “Essentials” and is just $9.99. There’s a demo so you can try it out first.

It syncs with iOS and with other Macs, by the way. Sync is free. And of course it comes with extensive documentation, and Omni’s awesome support humans are standing by.

Get it while it’s hot!

Frontier Diary #1: VM Life

It’s been years since I could build the Frontier kernel — but I finally got it building.

It’s really a ’90s Mac app that’s been Carbonized just enough to run on MacOS, but it’s by no means modern: it uses QuickDraw and early Carbon APIs. It’s written entirely in C.

I got it building by installing MacOS 10.6.8 Server in VMWare. Installed Xcode 3.2.6. And now, finally, I can build and run it.

What is Frontier?

Frontier — as some of you know — was a UserLand Software product in the ’90s and 2000s. I worked there for about six years.

The app is a development environment and runtime: a persistent, hierarchical database with a scripting language and a GUI for browsing and editing the database and for writing, debugging, and running scripts.

The Nerd’s Guide to Frontier gives some idea of what it’s like, though it was written before many of the later advances.

Maybe you’ve never heard of it. But here’s the thing: it was in Frontier that the following were either invented or popularized and fleshed-out: scripted and templated websites, weblogs, hosted weblogs, web services over http, RSS, RSS readers, and OPML. (And things I’m forgetting.)

Those innovations were due to the person — Dave Winer — and to the times, the relatively early web days. But they were also in part due to the tool: Frontier was a fantastic tool for implementing and iterating quickly.

The Goal

The high-level goal is to make that tool available again, because I think we need it.

The plan is to turn it into a modern Mac app, a 64-bit Cocoa app, and then add new features that make sense these days. (There are so many!) But that first step is a big one.

The first part of the first step is simple, and it’s where I am now: mass deletions of code. Every reference to THINK_C and MPWC has to go. All references to the 68K and PPC versions must go. There was a Windows port, and all that code is getting tossed. And then I’ll see the scale of what needs to be done.

(Note: my repo is a fork, and it’s not even on the web yet. The code I’m deleting is never really gone.)

I’m doing a blog diary on it because it helps keep me focused. Otherwise I’m jumping around on my side projects. But if I have to write about it, then I’ll stay on target.