inessential by Brent Simmons

January 2004

Mac Software Business mailing list

Just about every day I talk with people who want to start a Macintosh software business.

People have all kinds of questions: can I make money at this? How much should I charge for my app? Where should I advertise? Should I do my idea as a plugin or as a separate app? What e-commerce provider should I use? Should I get a booth at Macworld?

I have opinions on some of these questions, but there are tons of things I don’t know about, and I have questions too sometimes.

So I decided to start a new mailing list on Yahoo Groups.

The description: “This group is for small, independent Macintosh developers who want to talk with other developers about the business of Mac development. Questions on pricing, packaging, advertising, e-commerce providers, and so on are on-topic. Note that this list isn’t a vehicle for promotion: announcements and press releases are off-topic.”

(You don’t have to have a business up-and-running already: it’s for people who want to start a business too.)

Unison

Unison is a new Usenet newsreader from Panic. I haven’t downloaded it yet, but it looks great from the screenshots, which you’d expect from the Panic folks.

Orkut

I kind of got sucked into Orkut for a little while on the weekend, though I haven’t done much with it since. I’ve registered with a few of the other services too, but I haven’t used them much.

I’m curious—what do you think about these services? Do you use them? If so, is it mostly for fun, or is it an important part of your work? What things do you like about them?

What is a weblog?

Whenever we update the Sites Drawer (the list of feeds that comes with NetNewsWire) I look at a bunch of feeds and their descriptions.

Here are some descriptions from the list, just to give you a flavor:

Random Ramblings Regularly
Rants, raves, and random thoughts on life and American culture
Rants on spam, business, digital music, patents, and other assorted random stuff.
random thoughts on games, art, geek culture and living in new york.
random and occassionally coherent musings
random musings and explorations from the mind of [withheld]
random thoughts and ramblings

The people have voted! If anyone ever asks you what a weblog is, you can state, with confidence, that it’s random rants, ramblings, and musings.

It is also, on occasion, punditry. And good old-fashioned commentary is not unheard-of.

Is Dean Angry?

Here’s what confuses me:

Some people say that TV inaccurately portrays Dean as angry.

I don’t know where that comes from.

Dean built his campaign on the idea that you can use the Internet to build a grass-roots movement of people who are angry at Bush over the Iraq War, the Patriot Act, unemployment, and the environment.

It worked—he tapped into an already-existing anger. He was an angry person leading angry people. Why not? Maybe people thought they had really good reasons to be angry at Bush.

If the war isn’t just, if people died for no good cause; if our liberties are being taken away for cynical and mean reasons; if the environment and the economy are being destroyed to line the pockets of friends of Bush... if you think all that, I don’t see how you can avoid being angry.

Maybe Dean wants to appeal to a broader set of people by appearing more Presidential and less angry. That’s probably the right move to make right now. But blaming TV for reporting on Dean’s anger seems surreal to me.

The question

I’m thinking ahead to the general election.

Both parties will try to set the agenda, of course. (They’re supposed to do that.) The Republicans will talk about three things, I think:

1. Which candidate will continue to take the war on terror to where the terrorists live?

2. Which candidate will keep the IRS from grabbing all your hard-earned money?

3. Which candidate will prevent homosexuals from destroying marriage as we know it?

That’s three things, national security, economy, and values.

Lots of people have suggested that this general election will be about one of the first two, national security or the economy.

But I have a feeling that it may be about values. It will be about civil unions.

I’m remembering Dukakis. During a debate he was asked about his stand on the death penalty—but he was asked in a very personal way. He was asked what he would do were his wife murdered.

His answer should have reflected some emotion that people could relate to. Instead he gave a dry, dispassionate answer about why he’s against the death penalty. It made him seem not exactly human, and it played right into the stereotype of clinical liberals from the northeast.

I imagine Bush or a moderator asking a similar question of the Democratic nominee, but this time it’s “What would you do if your son married another man?”

How could you possibly answer this question? You support civil unions. But this is about you and your family, and people want an answer they can relate to.

It’s not as black-and-white as the question Dukakis answered. You can’t express elation or disgust, anger or joy, but you have to express something.

Looking months into the future, I can imagine where in a close election the whole thing may turn on the answer to this question.

Defending TV

You might imagine how much it excites me to see candidates use weblogs and RSS and so on in their campaigns. I love it.

But today I’m going to do something perhaps unexpected: I’m going to defend TV.

The first thing is, I register campaign coverage, no matter what the medium, as pure pleasure.

I’m a media omnivore. I enjoy reading magazines (the New Yorker ran a profile on Dean a couple weeks ago which was quite good). I enjoy reading weblogs for every candidate and point of view. I enjoy listening to the radio.

And I enjoy watching the coverage on TV.

Some of the reasons to watch TV are obvious. The candidates are not actually in Washington state, so the only way I can see them is on TV. I love to watch the debates especially, but I also like to see them talk to Judy Woodruff, Tim Russert, Charlie Rose, and others. Since I don’t get the chance to talk to the candidates every day, journalists have to stand in for me.

One of the knocks against TV is that it’s tough on candidates in seemingly bizarre ways. The talk about the “Dean scream” for instance seemed to go on and on. But there’s a point to this: I want TV to be tough on the candidates. It’s a test. It’s part of the process. I want to know how the candidate handles pressure.

The media will (or should be) tough on the President too, and these candidates want to be President.

And it looks to me as if TV reporters are equal opportunity sharks. If they smell blood, there’s a feeding frenzy. It doesn’t matter who the candidate is. (Exception: once it comes to the general election, Fox News will be clearly in favor of Bush.)

I think I detect a subtext that bothers me. It goes like this imagined conversation:

A: Hey, TV is stupid, all sound-bites, no substance.

B: Yes, well, you know you can read about the candidate on the Internet, go to meet-ups, read some great stuff in print magazines, and so on. It’s up to you find good information.

A: Oh, sure, I do all that. It’s not me I’m worried about, it’s Joe Sixpack who just watches TV that I’m worried about. He’ll just believe whatever he’s spoon-fed.

All I can say is, I know Joe Sixpack, and he’s better than that.

And if Joe doesn’t vote the way you do, it’s not because of TV, it’s because he preferred the other candidate for his own reasons.

Blaming TV is like blaming the wind for losing the football game.

Cool tune: Not Going Alone

Paul Kimball (a good friend from my college days) posted his first GarageBand composition, Not Going Alone.

I first heard it last night, and it’s still in my head this morning. It’s more demo than radio-ready, of course, but the feel of it comes through with the everyman guitar sound and especially the melody and harmonies and Paul’s singing.

Wall of shame for RSS feeds?

I get bug reports every day about feeds that don’t display correctly in NetNewsWire. At least 99% of the time it’s because the feed is invalid, but not so invalid that NetNewsWire gives up on it.

Most of the time I encourage the person who submitted the bug report to report the bug to the feed producer. I have no idea how often that happens—at any rate, it seems to have little affect. Some feeds (some popular feeds, even) stay broken.

Somebody suggested creating a wall of shame for broken RSS feeds. (I apologize: I don’t remember who suggested it.)

Part of me loves the idea. I’d create a weblog where I and other people could post reports of broken feeds.

But I don’t really like the idea because it’s so, well, negative. The big picture is to encourage syndication, not play syndication cop.

(Imagine a site saying that they took down their RSS feed because the syndication police kept bugging them. It would happen.)

Anyway, in short—wall of shame, no. But I’m still thinking about ways to boost the quality of feeds.

RSS WinterFest

If you have a Mac, and you’re listening to the RSS WinterFest webcast, how did you manage it? I can’t seem to figure it out. I get an error “Session may not exist; Requested format may be wrong or general error.”

Update: Never mind—Sheila figured it out.

Update 2: And Alex Williams sent me the correct URL to use.

When should changes take effect

Another question about Info/Inspector windows on OS X: when should changes take effect?

Some info windows (like NetNewsWire’s) have an OK or Apply button. Others update the object as you make changes in the Info window.

I think having a button is somewhat old-school: it’s very much like the days when non-document windows tended to be modal. You’d click OK to make the changes or Cancel to not make the changes.

But yet a button in a modeless window gives the user the chance to be explicit about making the changes. You can back out halfway through, and just not click the OK or Apply button.

Such a button becomes a problem, though, if you have multiple panes of info. To what does the button apply? All the panes where you made changes, or just the current pane?

(In case it’s not obvious, I’m adding some per-feed prefs—such as for feed validation—to NetNewsWire, and I’m looking at prior art in thinking about how to add extra info to the Info window. And I’m finding that prior art is a jumble: there is no clear consensus as there is with preferences windows. Ideas are of course welcome.)

Info/inspector windows on OS X

Here’s something interesting—to me, anyway, as an OS X developer.

There is a great deal of standardization on the various types of windows in OS X. For instance, preferences windows are usually either windows with a toolbar or a set of tabs. Look at the preferences for Safari, for example: you choose which preferences to edit by clicking an icon in a toolbar. Same thing with Transmit, NetNewsWire, OmniOutliner, and on and on.

But there is much less standardization for Info/Inspector windows. (I mean the windows that appear that tell you more about your selection; what you get when you hit cmd-I or shift-cmd-I.)

In no particular order, here’s a list of some of the different styles I found:

1. The Finder has disclosure triangles for expanding/collapsing.

2. OmniOutliner is similar to the Finder but instead uses a button that extends most of the width of the window.

3. OmniGraffle uses a toolbar that includes a popup menu for choosing which information to look at.

4. NetNewsWire has no toolbar or tabs or anything since it doesn’t display that much information. Transmit’s is cosmetically different but the same basic style.

5. iTunes has a big window with tabs and Previous and Next buttons. It’s actually modal, to my surprise.

6. iCal uses a drawer rather than a separate window.

7. Terminal and Interface Builder are like OmniGraffle with its popup menu, but with no toolbar.

8. xCode has tabs (like iTunes), but it’s not modal and there are no previous and next buttons.

9. BBEdit has an info window somewhat like NetNewsWire’s and Transmit’s in that it doesn’t have multiple pages, but it’s modal.

It looks to me as if two basic styles are needed, one for info windows with a little info and one for info windows with multiple panes. I think the first case is pretty well solved: it’s the second case where a convention would be useful.

I’m not proposing anything, by the way, I’m just saying that a convention would be helpful to developers and users.

Jon Udell on easy RSS subscribing

Jon Udell: What RSS users want: consistent one-click subscription.

I agree with Jon, which is why I added feed-scheme support to NetNewsWire. Other newsreaders already support this method, with more to come.

But before this becomes truly useful, three things are needed:

1. There isn’t a standard graphic yet. There should be something that’s as much a standard as the orange XML graphic.

feed graphicI asked Bryan Bell to make a graphic that says FEED, since it’s the feed URL scheme. But then it was suggested it really should say SUBSCRIBE, so it’s more clear what you’re doing. But then that would make the button quite a bit larger, out-of-step with other buttons... and there I set it aside for a while.

A standard graphic is still needed.

2. More newsreaders need to support this. Though a bunch do, with more on the way, they’ve not all announced support for this convention.

Philippe Martin added an important piece of the puzzle (on OS X) by adding an easy way to set your default newsreader by using his free IC-Switch app.

If you’re an OS X developer, and you have questions about how to support the feed URL scheme, I’ll be happy to answer your questions.

3. People with websites need to know about the convention. People who create default templates for weblog publishing systems need to know about it. This is straightforward evangelism: explain the benefits of it, give people a cool graphic and an easy howto, and ask them to add it.

Client-side validation

Mark Pilgrim, in If people won’t go to the validator, suggests running the validator on the client rather than on the web.

Last week I got a surprising amount of requests from NetNewsWire users who’d like to have a validator built in to NetNewsWire. (Many of these people are people who test and monitor their own feeds with NetNewsWire.)

What I could do—what I’d like to do—is include Mark’s and Sam Ruby’s validator in NetNewsWire. The validator would stay out of the way by default, but it would be there for people who want it.

There’s an issue, though: the validator is open source, licensed via the Python license, and I don’t know if I can include it with NetNewsWire. (License gurus please clarify.)

But more importantly, licensing issues aside, I wouldn’t do it without Mark’s and Sam’s agreement.

(In case you’re wondering about the technical details: the validator would be included unmodified, as a set of files on disk, but inside the app package, in Contents/Resources/).

XML detective work

One of the things about XML and web services is that debugging can be tricky. The whole idea is that different applications on different platforms can read and write files that each understands.

The tricky part comes when something doesn’t work. Where’s the bug?

Case study in detective work

A NetNewsWire user reported a bug to me: he had uploaded his NetNewsWire-generated subscriptions OPML file to feeds.scripting.com, and feeds.scripting.com reported an XML parsing error.

The obvious thing to think is that there’s a bug in NetNewsWire’s OPML generator. That was my hunch, so I asked for a copy of the file so I could figure out the bug and fix it.

The second obvious guess is that there’s a bug in the XML parser used by feeds.scripting.com. However, in my experience it’s usually the newer code that has the bugs, and NetNewsWire is newer than UserLand’s XML parser.

Debugging

The first thing I did was test the OPML file with a number of different tools. First I used xmllint. It didn’t report any errors. Then I tried opening it in various web browsers, since web browsers include XML parsers these days. The ones I tried (Safari, Firebird, and Internet Explorer) all reported no errors. It even opened fine in OmniOutliner.

Hmmm.

So finally I uploaded the file to feeds.scripting.com—and it worked! No XML error reported.

Let’s get this straight: it looked like the bug didn’t exist at all. The file NetNewsWire generated was fine, and the XML parser at feeds.scripting.com read it fine.

But I knew better than to stop there. Rule one of bug reports is that you don’t invalidate what the user says, just because you don’t see it yourself.

Think. What could be the problem?

The answer that satisfies

I had uploaded the file with Safari. Could the user have used another browser?

I tried uploading it with Firebird and it worked fine. Then I tried uploading it with Internet Explorer—and feeds.scripting.com reported an XML error.

Yes! My money was on the problem being a bug in IE: it somehow mangled the file on uploading it. So I wrote back to the person who reported the bug and told them what I’d found.

The mystery remains

But it turns out that they’d used Safari too.

And now, when they try it again, it works fine.

So what’s the real answer? Why was there a bug? Why does it work now?

I don’t know. But if everything is working, and no software is reporting any errors at all, then you have to let it go.

Which is, I think, an unsatisfying ending to this, but it’s a common ending when debugging XML and web services.

Steven Frank’s tip for improving your RSS

Steven Frank has a great simple tip for improving your RSS: subscribe to your own feed (or feeds).

What if your feed appears weird, or doesn’t appear at all, in NetNewsWire? Validate it. Ctrl-click on the subscription and choose Validate this Feed. (Yes, this command should also be in the main menu. In the future it will be.)

It the feed is valid, but NetNewsWire does something strange, then let us know about it: it’s mostly likely a bug.

But if the feed is not valid, you have a few options for fixing it:

1. If you’re running your own software, then you may have a bug to fix.

2. If you’re running someone else’s software, but you’re using a customized template or script to generate your feed, check your work.

3. If you’re running someone else’s software, but you haven’t customized anything, report a bug to the person or company who created your software. Be sure to include the URL of your feed, so they can validate it too and see what the problem is. (Important: be nice. Software has bugs. Most developers are conscientious and work hard at fixing bugs.)

Iowa!

The political campaign junkie in me is hugely excited for Iowa.

Sure, politics is serious stuff, but I can’t help enjoying it as if it’s a horse race, or as if it’s a season of Survivor but with smarter people and much higher stakes.

Robb Beal on ignoring indie apps

Robb Beal asks: Why do they ignore the indie apps?

He’s talking about when someone writes about user experience: why write just about Apple’s apps?

I have a couple ideas why:

1. Apple can take it.

If the piece is very critical (as it often is)—such criticism could hurt the sales and reputation of a small developer disproportionately.

Say I don’t like how app x (by small developer y) does something, and I notice that, say, Mail or iCal or whatever does the same thing. I’m going to write about the Apple app.

2. Mac users all use software from Apple, so your audience knows what you’re talking about.

Even if an article is talking about a specific app you don’t use, there’s a good chance you have it in your Applications folder and can launch it. No need to download and install anything new.

People write for an audience, and writing about Apple software has more of a built-in audience.

Update: see the comments—Buzz Andersen makes the point that indie apps are talked about all the time.

Update 2: don’t misunderstand—Robb is talking about writing about user interface, not about writing about indie apps in general.

Update 3: You might think that I’m complaining about NetNewsWire coverage or coverage of indie apps here. I’m not! I’m replying to something Robb Beal said, and saying why I think there are good reasons that people who write about user interface often write about Apple apps.

Why the compromise?

So—why the compromise position? Taken together, these reasons added up.

1. Most users don’t care about this issue; they want to read the news.

The reason NetNewsWire is forgiving with RSS feeds is because other aggregators that came before NetNewsWire were too. Did I get bug reports like “I can’t read this feed in NetNewsWire, but I can read it in x?” Yes.

2. There’s the prisoners dilemma.

Given that only one other aggregator developer was willing to require well-formed-ness for Atom, I was going to get bug reports for Atom feeds. “It works in x, but not in NetNewsWire, it must be NetNewsWire’s bug, right?”

I don’t have time to spend on bug reports for bugs that aren’t even my bugs.

3. There’s an issue of perceived bias.

The reason I wrote my own weblog software was so that people wouldn’t suspect that my software is designed to work best with system x. There’s a similar issue here: I’m on the RSS Board, and yet I don’t want people to think NetNewsWire is designed to work against Atom.

There are some ironies here, of course—one of them being that I believed that my initial choice was the best thing for Atom. (One person jokingly suggested I be removed from the RSS board for being too good to Atom.)

In fact, I still believe my initial choice was the best thing for Atom.

But my job is writing software that people like; my job is not to take care of Atom.

Compromise on Atom and XML

Tim Bray suggested a setting for how strict a parser should be with a given feed. Eric Albert suggested a smiley face (as in iCab) to distinguish well-formed from non-well-formed feeds.

I’ve gotten lots of email on the topic of Atom and requiring well-formed XML. I’ve read about this topic pretty much everywhere it’s popped up. And I conclude, reluctantly, that a compromise is the best thing.

I encourage Nick Bradbury—and any other aggregator developer who was planning on requiring XML well-formed-ness for Atom feeds—to take a similar position. Nobody should take a bullet over this.

What I plan to do in NetNewsWire is something like the following—though details may change, of course.

1. Have global and per-feed settings for requiring well-formed XML. The default will be no, to not require it. My Atom and RSS parsers will both work around non-well-formed feeds in the same way. (They’ll most likely use the same code.)

2. Have an optional indicator of some kind that displays when a feed isn’t well-formed XML. (Probably not a little frowny face, but who knows.) This feature will also be turned off by default.

3. Make the Validate this Feed command more visible. Right now it’s available only through a contextual menu; it should be easier to find and use.

If you’re a NetNewsWire user, and you don’t care about this issue, it will stay out of your way. You’ll notice an extra pref and a command in the menu bar, but NetNewsWire won’t be complaining about non-well-formed feeds all the time.

You’re not expected to care about this issue, by the way—you want to read the news. That’s totally cool.

And me, I just want to get back to work fixing bugs and adding new features.

Show Errors and Warnings

A tip for Xcode users...

One of the things about Xcode is that, when a build has errors or warnings, it’s not as obvious as it was in Project Builder.

So what I did was assign a keyboard shortcut—shift-cmd-E—to the command to show errors and warnings. It’s become automatic after doing a build: I hit that shortcut to see what happened.

El diablo

So here’s how software development goes sometimes.

I’ve been working on NetNewsWire’s error reporting when a download fails—and, suddenly, out of 193 feeds on my dev machine, there are no download errors. No DNS errors, no timeouts, just 193 out of 193 downloads.

You can tell me it’s impossible, and I’d say that normally it is—except that the Web behaves in diabolical ways when confronted with a developer with a compiler.

Atom, RSS, auto-discovery

Which, Atom or RSS, should NetNewsWire prefer when doing feed auto-discovery?

Assume that there will be a preference to let you choose which you prefer. The question is: which should be the default?

Also assume that I don’t want to put up a sheet prompting the user to choose whenever NetNewsWire finds both. That’s a bit of complexity the newsreader should take care of.

So, the question is, which? I can see good reasons for each. What do you think?

RSS comments feeds

In the past I’ve mentioned that I have an RSS feed for the comments to my site. For me personally it’s more useful than email notification of comments.

It turns out that Adam Rice has a comments feed template that works with Movable Type. (You may need to view source to see it.) There are probably other such templates, for Movable Type and other systems.

What I’m saying is—if you’re already using a newsreader, and you have a weblog, and you don’t have a comments feed, try it out, you may be surprised by how much you like it.

Update: Adam Rice wrote in the comments to say that credit should be given where due. Absolutely right. Adam’s template is based on one created by Mark Pilgrim.

More details about Postel’s Law, Atom, and NetNewsWire

Here’s something that happens all the time. An RSS feed has errors in it. It displays in NetNewsWire, but it displays incorrectly, with some weirdness. A NetNewsWire user validates the feed and sees that it has errors. Then the user emails me asking if I can fix the weirdness anyway so it would display perfectly.

Here’s the thing: being forgiving doesn’t always work. You end up not showing exactly what was intended, and users notice it, and they want to see what was intended.

It never ends, either. There’s always something new to work around. What happens is all the pressure to make things work comes down on the aggregators.

Despite all that, I agree with Postel’s Law. Atom is not a special case.

But with Postel’s Law you still have to make decisions. What’s the baseline? I’m saying that the baseline for Atom in NetNewsWire is XML well-formed-ness. NetNewsWire will probably end up liberal regarding other aspects of the spec. (I’m hoping there will be some guidelines regarding what to be liberal about and what to be strict about; but, if not, I’ll work it out anyway.)

What does this mean in practice? Some stats...

So far I’ve subscribed to 34 Atom feeds to test with.

I just tested each against the Feed Validator: 14 of these feeds are invalid according to the Feed Validator.

NetNewsWire displays all but two of them. In both cases the error is an XML parsing error.

So, obviously, there are degrees of strictness. NetNewsWire will not be as strict as the Feed Validator—not even close.

Choosing XML well-formed-ness as a baseline is not some unrealistic dictate that will prevent Atom from being popular.

I like Atom, by the way, and I’m applying this standard because I like Atom. I would be utterly pleased if in the future people would say things like “Well, you pretty much know it’s going to be well-formed XML, because it’s an Atom feed.” In other words, I’m doing what I can to make sure this future comes about, where people can do cool and creative things with Atom feeds and real XML parsers. I want Atom feeds to have the reputation of being high-quality.

You might wonder in what ways is NetNewsWire’s RSS parser forgiving of XML errors. It’s not as forgiving as you might think.

First thing to know is that it uses an XML parser (Apple’s CoreFoundation XML parser). There’s no pseudo-parser here.

It tries to be forgiving of string encoding errors with this algorithm: first it tries to parse the XML with the encoding specified in the feed (or UTF8 if not specified). If the parser won’t parse it, then it tries a few other encodings. Sometimes this gets the job done, though there may be some loss of fidelity.

It also tries to be forgiving of unencoded ampersands—but it does so in an inelegant way, and what you end up with is a feed where all the HTML tags are visible in the descriptions.

So those are also the two things NetNewsWire is not doing for Atom. In the case of the two feeds with errors, it’s possible that applying these work-arounds might have worked. But then the bugs in these feeds might never be found. (And it looks possible that in the case of one of them it’s a bug in the weblogging software that generated the feed. Everybody wants bugs like that to get fixed.)

FeedDemon and Atom

Nick Bradbury, FeedDemon developer, has posted his thoughts on the subject of Atom and valid XML: “FeedDemon will also support Atom, but if an Atom feed isn’t well-formed XML, FeedDemon will display an error rather than try to parse it.”

This reminds me of another point—NetNewsWire’s error-reporting user interface isn’t that good. I plan to improve it, to make errors more obvious. I’m also considering a preference where you could tell the RSS parser to require valid XML.

Conspiracy?

On the subject of NetNewsWire requiring that Atom feeds be well-formed XML, Mark Pilgrim wrote:

A member of the RSS advisory board -- a group whose charter explicitly states that you "advocate for RSS" -- has announced that he will use his product's dominant market position to punish his own paying customers by applying a double standard that makes Atom appear less useful than RSS.

I'm not usually given to conspiracy theories, but Jesus H. Christ, are you f$@#ing kidding me?

Good question.

I’m not trying to cut down on Atom’s chance of being popular. On the contrary—what I haven’t expressed is that I’m excited by the chance to do this right, to not have the ugly workarounds in my code that exist just to parse that minority of bad RSS feeds.

I certainly didn’t discuss my decision with other members of the RSS board.

If Atom’s popularity is dependent on whether parsers are liberal or not, then that’s a problem with Atom, or Atom feed generators, not the parsers. I don’t think that this is the case: I think Atom will be popular whether parsers are strict or liberal. And I think NetNewsWire will help Atom become popular.

What I’d like to see is a commitment to well-formed XML on the part of everybody that has anything to do with Atom. Atom has the chance to set a high standard, not just as a good spec but as good-in-practice. I bet Mark agrees with me on this.

User focus, NetNewsWire, and Atom

In a comment to a previous post, Scott Palmer suggested that I’m wrong about being strict with Atom parsing. Scott wrote, “The key is not the code, the key is the customer. You have a GREAT user focus, please keep it.”

My reply was that I’m being strict because of my user focus.

Here’s my new slogan: Don’t let bad feeds punish good users.

Every minute I spend making my Atom parser more forgiving of not-well-formed XML is a minute taken away from working on features people are asking for, things like searching and synching and everything else.

If there are some bad feeds out there—and there will be, no matter what—it’s not fair to NetNewsWire users if I spend time dealing with these feeds.

And if NetNewsWire is strict, it may make it more likely that Atom feed producers will produce good XML, which is good for NetNewsWire users.

NetNewsWire 1.0.8b1: bug fixes, bandwidth

NetNewsWire 1.0.8b1, full and Lite versions, has been posted.

We’re hard at work on 1.1, which will have a bunch of new features—but 1.0.8 is just for fixing a few important bugs. (We don’t plan to make any changes in 1.0.8 beyond what’s described below.)

1. The most important change is what happens when you click on a feed-protocol URL. In previous versions, NetNewsWire would just subscribe to the specified feed. Now it prompts for confirmation. The reason for this is because, if you use a Mozilla-family browser (Mozilla, Camino, Firebird), there was a way for a website to subscribe you to a feed you didn’t ask to be subscribed to.

2. A small bug was fixed: you can now drag feed-protocol URLs into the Subscriptions pane and have it work.

3. NetNewsWire Lite gets a feature that before now was only in the full version—it caches feeds on disk between runs. Though we thought of this as a selling-point for the full version, we decided that the Lite version should have this feature too because it reduces the amount of bandwidth it uses.

Postel’s Law, RSS, and NetNewsWire

So, you might wonder, why is NetNewsWire’s RSS parser forgiving?

When I started writing NetNewsWire, there were other aggregators already out there that were forgiving.

For example, what happens when an RSS feed has an unencoded ampersand (and is thus not-well-formed XML)? Should NetNewsWire work around the error or not?

(Unencoded ampersands are possibly the most common of errors in RSS feeds.)

At the time I was writing the RSS parser, I believed it should try to work around the error, so it could read feeds that other aggregators could read.

Was that the right or wrong choice? It’s debatable.

But I do like that since Atom is a new format—and because NetNewsWire is already-established software—I have a chance to do it differently.

Postel’s Law, Atom, and NetNewsWire

So a big topic lately has been Atom and Postel’s Law. (Postel’s law is the principle that you should be conservative in what you create and liberal in what you accept.)

Though I haven’t said it publicly before, my plan with Atom support in NetNewsWire is to make the parser stricter than NetNewsWire’s RSS parser.

There are really two levels to this, and I’m talking about the first level.

1. XML errors (unencoded ampersands, string encoding errors, well-formedness errors). If the XML parser NetNewsWire uses returns an error, then NetNewsWire will not try to work around it and parse it anyway.

NetNewsWire’s RSS parser does try to work around some of these errors; the Atom parser will not.

2. Spec errors. Let’s say that Atom 0.6 defines a new element named “foo,” but then NetNewsWire is parsing an Atom 0.5 feed and it contains the foo element. What should NetNewsWire do? Ignore it? Or use it? In this case NetNewsWire might use the element. (Assuming it’s well-formed XML, that is; assuming it passes through #1 above.)

Tim Bray wrote:

An Atom feed is going to be defined as an XML document, which means that if it’s not well-formed then it’s not Atom. All it needs is for one (I repeat, one) popular newsreader with a large installed base to enforce this policy (stop parsing and display an error to the subscriber) to turn this from de jure to de facto reality. This works because Atom doesn’t have an installed base. Let’s name names; if any one of NetNewsWire or FeedDemon or Radio adopted this policy for Atom, it would be game over; people who’ve gotten used to these aggregators are not going to switch clients because some upstream feed producer is a bozo.

That will be NetNewsWire’s policy: if an Atom feed is not well-formed XML, it’s an error.

Minor Panther irritation

Here’s something that changed in Panther that drives me nuts several times a day:

1. I often hit cmd-option-h to hide everything but the app I’m working in.

2. Then later I want to hide the app I’m working in and get back to the Finder, so I hit cmd-H.

3. I expect the current app to hide and the Finder to become active.

4. But nothing happens. (On Jaguar the current app hides and the Finder becomes active, as I expect.)

Web services for newsreaders?

If you haven’t seen feeds.scripting.com yet, it’s a place you can share your OPML subscriptions files. It has a rankings page so you can see what’s popular, you can see who subscribes to a given feed, and if you’re a member of the site there are a few other cool features.

Other web sites have some similar features, of course. (Technorati is one of my favorites.) I bring up feeds.scripting.com because it’s new and so it’s a good time to ask what you think about web services for newsreaders.

We at Ranchero don’t plan to create any web services. But we do plan to make NetNewsWire work with various newsreader-oriented web services—so here are a couple questions:

1. What existing web services interest you?

Examples: you might wish NetNewsWire uploaded OPML files automatically to feeds.scripting.com or somewhere. Or you might wish NetNewsWire made it easy to create Feedster search feeds. And so on.

And here’s the bigger question...

2. What new web services would you like to see?

I have to admit, I don’t spend much time thinking about web services, so no really good examples come to mind. And there are actually a number of good services out there already, some of them under-publicized, so it may be that a bunch of good ideas have already been done.

But I bet you can come up with some new ones.

Gruber on GarageBand

John Gruber: “What’s so cool about GarageBand is that it exemplifies the market that Apple is going after. People who want to use their computers to make cool things. People who want to be producers, not just consumers.”

I’ll add one thing to that: of all the iLife apps, iTunes is the most important but the least interesting. It’s important for obvious reasons, because of iPods and the music store and that it runs on Windows.

But it’s the least interesting because you don’t actually create music with it. You create playlists, sure, but that’s not the same level of creativity as creating the actual music (or taking pictures or making movies and DVDs). iTunes is like shopping in a really great art gallery, but it’s not the same as being a painter.

Why I’m switching to Mailsmith and SpamSieve

I’m switching from Apple Mail to Mailsmith and SpamSieve.

I had grown increasingly unhappy with Mail back in the Jaguar days. Performance was a big issue, but there were also user interface issues—the big one being that I couldn’t navigate the mailbox list via the keyboard.

Another issue was that the spam filter was getting less and less effective and I was dealing with spam by creating filters again. There’s no way I want to go back to that world. (I spent five years in Eudora creating spam filters by hand.)

But I decided to stick with Mail for a while, since Mail would be updated in Panther. And when Panther shipped, there were some nice improvements in the new version of Mail, but it didn’t specifically address my problems.

And then performance got worse. Even just checking mail became this long process. At first I thought it had to be the server. But then I downloaded the Mailsmith demo—and checking email was quick.

I also downloaded Eudora and gave it a shot. I had used Eudora for many years in the classic Mac OS. But I didn’t really like it in OS X: something about the look of it these days just rubbed me the wrong way. Just a personal taste thing, I’m sure.

So I used Mailsmith some more—and I found I liked it. It was faster than Mail. It’s very scriptable and customizable—for instance, I wanted to give some of the menu commands the same keystrokes that I was used to in Mail, and I could. Mark as Spam is now shift-cmd-J in my copy of Mailsmith.

Two other wonderful features of Mailsmith: it does not display HTML email and the text editing engine comes from BBEdit. But the very coolest feature may be SpamSieve.

Simply put: it catches my spam far more accurately than Mail ever did. Mail never came close. That’s the main thing SpamSieve has to do it, and it does it.

But it goes beyond that—you can see statistics on how well it’s doing. You can look at and edit the blocklist and whitelist. (Not something I’ve had to do, though.) My favorite of these extra features is the Show Corpus command. It shows you the words SpamSieve has seen, how often they’ve been in spam vs. good messages, and what the spam probability is. This fascinates me.

For instance, the word “terminate” has appeared 13 times since I started using Mailsmith. It has appeared in spam 12 of those times. Another for instance: any email sent to webmaster@ranchero.com has an 89% probability of being spam.

SpamSieve is a generous piece of software, in that it does its job very well but then gives you the extras that make it fun. And it’s written by Michael Tsai, another small, independent developer with a weblog.

GarageBand

I’ve been waiting 20 years for GarageBand.

Back in the early ’80s, when I was a teenager—before there were even Macs—I had an Apple II Plus with a cool piece of software named ALF II. (This was even before the sitcom named Alf.)

(Actually, it wasn’t just software: there was a card you had to plug into the computer.)

It was something like an early, early GarageBand—you could create music and save songs and play them back. The music played through some car-stereo speakers I plugged in to the ALF card.

You had to actually enter notes into a staff on-screen, using the twin roller-paddle-things that came with the Apple II Plus. (No mice yet, at least not for the mass market.) There was no way to plug in an instrument and just play.

It had up to nine tracks, and it had some basic things like setting the synthesizer settings and telling sections of a track to repeat.

It was cumbersome and the sound was totally electronic, but it worked. I made dozens of songs. I loved it.

And I’ve been waiting 20 years for the chance to do that again. Sure, I could have had this already, but for a bunch more money. Since composing music was just a hobby for me, I never wanted to spend a bunch on music software. Now I don’t have to.

This software excites me personally far beyond anything Apple has announced since OS X. More than iPhoto, iTunes, Safari, iPods, and everything else put together.

Of course, I haven’t actually tried it yet, but it only has to be half as good as the demo to be a wonderful (to me, at least) piece of software.

The challenges of synching

I predicted the other day that synching would appear in lots of newsreaders in 2004. (Some have it already, yes, but they don’t have it as I’ve defined it below.)

A good question would be: why isn’t synching already a feature of every newsreader already? It can’t be that hard, right—just read and write from a file somewhere that two copies of your newsreader can access.

I mean, what’s the hold-up? You just need something like a .newsrc. No big deal, it’s an old problem that’s been solved many times before.

Okay, so the rest of this post will be about the challenges in implementing synching.

What is synching?

First we need to define what synching is. It’s really a collection of features and requirements.

1. It’s merging, not cloning, of subscription lists.

2. It also synchs read/unread states of individual items.

3. Your newsreader uploads and downloads your synch data so you don’t have to do it manually with a browser or FTP client or whatever.

4. Your newsreader knows (or at least guesses) when it needs to download and upload synch data.

5. It works between different newsreader software on different operating systems.

I’ll take these one at a time.

Merging

Cloning is easy, merging is hard—but synching has to be merging.

For instance, imagine you have a newsreader at home and one at work. You subscribe to the same feeds—except that at work you also subscribe to some at-work-only feeds that you can’t get at home.

So when you get into work in the morning, you want to synch your data from last night at home. If it’s just cloning your subscription list, then the additional at-work-only feeds would get deleted, since you aren’t subscribed to them at home. Since we’re merging, not cloning, your at-work-only feeds do not get deleted.

But this leads to an interesting problem: what happens if you unsubscribe from a feed at home? The synching mechanism won’t delete it from your work subscription list, because for all it knows that feed could be a work-only feed.

And there’s another entire set of problems that come up because for most newsreaders the subscription list is an outline rather than a flat list. Merging hierarchies is far more difficult than merging flat lists.

Synching read/unread states of news items

This is possibly the toughest of the challenges.

In an ideal world, you can identify every item in an RSS feed with a unique id of some sort. So the synch data would be able to pair a unique ID with its read/unread status.

But not all versions of RSS have the concept of a unique ID. And, even in the versions that do have unique IDs, they’re not mandatory, and lots of feeds don’t use them. (And sometimes feeds have a terrible bug: they have unique IDs that aren’t actually unique.)

So, in the absence of a unique ID, how do you identify an item in a way that will work every single time? Answer: you can’t. There is no solution that will work every single time. (And this is why sometimes you notice in your newsreader items that are unread that you know you’ve read. They’ve been edited.)

Even if you include an entire item, all its text and links and various elements, it’s possible that the item was edited between leaving home and arriving at work.

So instead the synching has to do the best it can. Any format will probably use links and titles and whatever else so newsreaders can do a best guess. (I suspect that most developers hate situations like this, by the way, and it may be the single biggest reason why synching isn’t yet universal among newsreaders.)

Uploading and downloading

You’ll want to tell your newsreader where to save your synch data so you can get it at home and at work.

You might say, why not use .Mac? Because not everyone has an account. Because your newsreader at work might be on Windows. (And there are some other technical reasons which I’ll skip.)

Why not use FTP? Or HTTP-upload? Or...?

The answer is probably that a couple different methods may need to be made available. (FTP and HTTP-upload seem like obvious candidates, but I’m just guessing.)

But here’s the deal: I doubt that every newsreader already includes code for uploading files by the various methods people will want to use. Sure, there are libraries available, but newsreader developers will still have to write code and do a bunch of testing. Even a seemingly small thing like this still takes time and effort.

Knowing when to upload and download

This may be the easiest part.

When you launch your newsreader, it can ask if you want it to synch. It would then download your synch data from wherever and do the synch.

Similarly, when you quit, it could ask if you want to upload your synch data.

There may be more sophisticated algorithms that would make sense too, but the above is I think a good minimum.

Different newsreaders, different operating systems

This means getting a bunch of developers to agree on a format for synch data. That’s probably the easy part—the hard part will be testing to make sure X can synch with Y and Y can synch with Z and Z can synch with X.

That, by the way, is where you come in.

RSS Bandwidth and Gzip compression

There are several ways to cut down on the amount of bandwidth your RSS feeds use.

The very best way is to make sure your server returns 304 Not Modified responses. (See HTTP Conditional Get for RSS Hackers for more details.)

But today I’m talking about another method (which should be used in addition to conditional Get): gzip compression.

The idea is simple: the server compresses the feed before returning it to your newsreader. This means less bandwidth is used because fewer bytes are transferred.

One of the nice things about this is that if a given newsreader doesn’t support gzip compression, a server will return an uncompressed feed, so you’re not locking anybody out. But it turns out that lots of newsreaders do support gzip compression, so it really is worth the effort. (Ted Leung has been maintaining a list of which newsreaders support it.)

Today I (finally) turned on gzip compression for my feeds. (Here’s a handy page you can use to check for gzip compression on any URL.)

Since my feeds are generated by PHP, I used Dean Allen’s super-easy instructions for turning on gzip compression. If your feeds are static files (which is probably the best choice) then you can probably use Apache’s mod_gzip or mod_deflate.

CSS in RSS (yet again)

I’ve been getting email asking what I think about CSS in RSS, in light of Richard Soderberg’s how-to, Joi Ito’s use, and Sam Ruby’s guidance.

I have a few thoughts:

1. Some people think it’s an abomination. Other people like it. But, no matter what, it can’t be stopped. Some feeds will include CSS.

2. The people who think it’s an abomination are correct. Part of the idea behind syndication is that the reader has control over the display, not the creator.

3. The people who like it are also correct. CSS formatting allows for a little more personality.

4. The best thing would be to allow each newsreader user to choose whether to use CSS or ignore it.

5. Making it a choice is difficult if the CSS is linked-to from within the description (or equivalent) of each item in a feed.

Sam Ruby wrote: “Overall my recommendation is to not to try to tunnel this information inside the content, but to place it outside the content (either at the item or the feed level).  This would make it easier for aggregators to identify that a stylesheet is desired and to take the appropriate action.”

I agree completely.

Unfortunately, there is at present no accepted method for doing this in RSS. There is for Atom, however, and I’d like to see something similar for RSS.

Now... before I hear from people who say CSS in RSS is an abomination, and therefore there should be no recommendation for how to include CSS in RSS, let me re-iterate two things:

1. People are already including CSS in RSS, and there’s no stopping that. What they’re doing is legal RSS.

2. If Sam’s recommendation of moving it into a separate element were followed, then newsreaders would be able to give people a choice of using or ignoring CSS. (When the CSS link is in the description, a newsreader would have to parse the description and find the link and nuke it. That’s ugly stuff. You do not want your newsreader to have to do things like that.)

And, finally, there’s another thing to consider: some newsreaders present items all on one browser page. When CSS is included in an item’s description, that CSS applies to the entire page, not just to that one item. The word for that situation is havoc.