On Beta Testing
Something that hasn’t been written about Vesper: it had the best beta test I’ve ever been a part of.
We used Glassboard, which worked very nicely for discussion. I knew it would work because we had used Glassboard to beta-test Glassboard.
The greatest beta testing group I’ve ever been a part of was the NetNewsWire beta mailing list. It was a discussion mailing list originally hosted at notabug.com (which breaks my heart to remember), and later at ranchero.com.
It had a couple dozen pretty active people and a few dozen more who didn’t post quite as often. What I would do is post super-early builds — not even betas, not even alphas, but development builds right off my machine — and we’d talk over everything.
Not just bugs but every detail large and small, every idea, every feature request, every aspect of design and behavior. Even though NetNewsWire was my thing, it was very much a collaboration with a great bunch of people. That collaboration played a major role in the quality and success of the app. I’ve thanked those people and thank them again.
From the outside it may not have looked like it, but development of NetNewsWire was always a very social experience. (Same with MarsEdit.)
And the thing I miss most about NetNewsWire is that mailing list.
* * *
This style of beta testing isn’t something I just accidentally fell into. It came from the mid-’90s. UserLand had just released Frontier’s free “Aretha” version, and there was a mailing list for people using Aretha.
I’d never been a part of anything like that. There were all these people talking about everything about the app. It was collegial and interesting and fun — and Dave Winer, the developer, was so open about everything, and he listened. It seemed like a miracle to me that such a thing could exist. I loved it. I’d been waiting all my life for such a thing, for a community like this.
I threw myself into it, then ended up working with Dave informally on some small projects, and later took a job at UserLand (which was my dream job, for sure).
(Another great mailing list at the time was Chuck Shotton’s list for MacHTTP, later named WebSTAR. I was an enthusiastic, though not at all accomplished, developer of WebSTAR plugins. I made $0 on my plugins! But I loved writing them.)
When my time at UserLand ended in 2002, and I started working on NetNewsWire, one of the first things I did was start a new mailing list, and some of my friends from the Frontier community joined me on the NetNewsWire list, and they formed the seed and the backbone of the NetNewsWire mailing list.
It might seem funny to think of beta lists as having children and grandchildren, but the NetNewsWire list was very much the child of the Frontier list, and the Glassboard and Vesper lists were the grandchildren.
* * *
Anyway: that’s how you do beta testing. Get good people and let them talk things over. And listen.
* * *
One of the rules I’ve used — which I probably got from Dave — is not to argue with “I bet lots of people are like me and want feature X,” but instead say why you specifically want feature X, or why you’d prefer some behavior or design change.
In other words: instead of just asserting that a thing would be better or more popular if done a different way, tell a story with details.
Maybe that’s not right for every beta test, but that’s what works for me. I like stories. A single person can convince me with a good story. Voting is not necessary or desired.