Usability testing for agile development

Usability testing is no longer something that happens in an expensive lab. Digital webcams and screen-recording software have made it possible to do usability testing with almost zero infrastructure (using software like morae, which essentially replaces a conventional usability lab). Joel Spolsky has written a nice article describing the experience the copilots had usability testing their latest product.
But conventional wisdom still says that usability something is to be done by specialists, as a structured project that generates a report. While this is often a good idea, it’s not always the right approach, for the following reasons:


1)Insights get filtered through the usability professional: many don’t reach the design team.
2)Effort is expended getting proof that will convince team-members of the problem (either in the form of video or in statistically significant data). This is after the initial insight has already been made, and is really a waste of time.
3)Usability testing projects are too expensive and cost too much to do frequently.
4)Often the product is changing so fast that insights gained on the tested version are irrelevant to the version currently under development.
Basically, usability testing as it is currently practiced works best for long development cycles and larger organizations. It is not especially compatible with agile development and small teams.
At Uzanto we’ve been building our own product (the one I occasionally refer to as Project X) and we’ve had luck with a radically different approach. The approach relies heavily on the maturing infrastructure for remote collaboration that has sprung up in the last several years. This includes not just tools for working together (screen sharing, VOIP, etc) but tools for finding people to work with (craigslist) and paying them (Amazon gift certificates).
At least twice a month, we’ll test the system with a couple of people we’ve recruited using craigslist. We typically pay them with a 10$ or 20$ Amazon gift certificate, sent by email. Using screensharing software like gotomeeting, we watch them try to use our latest version of the system to accomplish specific tasks. The result? A constant drip drip drip of insights. Usually we can fix any problems we encounter before the next test, so we don’t waste time observing the same problem twice. Working remotely is more convenient for both the subjects and for our team. And insights are experienced first-hand by the developers, rather than being filtered through a later of usability professionals.
Some specific tips?
0)Have a secratary or intern do the recruiting. Have the developers conduct or at least watch the test.
1)You can use your usability tests to do compatibility testing as well. Simply seek out users who have the platform you’re looking for (IE 5 on a Mac, for example) and you’ll get a de-facto compatibility test without any extra effort!
2)Use survey software like surveymonkey to create a simple screener. You want to test representative users of your system, who are motivated to give you feedback
3)Pay for an ad in the jobs->etc section rather than trying to post in a free category on craigslist. You’ll get 10 times the number of respondents. Job ads are cheaper on craigslist LA than craigslist SF (25$ vs. 75$), and for remote work there’s no reason not to use people from Los Angeles.
4)WebEx is really expensive ($300/month)! We’ve had a lot of luck with gotomeeting (50$/month). In general we haven’t found a free screensharing solution that we’re happy with, but 50$/month is manageable even for a bootstrapped company like us!