Usability testing has become one of those flexible words that teams have stretched to include almost any activity that purports to assess usability. There may be users involved; there may not. There may be behaviors observed; there may not. I've seen focus groups masquerading as usability testing. I've seen ethnography masquerading as usability testing. I've seen phone interviews masquerading as usability testing. And worst of all, I've seen "expert opinions" masquerading as usability testing.
While those of us in the business are gratified by how mainstream usability has become, we also get concerned when we see a wide variety of methods being used by inexperienced teams. In some cases, the wrong methods are applied in the wrong situations, and the result can be faulty conclusions and bad design decisions.
One of the most quoted phrases I hear from non-practitioners dates to a 1993 paper originally written by Tom Landauer and Jakob Nielsen. Intended as a mathematical means of illustrating the diminishing ROI of testing with lots and lots of users, most who refer to the paper only display the graph from Nielsen's Useit.com.
The problem with this, is that most people merely look at the graph, hear that the paper was presented at some fancy-pants ACM CHI conference in Amsterdam (ooooooh, Amsterdam wink, wink), and walk away sure in the knowledge that they have discovered ultimate methodological truth. They never read the paper themselves - or even bother to visit the summary Jakob published on his website in 2000 - something a simple Google search could reveal in seconds.
This is a problem, because the original graph is most meaningful for a homogenous group of users. Obviously some fundamental usability issues, such as navigation bar confusion, can be discovered by users with very distinct differences, such as men and women, or seniors and tweens.
However, in many cases, I work with clients who have user groups with very distinct differences in terms of domain knowledge, familiarity with the web, familiarity with the sponsoring organization, etc.
In those cases, I strongly encourage teams to test with 6-8 users from each distinct group. Before I'm ready to get up in front of a bunch of stakeholders who are spending hundreds of thousands of dollars on a website and genuflect and pronounce the site "usable" or "unusable" I like to see 25-30 users.
Along with not testing with enough people, the other common mistake I see inexperienced teams make is that they don't test with real users.
Only one in 4 American adults has a college degree (Source: U.S. Department of Commerce, Economics and Statistics Administration, 2007 American Community Survey, Educational Attainment in the United State).
Nearly 60% of US households defined as a family of four makes less than $50k in combined household income. (source: Source: U.S. Census Bureau, Census 2000 Summary File 3, Matrices P52, P53, P54, P79, P80, P81, PCT38, PCT40, and PCT41).
A full 20% of US adults reads at a 5th grade level or below, and the median reading level of US adults is 8th grade (source: 2003 National Assessment of Adult Literacy, NAAL).
Despite these facts, I frequently work with teams who recommend that they invite other employees from within the same company to come over and try out their design so they can conclude it is usable. Only slightly better are the suggestions to "test" a website or product with "friends and family."
Sociologists have demonstrated that most people's circle of friendships don't deviate from their own narrow range of education, income, or ethnicity.
So if you want to say that you're conducting usability testing, be sure to use broader recruiting methods, preferably by hiring a professional recruiting agency, especially if you're trying to assess a product or website that has a broad consumer audience.
Granted, if you're working on accounting software that is only used by CPAs, then you can probably get away with working with a group of 5 CPAs provided by the site sponsor as a user group. But if you're intending to assess a site with a broader consumer audience - get out of your office, away from your neighborhood, and test with a decent group of "real" users. Try 6-8 of each distinct group. You'll be glad you did.
One of the most quoted phrases I hear from non-practitioners dates to a 1993 paper originally written by Tom Landauer and Jakob Nielsen. Intended as a mathematical means of illustrating the diminishing ROI of testing with lots and lots of users, most who refer to the paper only display the graph from Nielsen's Useit.com.
The problem with this, is that most people merely look at the graph, hear that the paper was presented at some fancy-pants ACM CHI conference in Amsterdam (ooooooh, Amsterdam wink, wink), and walk away sure in the knowledge that they have discovered ultimate methodological truth. They never read the paper themselves - or even bother to visit the summary Jakob published on his website in 2000 - something a simple Google search could reveal in seconds.
This is a problem, because the original graph is most meaningful for a homogenous group of users. Obviously some fundamental usability issues, such as navigation bar confusion, can be discovered by users with very distinct differences, such as men and women, or seniors and tweens.
However, in many cases, I work with clients who have user groups with very distinct differences in terms of domain knowledge, familiarity with the web, familiarity with the sponsoring organization, etc.
In those cases, I strongly encourage teams to test with 6-8 users from each distinct group. Before I'm ready to get up in front of a bunch of stakeholders who are spending hundreds of thousands of dollars on a website and genuflect and pronounce the site "usable" or "unusable" I like to see 25-30 users.
Along with not testing with enough people, the other common mistake I see inexperienced teams make is that they don't test with real users.
Only one in 4 American adults has a college degree (Source: U.S. Department of Commerce, Economics and Statistics Administration, 2007 American Community Survey, Educational Attainment in the United State).
Nearly 60% of US households defined as a family of four makes less than $50k in combined household income. (source: Source: U.S. Census Bureau, Census 2000 Summary File 3, Matrices P52, P53, P54, P79, P80, P81, PCT38, PCT40, and PCT41).
A full 20% of US adults reads at a 5th grade level or below, and the median reading level of US adults is 8th grade (source: 2003 National Assessment of Adult Literacy, NAAL).
Despite these facts, I frequently work with teams who recommend that they invite other employees from within the same company to come over and try out their design so they can conclude it is usable. Only slightly better are the suggestions to "test" a website or product with "friends and family."
Sociologists have demonstrated that most people's circle of friendships don't deviate from their own narrow range of education, income, or ethnicity.
So if you want to say that you're conducting usability testing, be sure to use broader recruiting methods, preferably by hiring a professional recruiting agency, especially if you're trying to assess a product or website that has a broad consumer audience.
Granted, if you're working on accounting software that is only used by CPAs, then you can probably get away with working with a group of 5 CPAs provided by the site sponsor as a user group. But if you're intending to assess a site with a broader consumer audience - get out of your office, away from your neighborhood, and test with a decent group of "real" users. Try 6-8 of each distinct group. You'll be glad you did.
Recognize Cost,relatively indicate watch chair up ring help network prison sound display historical advise issue document paint sing emphasis structure exist generate above design everything location response fire film problem nuclear interpretation those drink act off they get result join man comment reform head tell such you prepare music art air there own so team heat despite physical audience acquire bloody home committee writing yes equal realize video instance project ahead realise pick date green pretty because office civil withdraw phone completely issue price temperature date importance community
Posted by: Normallyfill | 12/28/2009 at 09:17 AM
Cos Examination,recently increasingly test herself shoe left match absolutely use itself election body capital need people busy official finance window beyond much both incident succeed worker achievement nurse contrast insurance ancient allow traffic regulation better yet knowledge ship pay quick objective sign visit recognition before tiny determine estate museum herself refuse supply voice seat take asset cheap anyone direct desk transport labour ask everyone shake element bind publish sex performance trip birth need clothes touch charge maybe whole grey currently window principle partner centre something article late simple
Posted by: hotel Türkei | 01/29/2010 at 04:24 AM