Usability, When All is Said and Done

I think I am one short in the journal's so here goes.

This quarter had been fleeting and enlightening. In a very short time, I feel secure in being able to plan, execute and summarize a solid usability test for a medium to small site. There are several reasons for this. My role in the group was to compile information. I got to put it all together and this helped me get a complete vision of the process. Putting each part of the plan together (since we each composed a different part) gave me a good overview of the entire idea. In the testing, I observed and executed an entire test myself. Here's the rub with this one, although I got to run the show on my own, I don't think I did such a great job. I did not push my participant to keep going. I let him give up. Of course I saw this after the fact so couldn't change it, but I got a clear message on how to facilitate well. Leading the participant to complete the task is difficult so to not give the game away. Not sure how I would've done it differently but would have tried to encourage him to open more doors.

It was in interpreting the data that you really see where facilitator errors are made. It's very difficult to pigeon-hole the participants but there obvious holes to put them in. For instance, the issue we had with finding stats for a particular player, our Task A. This proved to be a pain in the ass, pardon the expression. Even my participant, who is a Braves fan, had trouble with this. At first, he thought he had it, but then I asked him, 'do you really think Gonzalez had 40 hits is yesterday's game?'. Only then did he really look at the page to see where he was and only then did he see the option for 'individual games' and only then did he find it. It very backwards in how he found it and I still think I led him too much. I am have factored in that he is a Braves fan and had been to the site before. I pigeon-holed him into already knowing where to go to complete the tasks.

As an observer in the tests with our two non-USA born participants, I found it...interesting. We had a big discussions about testing non-Braves fans and decided to go with that option. We didn't really focus on whether or not to test non-BASEBALL fans. Not sure we made a good choice there. With hind sight being 20/20, I don't think it was a good idea. I think the participants were frustrated beyond a reasonable point for this experience in that they had no idea what we were talking about. And trying to put them in a 'real-world' situation such as, a bunch of guys who know nothing about baseball suddenly decide they want to know how many hit over attempts Gonzalez had in the last game, doesn't seem likely to me. I would reject them as participants now and look for other potential users. At least users who know something about baseball. I don't even see a newbie going that deep into a site for some time after they have paying around there for awhile.

Despite the technical difficulties we had with two of our tests, I don't think I would have used Victor's nor Carlos' videos in the final result. I don't think they were representative of the typical nor potential visitor to the site. So, I guess it worked out that their videos were incomplete. Carlos' cut off about 7 minutes in (which was almost at the end of the first task) and Victor's had no audio and stopped after a little over 30 seconds. The technical issues with the lab didn't effect me until I was responsible for compiling the videos. Since I only had two to chose from, I did not make a highlight reel, rather an edit of each task I felt would best represent all the participants.

I mentioned before that I executed an entire test and I would like to go into more detail about that. On my laptop, I have a program called ScreenFlow. It is mostly used for screen capture and I found it via my video class this quarter. I used this program to execute the test with my nephew. As you can see, you can select to both capture the screen and record the user. I think in those terms it worked great! Where I was at a loss for this test, was my failure to take good notes and play the observer while facilitating. Having those HTML files the testing software gives you is a great help. Since I lost the AV from two tests, it was the HTML files gave me the data I needed. Also, since I was there to watch them, that was a big help as well. My correction for giving the test on my laptop, take much better notes.

Bringing all this together for the Results report was also huge. Recapping the plan was basic but interpreting the data became almost mute because it was so obvious. This confused me because for some reason, I thought we would uncover something no one else had seen, and maybe we did, but I had these illusions of grandeur even though here was nothing magical about it. Still, not sure why I felt I would be in some Disney movie with all this but thought there be more to it. Don't get me wrong, I do feel successful in this venture and believe we did a class-A job, but feel like there should have been little birds, bunnies and deer running around being happy. Maybe I'm just tired.

This brings me to the team. What a great team. This is the first group that I have enjoyed working with because everyone did their job! Amazing how that works, isn't it? Thankfully I had a team because I know I could not have done this on my own. This was a big job. We each had a role, we each executed that role and we all came together in the end. I think it was Alex that brought us together and I was flattered to be asked to join the them. And you know what's even stranger? None of this felt like 'homework', it actually felt like a 'job'. Does that make sense? We had a goal. We had a site to 'fix', if you will, and I think we did that! Being in this group has changed my attitude about working in group projects, I will be more adamant about who I work with in the future, now that I've been on the right side of the tracks.

So let's wrap this up before I write a novel. The first day of class you sad something to the effect of 'usability is everywhere' and you gave the bus accident as an example of usability in the world. I saw that then but I see it differently or deeper now, I guess. There is usability not only in instructions, directions, web sites but also working with people. I know this sounds shifty, but there can usability with people, in a positive way. Yes, I am familiar with the phrase 'you used me' and that is not what I am talking about. I am referring to the usability of a team or group of people. Usability can be also be using your talents and strengths to contribute to the task or to get the best out of another. In all honesty, I wasn't really sure where I was going with this entry until I got to here. After reviewing the work this team delivered, when all is said and done, it was the team that made the usability project, usable.

Seven Common Usability Testing Mistakes

Since we are in the process I thought this would be a good article to review. Seven Common Usability Testing Mistakes discusses how many usability problems stem from errors in design and organization. The author, Jared M. Spool, travels to work with many companies and has observed theses mistakes regularly. I guess you could say he tested the usability testing process.

1. Do You Know Why you're Testing? This is difficult question to answer. I know we discussed this answer quite a bit when designing our test plan. He discusses that testers often use the tests to determine how users 'feel'. That is not usability, that is user experience. The main focus of the testing should be when and where the user begins to feel frustrated because they are unable to complete the task. I completely agree. What the participant feels is important in these tests but I see it as a secondary measure. Record the feelings but view them as a side-effect of the procedure.

2. Not Bringing the Team Together. This to me this goes without saying. If everyone isn't there paying attention, then not everyone will be on the same page. When you are not on the same page, communication suffers. I like his comparison the the 'Telephone'. Played that a lot when I was a kid.

3. Not Recruiting the right Participants. We discussed this in class and I believe we all agreed that friends and family members where not good choices. The ideal choices are those who fit the demo and come in to the process cold. I like what he says here about what can happen with the wrong participants, do they know too much so they won't experience the problems real users would find, is one issue. The next contradicts what we discussed to a degree. Spool says focusing too much on demographics and ignoring individual subtleties could lead to false conclusions. He says you should focus on user attributes like behavior. How will the users each behave and what attributes cause this distinction? I like that, too. However, I think it will narrow your field too much and become too niche oriented.

4. Not Designing the Right Tasks. This is huge. I had several problems trying to design tasks the first time I tried it in my Production Team class. And I fell into the issues of 'leading the participant', using the same language, etc. Spool touches on the first thing I thought of when considering testing, why wouldn't the user go to the search box? In a test he helped with for Ikea, this is exactly what happened. They were asked to find a bookcase, so they ALL went to the search field and typed in 'bookcase'. So the task was changed to become more of a story telling them what they were going to put in the bookcase, how much stuff they had and they called is shelving, not bookcases. This is the approach I would like to use. When originally wrote the tasks in the class i mentioned before, I tried to do that. 'You have an 8-year old daughter and you to register her for swimming lessons. Find the form.' I was half-way there. It is completely accurate that task design will effect your results. I don't see how this could be ignored. 'Context of use'. I will remember that and use it.

5. Not Facilitating the Test Effectively. Spool says the facilitating is learned skill. He says he has never met anyone that was adept to it. A good one will draw the right information without giving away the goods. They keep it interesting not only for the participant, but the observer as well.

6. Not Planning How You'll Disseminate the Results. Here he focuses on the getting the results to the right people so that action can be taken.  I would agree that reports are often not read. You may give them out, but they more than likely will not get read. He proposes review sessions after each test, an email discussion list about the test and interactive workshops. I like these ideas! I think theses are great ways to make sure everyone gets the results.

7. Not iterating to Test Potential Solutions. Although testing may illuminate problems, it fails to tell us how to solve them. There are several solutions for any given issue, but which one do you use? You will need to test again. Spool says paper mock-ups will work for this round. Do not cut corners. If your 'solution' is more confusing than the original problem, it could end in disaster. Follow-up testing is a must to make sure you have chosen the right conclusions.

I learned a lot from this article. most of it we discussed in class but it nice to read how these things work in the world.

What's In A Number?

A lot according to who you talk to. In fact, the question reminds of the fablesd, "how many licks does it take to get to the center of a Tootsie Roll Tootsie Pop?". The wise answered, "Three".

Looking of the world of sample testing, there are several views how many participants are enough to get a valid result. Let's start with What’s in a Number? By Carol M. Barnum. She states, "Whereas 7 (plus or minus 2) is the mantra for structured writing and other methods for organizing information, 5 (plus or minus 2) is the mantra for the number of participants needed in a usability test." They added that 5 will yield 80% of findings, although now they have lost faith in this number when it comes to testing Web sites. They said it will take more than 5 to uncover all the usability issues.

Jakob Nielsen thinks it should be more segregated into four different areas: heuristic, usability, participants and scenarios to gather a more comprehensive view of the web site. Nielsen says that a substantial portion of the user population should be used to correlate your findings.

The article also mentions combining methods when testing. I like this idea the best. I am a firm believer in mixing-and-matching to get the most of everything. This is what I will do.

iPad Usability: First Findings From User Testing

iPad Usability: First Findings From User Testing
by Jackob Nielsen

"It looks like a giant iPhone," is the first thing users say when asked to test an iPad. (Their second comment? "Wow, it's heavy.")

This is the opening statement of the article and it is exactly what I thought, too. The first time I held one, I also thought the second comment. I still have yet to find a way for the iPad to fit into my lifestyle.

Overall, I think certain assesments should have been obvious. The primary issue being that the main interface has not been customized to fit the larger screen. Given that the main screen sets the tone for the the other apps, I would have thought this would have been the most important interface to get right. Nielsen says that with the larger screen, the focus is further away from the tabs at the bottom of the screen and more difficult to discern.

He also discusses 'wacky interfaces'. Since there are no standards an no expectations, anything you can show, can become an interface. As a result, there are inconsistent interfaces. The examples he gives are:

In different apps, touching a picture could produce any of the following 5 results:

• Nothing happens
• Enlarging the picture
• Hyperlinking to a more detailed page about that item
• Flipping the image to reveal additional pictures in the same place (metaphorically, these new pictures are "on the back side" of the original picture)
• Popping up a set of navigation choices
  

Without rules, anything goes which will create confusion from app to app. This amounts to confusion resulting not finding the interface, too many varieties are harder to remember than a standard set and the user can activate an interface unintentionally and create even more confusion ('How do I get out of this?' or 'Where am I?').

I absolutely agree with his summary:

iPad apps are inconsistent and have low feature discoverability, with frequent user errors due to accidental gestures. An overly strong print metaphor and weird interaction styles cause further usability problems.

Groundhog Day (the movie) & A/B Testing...HUH?

Groundhog Day, or, the Problem with A/B Testing by Jeff Atwood

OK, I had to read the article before I got it either. Given my very amature standing with this 'testing' thing, I found it enlightening. This is how Atwood connects the two, "There's something delightfully Ouroboros about the epiphanies and layered revelations in repeated viewings of a movie that is itself about (nearly) endless repetition." And, where he goes is well, very cool!

I hope everyone reading this has seen the movie. If not, heed the spoiler alert! I will disclose how it ends and pretty much what happens in the whole movie...beginning....NOW!

Phil keeps going on these dates with Rita making mental notes as he goes along. His aim is to be what she wants, someone she thinks likes everything she does, always seems to know what she wants, etc. But, what is happening is a farce because it isn't real. Let's take this statement into the world of A/B testing.

What Atwood is saying is that A/B testing would have to go on forever to reach that perfect moment. The 'word' is that the span of Phil's Groundhog Day experience lasts 30-40 years. Don't think we have that kind of time, do we.

I also agree with his conclusions that A/B testing lacks feeling, it is empty and is dishonest. I agree because if you keep asking the same questions, tracking the answers, the questions will eventually be written to draw the desired answers. The key is to avoid such manipulation but Atwood doesn't cover that in this article. I am going to look for one that does.

Welcome to Summer 2010 Usability!!!

This quarter we are focusing on the usability and the experience that goes with it. In my initial reading, it occurs to me that a usability study can be very similar to a marketing survey in that you need a target, marketing tasks, goals & objectives (an objective can be to find problems) and provide projections. The usability test contains these aspects but uses different means of obtaining results.