Usability Testing Website

With all the great data Anna is getting and the potential explosion of open source usability going, there is a need for a good way to analyze usability data. One of the issues ethnographic and data intensive studies is the lack of a good way to take in and use all the data. The goal then is:

• Develop a simple way to review gathered data
• Develop a simple way to display information from that gathered data

The first step is to record the data in a way that will allow a flexible means of referencing for reporting purposes. The potentials are endless but a few quick ideas are things like graphs, charts, and statistical analysis. In order to provide the maximum amount of flexibility on how the data is stored the basic idea is to use a system of tagging reviews with a catalog of keywords. We can then define links and groupings of the keywords later and in organizing these keywords we can start pulling more concrete information. This aspect is still a little up in the air so any comments regarding this aspect are appreciated.

Update! I just tried typing along with a video and it was hard as heck! But, it was easily the best way to enter information. If you can pause the video to make sure you are getting the right thing it should be the fastest way to get down all the information from the video. If you pause a bit you should be in good shape to grab things like facial expressions and close to verbatim comments by the user. So, with this in mind comments regarding good prefixes would be nice. I started playing around with the following:

• action: clicked the background icon
• looks: frustrated (the time will be logged so a general time can be grabbed along with context of surrounding actions)
• task: send an email to someone with an attachment (this was with multiple tasks on a video session)
• says: I thought the attachment would be at the top, so I know it is attached
• asks: how do I do it
• expects: wallpaper not background
• reacts: when asked to login eyes open wide

Feel free to add to the list along with comments. They should be very short words b/c we will probably have the person actually type them along with a colon or delimiter in order to make the plain text somewhat readable and more clear to scan. It is also super easy to do something like explode(”:”, $comment) in PHP ;) Please comment on this!!! Update We have decided to be pretty verbose here. The user must use a : to delimit an entry. So, unless it is “foo : bar” you will have to clean it up later. Thanks Anna!

Just to give a quick idea of what kind of information is needed, here is a list!

Most of this is information that can be filled out before actually watching any videos.

Note It would be good if the person entering the video (the tester) filled this in when administering the test.

1. What OS is being tested?
2. What version of the OS?
3. What desktop environment?
4. What is the gender of the person taking the test?
5. What is the approximate age of the person taking the test?
6. How does the person taking the test rank his/her level of computer experience?
7. Where did the test happen? (Office/Lab/Home/ etc)
8. What language was used to administer the test?
9. What operating systems was the test subject familiar with?
10. What programs did the test subject report using (the most) at home/work?
11. At the beginning of the test, did the test subject seem: stressed/relaxed/happy/unhappy/bored
12. Did the test subject volunteer for the test? (Or was s/he volunteered by someone else?)

This is probably a bit tougher. We need a way for someone to type in what the user did in a way that will be easy and simple while still providing accurate information. The initial idea for this is a chat type interface that uses some basic keywords. Ideas regarding this are appreciated. Another thing to think about is that some session involve many tests. This means that a person might do many tests during the same session. There could be some overlap so some of the questions might be better answered after they finish the video.

1. Was the task completed?
2. If the task was completed, was it completed correctly? (That is, was the goal of the task fulfilled?)
3. What did the user do while s/he was trying to complete the task? (in excruciating detail. We need to know what the user did first, and what the user did second, and what the user did third, and what the user did after that. etc. ad infinitum.)
4. Did the test subject give up? (different from #1 – they may complete the task but not understand that they did.)
5. Did the test subject make faces while working on the task? If so, what expressions?
6. What seemed to confuse the test subject?
7. What suggestions for improving the UI do you have after watching this task?
8. What seemed to work well for the test subject?
9. How long did the test subject spend on the task?

After a person reviewed the video they will need to make sure they are happy with their answers. Spelling errors for keywords should be reviewed somehow. They should also have an area to make comments. This is important b/c 5 people could review the same video and conclude completely different things.

1. Did the test subject have any comments?
2. What, if anything, did the test subject expect to work, which did not work?
3. Does the test subject seem more stressed or less stressed at the end of the test? Happier or less happy? Eager to leave, or desirous to stay and play with the software/talk?

There are plenty of ways we can show the data but until we get the data into a database we’ll keep thinking about it more. Any use cases for this kind of thing is very useful.

Please feel free to comment on the wiki regarding any ideas, questions or concerns. Thanks!