In the above video… I was so busy trying to demo Axure pushing events into the repos and R Studio that I didn’t fully discuss the design philosophy that springs out of this unmoderated approach
My background is human factors, psych, linguistics, learning science, cogsci… etc.
So I fully buy into the view that:
"If you have to choose between what the user says and what they do… CHOOSE WHAT THEY DO ! "
A) Users can’t tell you why they do what they do
B) You are not the user… you can’t fully plumb their minds
C) It’s extremely easy to accidentally inject bias, constraints and leading ideas into ethnography
I agree if you say there’s nothing better than talking to users in their native context but in general the more you stay out of the elicitation process, esp initially… the better.
When I see UX folks doing screen walkthroughs sitting with a subject in a conversational way … I cringe… this is too often moving away from true user behaviors and needs… not towards them.
Philosophy… when testing a new feature or flow :
After the initial interviews, concept coding, categorization, profiles, scenarios, and functional prototype.
- Identify a group of 5 - 500 users who you have not talked to yet.
- Run them through an unmoderated usability test session with Axure similar to what’s in the video
- At the end of the session throw in psychometric questions… and possibly an open ended feedback request.
- Get a screen recording of every session
- Review the session data and code forms inputs into problems, ideas and ratings etc.
- Review the screen caps and identify anomalies.
- Do interviews and follow-up walkthroughs with the users who had especially good/bad/interesting results in the initial test
This way you are introducing your biases and cognitive influences AFTER the primary research … and you can efficiently test a lot of users cheaply
There’s many ways to do this of course … but that’s my current thinking: Don’t poison the well ! … get a really rich body of unmoderated research… and then follow up in person.
And one more wrinkle… the Axure event push test method in the video can be used for moderated sessions, too… and it probably should be used… because it does such a great job of unobtrusive event capture. …
Please share your thoughts and push back if this doesn’t make sense !!