Steve Jobs famously said: “A lot of times, people don’t know what they want until you show it to them.” We hear this constantly in product development, and while this saying has been somewhat demonized as of late, I would wager to say that most of us don’t really know what teaching looks like in practice.

Transcending traditional user interviews as a primary method for information gathering is relatively new. Most professionals in a position to gather user data haven’t found strong alternatives to this form of research or receive major pushback from stakeholders and people within their organization who prefer older methods.

There was a great talk given by Jon Kolko this year at Mind the Product entitled Design as Product Strategy where he described his experience researching and developing MyEdu, a product that would eventually be bought by Blackboard. What was so eye-opening about Jon’s talk was his position on formal interviews: He doesn’t conduct all. Instead, he performs behavioral ethnographic research and pulls data from video and audio recordings to create what he calls ‘utterances’ that can be transcribed into insights. The value Jon places on experiential data is total.

One of my favorite points from Jon’s talk was somewhat counterintuitive: He adds bias to the behavioral insights he collects during ethnographic research. For example, if James the college student doesn’t understand how to use career counseling, and you observe his frustration firsthand during your research, you can make the broad assertion that ‘Students don’t feel in control when they start making career choices, so we should add a feature to do x’. These assumptions about a large user group can then be tested through targeted prototyping.

What I like about this model is that it isn’t afraid to be biased. In fact, Jon’s model embraces bias as a means to test and iterate on features and direction. Without this bias, product managers have no friction from which to test against, and potentially waste large amounts of time interviewing too many users from a single group in order to control for this bias.

I applied this approach to my latest round of research while developing a gestural interface for the Microsoft Kinect. I came across a user who totally exemplified the disconnected relationship between communicative data (what people say they do) and experiential data (what people actually do). Prior to our user test, I asked him to share his opinions of the product. He told me in his own words:

“Yeah it’s pretty self explanatory, you can just grab it and turn the world and grab pictures.”

Later, I asked him to speak out loud as he was using the tool in order to capture his real time experience. I tried hard not to laugh when 30 seconds into his demo I heard:

“Have to try and...and think...have to remember how to do this. Actually, I have no idea what’s going on.”


In that moment, the subject’s actual experience did not match his own description about his experience.

If I had stopped at his opinion without capturing actual usage data, I would have (inaccurately) assumed that students found the Kinect application intuitive to use, when in reality they only thought the application was intuitive to use. With this insight, we targeted a feature set that supported a more intuitive interface, one where lightweight prompts were applied to certain UI elements as a form of ‘directing’ user interaction to an intended outcome.

While I conceptually understood the power of ethnographic research, the experience of hearing two different viewpoints from the same subject simultaneously made the value apparent and ‘real’ for me. Experiential data is very different from data derived from interviews, and you really can’t trust people’s descriptions of how they experience products, services or their environment - you have to watch them exhibit the behavior. Sure, you could use this type of research as a starting point, but all too often professionals use this information as the foundation for their work.

My recommendation? Use ethnographic research and incorporate bias the next time you gather data about your users. You might be surprised by the reduced level of rework, the quality of your product, and the emotional connection your users have to the finished product.