Steve Jobs famously said: “A lot of times, people don’t know what they want until you show
it to them.” We hear this constantly in product development, and while this saying has
been somewhat demonized as of late, I would wager to say that most of us don’t really know
what teaching looks like in practice.
Transcending traditional user interviews as a primary method for information gathering is
relatively new. Most professionals in a position to gather user data haven’t found strong
alternatives to this form of research or receive major pushback from stakeholders and
people within their organization who prefer older methods.
There was a great talk given by Jon Kolko this year at Mind
the Product entitled Design as Product Strategy where he
described his experience researching and developing MyEdu, a
product that would eventually be bought by Blackboard. What
was so eye-opening about Jon’s talk was his position on formal interviews: He doesn’t
conduct them…at all. Instead, he performs behavioral ethnographic research and pulls
data from video and audio recordings to create what he calls ‘utterances’ that can be
transcribed into insights. The value Jon places on experiential data is total.
One of my favorite points from Jon’s talk was somewhat counterintuitive: He adds bias to
the behavioral insights he collects during ethnographic research. For example, if James
the college student doesn’t understand how to use career counseling, and you observe his
frustration firsthand during your research, you can make the broad assertion that
‘Students don’t feel in control when they start making career choices, so we should add a
feature to do x’. These assumptions about a large user group can then be tested
through targeted prototyping.
What I like about this model is that it isn’t afraid to be biased. In fact, Jon’s model
embraces bias as a means to test and iterate on features and direction. Without this bias,
product managers have no friction from which to test against, and potentially waste large
amounts of time interviewing too many users from a single group in order to control for
I applied this approach to my latest round of research while developing a gestural
interface for the Microsoft Kinect. I
came across a user who totally exemplified the disconnected relationship between
communicative data (what people say they do) and experiential data (what people actually
do). Prior to our user test, I asked him to share his opinions of the product. He told me
in his own words:
“Yeah it’s pretty self explanatory, you can just grab it and turn the world and grab
Later, I asked him to speak out loud as he was using the tool in order to capture his real
time experience. I tried hard not to laugh when 30 seconds into his demo I heard:
“Have to try and…and think…have to remember how to do this. Actually, I have no idea
what’s going on.”
In that moment, the subject’s actual experience did not match his own description about
If I had stopped at his opinion without capturing actual usage data, I would have
(inaccurately) assumed that students found the Kinect application intuitive to use, when
in reality they only thought the application was intuitive to use. With this insight,
we targeted a feature set that supported a more intuitive interface, one where lightweight
prompts were applied to certain UI elements as a form of ‘directing’ user interaction to
an intended outcome.
While I conceptually understood the power of ethnographic research, the experience of
hearing two different viewpoints from the same subject simultaneously made the value
apparent and ‘real’ for me. Experiential data is very different from data derived
from interviews, and you really can’t trust people’s descriptions of how they experience
products, services or their environment – you have to watch them exhibit the behavior.
Sure, you could use this type of research as a starting point, but all too often
professionals use this information as the foundation for their work.
My recommendation? Use ethnographic research and incorporate bias the next time you gather
data about your users. You might be surprised by the reduced level of rework, the quality
of your product, and the emotional connection your users have to the finished product.