1. No category

What is CDI?
Customer Driven Innovation (CDI) is a repeatable method for
creating great ideas and products. It is based on a rigorous
understanding of your customers and their needs. Developed
from the principles of Design Thinking by the Stanford d.School,
similar variations are in use at companies across the world, such
as PayPal, eBay, Google, IDEO, Lego, and many more.
CDI, as presented in this book, is an overview of the proven tools
and techniques practiced by the Magento and eBay Enterprise
Product Design and User Experience teams. These are the same
methods we use to design Magento 2, Magento Connect, and the
Retail Order Management platform.
customer driven innovation
2
Why Follow CDI?
CDI is powered by the people who matter most –
customers. CDI quickly transforms customer
input into compelling creative direction and fast
solutions that can be iterated on. Said another
way, it is the fastest way to form a “minimum
viable product” design without relying solely on
assumptions. From a practical standpoint, CDI
offers a low-cost way to stretch research budgets
and shrink product timelines.
customer driven innovation
3
Is CDI Right for Me?
Yes, if you are a human being who cares about
great products. Store owners, engineers, and
designers – anyone can use CDI. No formal
training or technology is required to get started.
The CDI course taught in conjunction with this
book is intended to provide a solid foundation.
Later you can research and add more advanced
techniques.
customer driven innovation
4
What Will I Learn?
The methods sampled in this workshop are some
of the most popular tools and techniques in user
experience. At first you’ll apply each tool exactly
as instructed. As you progress you will quickly
develop a sense for when to use what techniques
and when to try new approaches. This holds true
for all phases of CDI – interviewing, prototyping,
and even simple brainstorming.
customer driven innovation
5
How To Use This Book
We have collected and compiled material from
numerous sources to provide a solid overview of
the overall CDI process. Some, but not all,
methods are described in detail. It also includes
templates for some methods. As a result this
book forms a great starter kit for your
introduction into CDI. Expect immediate
changes in the efficiency and impact of your
projects.
customer driven innovation
6
The Five Phases of CDI
customer driven innovation
7
The Five Phases of CDI
Diverge and Converge Throughout Your Project
Design Thinking leverages the power of divergent and convergent thinking to arrive at solutions to problems that are
often missed in traditional analysis. Divergent thinking compels designers to expand their set of ideas and concepts
about the problem, while convergent thinking drives us to focus our ideas towards an answer. By diverging and
converging at different stages in the project, we enable maximum creativity while simultaneously driving for results.
customer driven innovation
8
The Five Phases of CDI
Every phase of CDI has a goal – to succeed you
simply achieve each phase goal through whatever
method is most appropriate to your project.
1.
2.
3.
4.
5.
Empathize. Gain a real understanding of the users & problem
Define. Synthesize the data and understand root causes
Ideate. Brainstorm solutions to the problems identified
Prototype. Build a simulation to better understand your design
Test. Evaluate your design to learn if your solution really works
customer driven innovation
9
customer driven innovation
10
CDI Phase 1: Empathize
In CDI, you must relate to more than one perspective or mindset at a single time,
even when they’re opposed. Your goal is not to emotionally invest, but to
objectively uncover and understand as many viewpoints as you can.
customer driven innovation
11
Empathize
Creating meaningful solutions begins with
gaining a deep understanding of people’s
needs.
customer driven innovation
12
Empathize
This phase is about listening, observing, and
being open to the unexpected. With the right
preparation, empathizing with the customer will
inspire you with new experiences and
perspectives— and will give you a great
foundation for the next stages of your design.
IDEO & +Acumen, Human-Centered Design Workshop
customer driven innovation
13
Empathize Case Study:
eBayEnterprise.com Redesign
•  Research Questions
–  How do merchants go about looking for solutions for their business
problems? When selecting a possible vendor, what does their behavior
look like?
•  Methods
–  In-depth interviews with existing eBay Enterprise merchants to uncover
their product discovery behavior.
–  Card sorting exercises with prospective merchants with multiple rounds
of iteration.
–  Interactive prototype tests with external merchants.
customer driven innovation
14
Empathize Case Study:
eBayEnterprise.com Redesign
•  Findings
–  The website only plays a small part in a much longer overall sales cycle.
–  It is difficult for prospects to understanding product offerings when they
are categorized according to internal business units.
–  Prospect desire both general and detail information that can be easily
navigable and appropriately chunked.
•  Design Outcomes
–  Information architecture based on a customer needs.
–  Compelling, clear and digestible content that encourages lead
generation.
customer driven innovation
15
Empathize Case Study:
eBayEnterprise.com Redesign
Before
After
customer driven innovation
16
Empathize: Mindset
A healthy team is made up of people who
have the attitude that it is better to learn
something new than to be right.
Bill Buxton, Sketching User Experiences
customer driven innovation
17
1. Have a beginner’s mind
Our expertise and experience can quickly restrict us towards poor
preconceived solutions. Rather, strive to be inquisitive and curious about
why things are the way they are. Seek to be surprised as you learn, observe
and listen without judgment.
customer driven innovation
18
2. Get out of the building
Engaging your audience in their context, rather than bring them into yours,
is far more valuable because you will be able to capture actions, emotions
and values in a natural, real environment. By getting out of the building you
will uncover unexpected design opportunities.
customer driven innovation
19
3. Engage with extreme users
Discovering the behaviors of extreme users will amplify the needs and
desires of your audience. Moreover, they can show you surprising
workarounds to the problems you are trying to solve, which can lead to
surprising design opportunities.
customer driven innovation
20
4. Explore new areas of inspiration
Similar problems solved indifferent contexts can offer fresh insights to
address your design challenge. For example, luxury hotel concierges
inspired the development of Apple’s retail stores.
customer driven innovation
21
Empathize: Tools
The myth of innovation is that brilliant ideas leap
fully formed from the minds of geniuses. The reality
is that most innovations come from a process of
rigorous examination through which great ideas are
identified and developed before being realized as
new offerings and capabilities.
Tim Brown, CEO, IDEO
customer driven innovation
22
Define Your Target Audience
Any solution you design needs to be crafted
towards a target audience. Before you start any
research answer the following questions:
•  Who do we envision will use our solution?
•  What are their pain points, needs, goals and behaviors?
•  How are they currently solving these needs?
customer driven innovation
23
Tools and Methods
Secondary Research
Capture all existing knowledge around the design challenge
by interviewing subject-matter experts and stakeholders and
by reviewing internal documents, online research and
industry trends.
Competitive Analysis
Evaluate and audit competitors for their strengths/
weaknesses.
Analytics
Evaluate how people are using your solution by capturing
data through analytics tools.
Surveys
Capture data from a large number of people by asking them
to respond a set of written questions. Questions have to be
crafted extremely carefully as data can easily be
misrepresented.
Group Interviews
Interview a group of representative users. Group interviews
are valuable in gauging community/group dynamics. This
method is not recommended for typical design challenges
as the data can be easily corrupted by groupthink.
customer driven innovation
24
Tools and Methods
Semi-Structured
Interviews
Interview a participant by crafting a series of questions that
address specific research questions. Be open to the direction
that the conversation is taking as the participant might be
uncovering an unexpected area of inquiry. Adapt in-situ
questions towards that direction while at the same time
capturing the answers to the specific research questions.
Diary Studies
Collect data by asking participant to self-document
experiences for a set amount of time. Methods can include
questionnaire, journaling, photography, recording audio, etc.
Contextual Research
Meet with and study participants in their natural environment,
where they would most likely encounter the product or
service in question.
customer driven innovation
25
Semi-Structured Interview
•  Evoke stories
–  “Describe your best/worst
experience with…”
•  Ask for “show-and-tell”
•  Ask open ended questions.
–  Avoid yes/no questions
•  Seek surprises not validation
–  Avoid leading questions
•  Keep asking questions
customer driven innovation
Good Follow Up
Questions
•  Tell me more about…
•  How did that make
you feel? Why?
•  Why do you think it
was done that way?
•  What could have
been better?
•  What did you do in
response?
26
Semi-Structured Interview
Start Easy
Go Deep
What comfortable questions can you ask to build
rapport with your participant?
What questions can you ask to understand your
participant’s values and motivations?
Examples:
•  What is your favorite fast food restaurant and why?
•  How often do you drink soft drinks when you go to fast
food restaurant?
Examples:
•  Describe a time when you were really happy with your
self-serve drink experience.
•  What would happen if you were unable to drink diet soft
drinks?
customer driven innovation
27
customer driven innovation
28
Define
This phase is about synthesizing, or
absorbing, information from the previous
phase to help you see the challenge in a
meaningful way.
customer driven innovation
29
Why Define?
Because its the best way to:
•  Make sure you’re tackling the correct problem.
•  Involve your customer in the innovation process
as they share unique personal perspectives.
customer driven innovation
30
When you don’t define
If you jump to a solution without deep knowledge of the problem, you will end up reverse engineering
the problem. As time goes by, these undefined criteria lead to a disjointed product. I.e, If you don’t know where the road should go, you just make lots of different types of roads instead.
customer driven innovation
31
When it’s done right, you will…
•  Observe the problem chain and identity the
root cause of a problem, not just the obvious
surface level signals.
•  Identifying and solving the right problem is the
heart of innovation.
customer driven innovation
32
Methods: Define
Step 1: Customer Problem Statement
Step 2: Customer Journey Maps
Root Cause Analysis
Additional ways to define:
•  Conceptual Models
•  Task Analysis
•  User Scenarios
•  Use Cases
customer driven innovation
33
Customer Problem Statement
A problem well stated is half solved.
- Wally Davis
customer driven innovation
34
What’s a Customer Problem Statement?
A statement that clearly describes the issue and
includes a vision and method to solve the
problem.
customer driven innovation
35
Method: Customer Problem Statement
5 W’s method
1. WHO - Who is the customer?
2. WHAT - What goal has to be achieved?
3. WHEN - When does the issue occur?
4. WHERE - Where is the issue occurring?
5. WHY - Why do you think the problem exist?
customer driven innovation
36
Customer Problem Statement
Add root case after completing root cause analysis
customer driven innovation
37
Customer Journey Maps
customer driven innovation
38
What is a Customer Journey Map?
An journey map (or “experience map”) helps create a
shared empathic understanding of customers'
interactions with the company or product over time.
Search
Evaluate
Decide
Buy
+
_
Use it when you need to:
1.  Understand your customer’s
end to end experience.
2.  Identify strengths and areas
of opportunity for your
current product lifecycle.
customer driven innovation
39
Why use Customer Journey
Maps?
•  Gain Strategic Insights
–  A holistic view of a customer’s mindset, motivations, experiences, and key trigger
points can give us unique insights into how to design for that customer.
•  Drive Alignment
–  Align teams or stakeholders to strategic initiatives.
•  Formulate a Plan
–  Journey maps can provide a formalized framework to ensure continuity and
consistency across all integrated communications and touch points.
•  Gain a Deeper Understanding
–  Empathy is key to designing a great user experience. Journey maps are a great
way to “walk in the user’s shoes”.
customer driven innovation
40
Customer Journey Map (Example)
Source: Adaptive Path
customer driven innovation
41
Customer Journey Map (Example)
Source : Rail Europe
customer driven innovation
42
Customer Journey Map (Example)
Source : Lego
customer driven innovation
43
Root Cause Analysis
customer driven innovation
44
What is a Root Cause Analysis?
A root cause analysis helps us to understand the core
cause of a problem, why it happened, and thus provides
the basis for building a sustainable solution.
customer driven innovation
45
Why do a Root Cause Analysis?
•  Get below the surface problem
Usually the problem you see is just a symptom of a deeper underlying issue.
•  Break the chain
Engage this method to break the cause-effect chain and prevent the problem from
recurring.
•  Holistic customer experience
As this method can be conducted at several levels of complexity, it has the
capability to bring forth all the possible fault-lines in a system. This method help to
fix errors of all magnitude (not just the primary problem) thus contributing to
creating an holistic customer experience.
•  Create a plan
This method identifies the seriousness of a problem and helps to dedicate
appropriate time and effort for its solution.
customer driven innovation
46
Methods: Root Cause Analysis
• 
• 
• 
• 
5 Whys
Fishbone Diagram / Tree Diagram
Fault Tree Analysis
Event and Cause Factor Analysis
customer driven innovation
47
5 Whys (Example)
Problem: You are on your way home from work and your car stops in the
middle of the road.
WHY?
Car Stops
WHY?
Ran Out of
Gas
No Money
WHY?
Root Cause
WHY?
WHY?
Went Drinking
last night
Slept Late
customer driven innovation
Forgot Wallet
48
5 Whys (Template)
customer driven innovation
49
Fishbone Diagram
Fishbone diagrams provide systematic ways of looking at effects and the
causes that create or contribute to those effects.
People
Method
Measurement
Problem
Machine
Material
Environment
(Conceptual Representation)
Source : reliabilityweb.com
customer driven innovation
50
Fishbone Diagram (Example)
Source: www.ascd.org
customer driven innovation
51
Fault Tree Analysis
This is a top-down approach where events are placed at the root of a “tree of
logic” and all situations and events causing effects are added to the tree as a
series of logic expressions
Source: www.conceptdraw.com/
customer driven innovation
52
Event and Cause Factor Analysis
Method to look for any possible risk impacts and understand the situations
where change is occurring.
Source : www.wecc.biz
customer driven innovation
53
customer driven innovation
54
CDI Phase 3: Ideate
Have you got a well-defined, precise problem? Now focus
on finding a solution. In the ideate phase, you let can your
creativity go wild – there are no bad ideas!
customer driven innovation
55
Ideate
Ideation = “Idea Generation”
customer driven innovation
56
What is Ideation?
Ideation is the process that leads to the creative solution
to our problem – the “Big Idea”.
The best idea is rarely (if ever) the first idea. Our first ideas
tend to be the most practical in their approach. But if the
solution were obvious, would there really be a problem to
solve?
Generating lots of different ideas helps remove the filters
from our thinking. We make creative leaps from one idea
to the next, and begin to develop deeper concepts. Don’t
hold back on your ideas during ideation, the wildest or
most absurd ideas may hold the key to a brilliant solution.
customer driven innovation
57
Idea Curve
customer driven innovation
58
Ideation in CDI
The point of the Ideation phase of Customer Driven
Innovation is to ultimately arrive at an innovative
solution to the problem.
We begin by generating lot of ideas with varying
approaches to solving the problem. We then
discuss the merits and drawbacks of some of the
most “interesting” ideas. Then from these “best
options” we narrow down to a single approach,
which may be an idea that emerged from
discussions, borrowing bits and pieces from the
other ideas along the way.
customer driven innovation
59
The Ideation Environment
Ideation is most productive in an environment that is
conducive to the process and supportive of its
participant(s)
The best ideation environment is one where people
are free to…
–  Think
–  Speak
–  Fail
customer driven innovation
60
Rules for Participation
In order to engage in the Ideation process
participants must...
•  Be open-minded: There are no bad ideas
•  Be optimistic: Defer judgment
•  Be courageous: Replace fear with curiosity
customer driven innovation
61
Common Ideation Methods
•  Rapid Sketching
–  Rapidly generate as many design ideas as you can
•  Active Brainstorming
–  Leverage the collective thinking and energy of the group
•  Process Flows
–  Create flow charts to visualize the steps of the process and
identify opportunities
•  Participatory Co-Design
–  Have the team co-design with people who will ultimately use
the solution
customer driven innovation
62
Activity – Rapid Sketching (Crazy 8s)
•  Here’s how it works
–  Each team member quickly sketches eight ideas
–  This happens in five minutes with 40 second intervals.
–  Sketches are done on single sheet of paper (11x17)
–  Two rounds may be necessary but optional.
•  What happens next?
–  Ideas range from boring to absurd. That’s OK.
–  The last sketches are really difficult and usually absurd.
–  This enables sharing, voting and idea convergence.
•  Anything else?
–  Avoid focusing on too many details or self criticism.
–  Sharpies and a stopwatch app (Bit Timer) are a must.
customer driven innovation
63
Crazy 8’s Sketching
Image courtesy of Google Ventures.
customer driven innovation
64
About Sketching
•  It’s easy and anyone can do it
Sketching is not about having a pretty picture it’s about
communicating the idea.
•  Quantity not quality
The more sketches you have the more pieces of the puzzle you
have.
•  Go fast and go crazy
Ideas can be fleeting, don’t get bogged down in the details or
worry about feasibility.
•  Let the pictures do the talking
Don’t use words or write an explanation of the concept. Rough
sketches leave room for imagination.
customer driven innovation
65
Rapid Sketching – Crazy 8’s
customer driven innovation
66
Rapid Sketching (Template)
customer driven innovation
67
Design Discussions
The process of evaluating your design explorations
and brainstorming often requires a group discussion
to reach consensus. Design discussions, if done
properly, will be critical to team success.
“Always keep your words soft and sweet, just in
case you have to eat them.”
- Andy Rooney
customer driven innovation
68
Right vs. Wrong? Good vs. Bad?
•  In evaluating designs you must divorce yourself from
the idea that what you like is “good” and what you
don’t like is “bad”.
•  There is no “right” or “wrong” in design…
•  A “good” design helps the user achieve their goal –
there can be more than one way
•  A “bad” design does not help the user achieve their
goal. However a “bad” design could be “good” in
another context – remember the challenge you are
trying to solve.
customer driven innovation
69
3 Traits That Lead to Design Success
1. Be Respectful
–  Respect every idea and every team member
–  Your “a ha” moment can come from anywhere
2. Be Open
–  Come to work everyday with an open mind, don’t assume you
already know all the answers
–  Put your ego aside – it’s not about knowing the right answer – it’s
about finding the right answer
3. Be Fearless
–  “Kill your darlings” if needed – don’t over-commit to a concept
–  Embrace new ideas, no matter how outlandish they may seem at
first
customer driven innovation
70
The “PNP” Sandwich
•  An honest evaluation of a design will consider both the positive
and negative traits of that work – good feedback does the same.
–  This is an effective way to deliver feedback to “cushion” the receiver
–  To work effectively the positive and negatives must be honestly delivered
•  “PNP” = “Positive Negative Positive”
–  Open with a positive insight
–  Follow up with the feedback that may be considered negative by the
receiver
–  Close with another positive insight
customer driven innovation
71
customer driven innovation
72
CDI Phase 4: Prototype
Because we can’t easily test a fully built-out solution, we’ll build a
simulation, or prototype, to represent and better understand our design.
customer driven innovation
73
What is a prototype?
A model of the real thing, but not the real thing. A
prototype will not contain every detail and
nuance of the real product. Prototypes should be
faster, cheaper, and easier to build than the real
product.
customer driven innovation
74
Why bother with prototypes?
If we have a brilliant solution idea, why bother to
prototype it – why not just build it and launch our
product?
•  What if your idea turns out to not be as perfect as you
thought?
•  The cost of making a mistake after launch is much
greater than the cost of making a mistake early in the
project.
–  Bad prototype? Throw it away and try another idea.
–  Bad product? Company goes under.
customer driven innovation
75
The cost of a mistake
Cost
So that you don’t
build the wrong
product here
Build and test
prototypes here
while it’s cheap
Time
customer driven innovation
76
The primary value of a prototype lies in the
opportunity to test and validate the concept
with users prior to building the real product.
customer driven innovation
77
The prototyping process
1. 
2. 
3. 
4. 
Decide on the type of prototype needed
Determine the level of fidelity
Define the user scenarios
Build it!
customer driven innovation
78
1. What type of prototype?
Storyboard
Embrace your inner comic strip artist and draw the experience frame by frame.
Storyboards are great for understanding the end to end process as experienced by the
user. Too often, we only focus on a narrow portion of the experience – storyboards fix
that.
Physical
The obvious choice for hardware products, physical prototypes can also act as 3D
storyboards to force you to take a larger view of the overall experience.
Paper
Nothing beats paper and pencil for speed and cost. Anything can be drawn – from
physical products to user interfaces.
Digital
A more advanced, and also more costly, version of a paper prototype. Digital prototypes
have the advantage of being able to look and feel more similar to the real thing, and
contain more interactivity than other prototypes. But be careful, it’s easy to sink way too
much time and money into creating a digital prototype. Keep it as simple as possible!
customer driven innovation
79
Example Prototypes
High fidelity physical prototype
High fidelity physical prototype
https://youtu.be/CO9I3bikFuY
http://www.emilymanders.com/?p=52
Paper prototype
Low fidelity physical prototype
http://www.creativebloq.com/mobile/practical-guidetactical-mobile-prototyping-6126240
http://www.lego.com/en-us/seriousplay/
customer driven innovation
80
Example Prototypes
Low fidelity HTML prototype
High fidelity HTML prototype
customer driven innovation
81
2. Fidelity – Low vs. High
•  Fidelity is simply how similar your prototype is to the
real product.
•  Low fidelity prototypes (like storyboards or paper)
have the advantage of being cheap and fast. We
recommend low fidelity prototypes early in your
project to quickly validate ideas.
•  High fidelity prototypes have the advantage of more
closely simulating the real product. Due to the time it
takes to build a high fidelity prototype, we suggest
reserving high fidelity work until after you’ve already
validated the initial concept with users.
customer driven innovation
82
3. Identify the User Scenarios
•  A user scenario is a simple narrative that describes the
user interacting with your product. This narrative
provides the boundaries for your prototype. Don’t try
to cover every possible situation, error state, and edge
case in your prototype. Stick to interactions that are
likely to happen in your defined scenario.
•  Note: a user scenario is not the same thing as a use
case or a user story. Though they sound similar, they
serve different purposes and have different details in
them.
customer driven innovation
83
4. Build It!
Now that you know the type, fidelity, and user scenarios for your
prototype, it’s time to start building. A few tips to help you along:
1. 
The process flow and interactions are more important than what it
looks like. Pretty is not the goal.
2.  Build as little as possible while still covering all steps of the user
scenarios.
3.  Focus on the outcome, not the craft. It doesn’t matter if your code
is hacked together and only works in Firefox. Popsicle sticks instead
of 3D printed parts? Perfect!
customer driven innovation
84
customer driven innovation
85
CDI Phase 5: Test
Does your solution work? Can it be improved? Use your
prototype on real people for real-time feedback so you can
iterate on your design. customer driven innovation
86
The Value of Evaluation
•  In the Empathize phase, we learned about generative
research – research that generates information about
our users’ needs. Now that we have have a prototype to
cater to those needs, it’s time to do some evaluative
research – let’s test the prototype and get some user
feedback!
•  Collecting user feedback on the prototypes is key to
success. No matter how brilliant you believe your
concept to be, without feedback from users, you’re just
guessing.
•  The more times you evaluate, the better – test early, test
often!
customer driven innovation
87
Types of Evaluative Research
•  Quantitative
The collection of empirical data, which is analyzed via statistical methods
to determine performance, correlation, causation, and trends.
Quantitative research typically requires large sets of data (i.e. a larger
sample population), which can be costly and time consuming.
Quantitative research is also difficult to use for determining a person’s
motivations or intent, as it usually collects data about the outcome only.
•  Qualitative
Exploratory research conducted on smaller groups of people, relying, in
part, on the researcher’s ability to interpret human motivations and
behaviors.
When we test prototypes, we usually choose qualitative tests. Qualitative
methods are usually faster, cheaper, and better at identifying human behaviors
with small sample sets (fewer participants).
customer driven innovation
88
Heuristic and Expert Reviews
•  Heuristic Review
–  A heuristic can simply be described as a general rule. A heuristic
evaluation is a comparison of your prototype to a standard set of
heuristics. The most popular set of heuristics for software
evaluation come from Jakob Nielsen’s 1995 article “10 Usability
Heuristics for User Interface Design.”
•  Expert Review
–  An expert evaluation is similar to a heuristic evaluation, except that
it adds in a person's interpretation of design best practices. Expert evaluations are usually done by consultants external to
your company who specialize in evaluative research. Typically,
they have run hundreds, if not thousands, of user research
sessions, and have a keen sense of what real people will do in a
given situation. As a result, they can be a valuable proxy when
recruiting actual users isn’t easy or affordable.
customer driven innovation
89
Cognitive Walkthroughs
Cognitive walkthroughs put you in the shoes of the user. This is the ultimate
game of “let’s pretend.” In a cognitive walkthrough, a small team, usually 2 -5
people working on the project, gather to interact with their prototype. A series
of primary tasks are identified, and then the team attempts to behave as an
actual user would to complete the task. At each step of the way, they ask
themselves the following questions:
1. 
2. 
3. 
4. 
Will the user intentionally try to achieve this action?
Will the user see that the correct action is available?
Will the user realize that the correct action will achieve their desired effect?
If the user takes the correct action, will they perceive that as progress
toward their goal?
If the answer to any of the questions is ‘no’ at any step in the task, then there is
opportunity for improvement. Because these four questions must be asked for
every step of every task by every person of the evaluation team, a full cognitive
walkthrough can be time consuming, but also very thorough.
customer driven innovation
90
User Interviews
In the Empathize phase, we used user interviews to learn about our users’ needs.
In the Test phase, you can interview more users to collect feedback on your
prototype. Evaluative user interviews are semi-structured, and may follow a flow similar to:
– 
– 
– 
– 
– 
Show the user the prototype
Present the user with a realistic scenario
Ask the user how they would expect to interact with the prototype to complete the scenario
Show the user the state the prototype would be in after they interacted as described
Inquire if the result state matches their expectations
When conducting an evaluative interview, be wary of asking for the
user’s opinions about the prototype. What you need to identify is how they will
behave when using the product.
If collecting opinion information is important (e.g. “do you like the colors? Is the
design pleasing?” etc.) ask those questions at the very beginning or very end of
the interview so you can separate preference feedback from behavioral
feedback.
customer driven innovation
91
Usability Testing
Usability testing is the king of evaluative research, at least in the field of
software and the web. If you’re not already familiar with usability testing, the
basic process is to create tasks for a test participant to do while interacting with
your prototype. With just a few (5-10) participants, you can identify the vast
majority of the problems with your current design.
• 
Three common variations of usability testing
1. Moderated in-person. A test facilitator is in the room with the participant and has
an ongoing discussion as the participant tests the prototype
2. Moderated remote. Similar to in-person testing, however the facilitator is in
another room (sometimes even in even another country). The facilitator can see
everything the participant does via live video feed, and interact with the
participant via voice or video chat.
3. Un-moderated testing. The participant interacts with the prototype
autonomously, and provides feedback by answering pre-scripted questions. A
video recording is made of the test session and view after the fact by the team.
This technique is most commonly used for consumer oriented websites and
software, through platforms like usertesting.com.
customer driven innovation
92
A Basic Usability Testing Process
1.
Identify the testing team
Facilitator
The person that will talk with the participant during the test. Ideally, the facilitator will not be
the designer on the project in order to avoid bias.
Note taker
It’s always helpful to have a second person take notes, so the facilitator can stay focused on
the discussion. The note taker must strive to be as objective as possible and not insert bias.
Observers
Members of the product team should observe whenever possible. Preferably from another
room or via a video recording. Too many observers in the room with the participant will add
unrealistic stress to the situation, altering the results.
2. 
Take the user scenarios defined to build the prototype and convert them into tasks
for the test participant. Writing good tasks takes practice. See
http://www.measuringu.com/blog/task-tips.php for some simple tips to get started.
3.
Write a facilitator’s guide using the tasks and the planned physical set up (room,
equipment, etc.)
customer driven innovation
93
A Basic Usability Testing Process
4. Recruit participants
–  Participants can be anyone that matches the basic profile of the user. Avoid
recruiting anyone familiar with the project, this will bias their feedback.
5. Run the test!
–  A typical session will last about an hour for each participant.
–  Record audio and video of every session (be sure to get the participants’
approval first)
–  Avoid the temptation to do more than 6 sessions in a single day. Fatique will set
in for the moderator and note taker
6. Analyze the data
–  Review all the notes and video recordings of the sessions
–  Identify issues
–  Rate each issue by criticality
7.
Report the findings
–  Reports can be as brief or as formal as needed
–  Make sure all the relevant stakeholders are present for the presentation
customer driven innovation
94
A Word About Surveys & Focus Groups
•  Surveys and focus groups are often suggested by stakeholders as ways to
evaluate ideas and designs. Be wary of them!
•  Surveys often fail at collecting valuable feedback because they rely on your
ability to ask perfect questions without meeting your audience. Chances
are, you can’t predict every permutation of every possible answer to every
question. Furthermore, did you ask all the right questions? Probably not.
Bottom line – avoid surveys for evaluating prototypes.
•  Focus groups are a poor method for learning about behavior. They are more
suited to judging people’s opinions about ideas.
customer driven innovation
95
After the Test – Now What?
Now that we’ve conducted our test, collected data, and analyzed it, we now
need to figure out what to improve. It’s time to iterate!
•  Test results will identify new problems that you hadn’t considered before – or
maybe confirm some you already suspected. •  It will be tempting to jump to solutions to those problems. Users didn’t see
that button? Make it bigger! Make it blink! Don’t do it! •  Go back to the tools in the Define stage, and do another root cause analysis
- find out why the problem really happened.
•  Consider the Empathize stage - did the test identify a gap in our
understanding of who the users really are and what they really need? Maybe the prototype failed because we solved the wrong problem. •  Once the correct solutions have been defined, build a new prototype and
test it again!
customer driven innovation
96
Why you MUST Iterate
1.  You can always make your product better, so don’t
stop trying.
2. Perfect is the enemy of done. Don’t wait to launch a
product till it’s “finished” because it never will be. Launch your product as soon as you can - as soon as
it’s ”viable” (i.e. you can sell it and have happy
customers) - then iterate on improvements as fast as
you can using the CDI techniques.
customer driven innovation
97
In Conclusion
In this booklet, we’ve only scratched the surface
of CDI’s power.
We encourage you to dig deeper into CDI and
Design Thinking by looking up the references
contained herein, and dive into the
recommended reading. It’s an exciting journey
that will lead you to better and more profitable
products!
customer driven innovation
98
Keep in Touch!
Feedback? Questions? Drop us a line any time.
Jonathan Atkins
Head of Product Design
Magento + eBay Enterprise
[email protected]
Eric Erway
UX Design Manager
eBay Enterprise
[email protected]
Tonima Das
UX Designer
Magento
[email protected]
Mary Lan
UX Design Manager
Magento
[email protected]
Stephen Tudor
UX Developer
eBay Enterprise
[email protected]
Scott Hensley
UX Designer
Magento
[email protected]
Keith Howard
Sr. UX Design Manager
Magento
[email protected]
Michele Miller
Sr. Manager Content Strategy
eBay Enterprise
[email protected]
Adam Hundt
Sr. Product Manager
Magento
[email protected]
customer driven innovation
99
Online Reference Materials
• 
Stanford d.school Design Thinking Crash Course - http://dschool.stanford.edu/dgift/
• 
IDEO Design Kit - http://www.designkit.org/
• 
When to Use Which User-Experience Research Methods - http://www.nngroup.com/articles/which-ux-research-methods/
• 
http://www.fastcompany.com/3033567/agendas/brainstorming-doesnt-work-try-this-technique-instead
• 
http://scottberkun.com/essays/35-how-to-give-and-receive-criticism/
• 
http://scottberkun.com/essays/23-how-to-run-a-design-critique
• 
http://www.ceptara.com/blog/how-to-write-problem-statement
• 
http://www.conceptdraw.com/solution-park/engineering-fault-tree-analysis-diagrams
• 
http://www.isixsigma.com/tools-templates/cause-effect/determine-root-cause-5-whys/
• 
http://www.au.af.mil/au/awc/awcgate/nasa/root_cause_analysis.pdf
• 
http://www.realitycharting.com/methodology/conventional-wisdom/rca-methods-compared
• 
http://rootcause.com/what-is-rca
• 
http://www.thoughtworks.com/de/insights/blog/you-need-understand-problem
customer driven innovation
100
Recommended Reading
• 
“101 Design Methods: A Structured Approach for Driving Innovation in Your Organization” by Vijay Kumar
• 
“Beautiful Evidence” by Edward R. Tufte
• 
“Communicating Design: Developing Web Site Documentation for Design and Planning (2nd Edition) (Voices That Matter)” by
Dan M. Brown
• 
“Designing the Conversation: Techniques for Successful Facilitation (Voices That Matter)” by Russ Unger, Brad Nunnally, Dan
Willis
• 
“Developing Design Sensibilities” by Jane Fulton Suri and R. Michael Hendrix
• 
“Digital Design Essentials: 100 ways to design better desktop, web, and mobile interfaces” by Rajesh Lal
• 
“Don't Make Me Think, Revisited: A Common Sense Approach to Web Usability (3rd Edition) (Voices That Matter)” by Steve Krug
• 
“Drawing Ideas: A Hand-Drawn Approach for Better Design” by Mark Baskinger, William Bardel
• 
“Envisioning Information” by Edward R. Tufte
customer driven innovation
101
Recommended Reading
• 
“Predictably Irrational, Revised and Expanded Edition: The Hidden Forces That Shape Our Decisions” by Dan Ariely
• 
“Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems” by Steve Krug
• 
“See What I Mean: How To Use Comics to Communicate Ideas” by Kevin Cheng
• 
“Shape of Design” by Frank Chimero
• 
“Sketching User Experiences: Getting the Design Right and the Right Design (Interactive Technologies)” by Bill Buxton
• 
“Sketching User Experiences: The Workbook” by Saul Greenberg, Sheelagh Carpendale, Nicolai Marquardt
• 
“Solving Problems with Design Thinking: Ten Stories of What Works” by Jeanne Liedtka, Andrew King, Kevin Bennett
• 
“The Design of Everyday Things: Revised and Expanded Edition” by Don Norman
• 
“The Visual Display of Quantitative Information” by Edward R. Tufte
customer driven innovation
102
Recommended Reading
• 
“Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective
Solutions” by Bruce Hanington, Bella Martin
• 
“Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal,
Make Better Design Decisions, and Teach through Design” by William Lidwell, Kritina Holden, Jill Butler
• 
“Interviewing Users: How to Uncover Compelling Insights” by Steve Portigal
• 
“Usability Engineering” by Jakob Nielsen
• 
“Visual Explanations: Images and Quantities, Evidence and Narrative” by Edward R. Tufte
customer driven innovation
103