ContentVerve - ConversionXL

Your Test is Only as
Good as Your Hypothesis
How to Cra! Solid, Data-Driven Hypotheses
ConversionXL Live 2015
@ContentVerve
If we get a responsive website, we’ll
make more money…
@ContentVerve
@ContentVerve
$$$ Cost of responsive redesign $$$
$$$ Serious loss of revenue $$$
$$$ Cost of “damage control” $$$
$$$ Cost of double work $$$
@ContentVerve
@ContentVerve
I’m a split test junkie!!!
@ContentVerve
Dumbass…
@ContentVerve
A solid test hypothesis is an informed solu"on
to a real problem - not an arbitrary guess.
@ContentVerve
It’s the star"ng point for further research.
@ContentVerve
1. The change you want to make
2. How this change will affect your prospects
3. The impact you expect to see as a result
@ContentVerve
1. The change you want to make
2. How this change will affect your prospects
3. The impact you expect to see as a result
@ContentVerve
X into, ______
X I can get more
By changing ______
X and thus increase ______.
X
prospects to ______
@ContentVerve
Hardwire Your Brain to
Think in Hypotheses
@ContentVerve
Think in terms of
“op"miza"on hypotheses”
not
“test hypotheses”
@ContentVerve
And now…..
A Really Stupid A/B Test!
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
Control:
Treatment:
2 months data & 490 downloads
@ContentVerve
Control:
Treatment:
2 months data & 490 downloads
No Real Difference
@ContentVerve
@ContentVerve
@ContentVerve
40%
Check prices
37%
Learn more
23%
Book
@ContentVerve
40%
Tjek prices
37%
Learn more
23%
Book
@ContentVerve
By adding a pricing page, I can bridge a serious gap in
the decision-making process, mo"vate more prospects
to say yes and thus increase the number of leads.
@ContentVerve
@ContentVerve
X into, ______
X I can get more
By changing ______
X and thus increase ______.
X
prospects to ______
@ContentVerve
Total number of days to run the test: 314 days
@ContentVerve
If I can get a li# of 33%
Total number of days to run the test: 35 days
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
By adding a privacy policy, I can reduce fric"on, get more
prospects to give me their info, and thus increase sign-ups.
@ContentVerve
Control:
Treatment:
100% privacy - we will never spam you!
@ContentVerve
Control:
Treatment:
18.70% drop in sign-­‐ups
Sta$s$cal Confidence Level 96%
@ContentVerve
By removing the word “spam” and adding a guarantee, I
can reduce fric"on, get more prospects to give me their
info, and thus increase sign-ups.
@ContentVerve
Control:
Treatment:
We guarantee 100% privacy. Your
information will not be shared
@ContentVerve
Control:
Treatment:
19.47% increase in sign-ups
Sta$s$cal Confidence Level 96%
@ContentVerve
By removing the guarantee, and adding the word “spam” I
can increase fric"on, get less prospects to give me their
informa"on, and thus decrease sign-ups.
@ContentVerve
@ContentVerve
24% drop in downloads
Sta$s$cal Confidence Level 99%
1 month of data / 349 conversions
@ContentVerve
1. Conduct Research
2. Hypothesize
3. Create Treatment
4. Test Treatment
5. Analyze
6. Run Follow-Up
Experiment
@ContentVerve
1. Conduct Research
2. Hypothesize
3. Create Treatment
4. Test Treatment
5. Analyze
6. Run Follow-Up
Experiment
@ContentVerve
1. Conduct Research
2. Hypothesize
3. Create Treatment
4. Test Treatment
5. Analyze
6. Run Follow-Up
Experiment
@ContentVerve
Conversion Research
@ContentVerve
Device Category
Image Removed
@ContentVerve
Browser
Image Removed
@ContentVerve
Browser - Desktop Sessions
Image Removed
@ContentVerve
Other reports:
Top landing pages
Site Search
New vs. returning
Nav overview
Site Speed
Unbounced sessions
Unbounced pr. device
New vs. returning pr. device
Learn to use advanced segments!
@ContentVerve
Slides Removed
Home Page
Category Page
Product Page
Cart
Checkout
Confirma"on
@ContentVerve
Home Page
Category Page
Product Page
Cart
Checkout
Confirma"on
@ContentVerve
By reorganizing the product page content, I can create
a more logical user expereince, reduce cogni"ve
fric"on, get more prospects to configure a product and
thus increase purchases.
@ContentVerve
@ContentVerve
Top-Selling Product Categories
@ContentVerve
Top-Selling Products
@ContentVerve
@ContentVerve
Slide Removed
1.
2.
80%
drop
off
3.
54%
drop
off
4.
14%
drop
off
5.
66%
drop
off
6.
4%
drop
off
@ContentVerve
How to Build Funnels Manually
Go to “All Pages” in GA
Type URL
Structure
Write Down
Unique
Pageviews
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
“In the economy of ac"on, effort is a cost…
Laziness is built deep into our system.”
- Daniel Kahneman
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
Al?
@ContentVerve
Michael
[email protected]
Al? Al?
Daddy?
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
Run Clean Tests, So
You Are Acutally
Ge$ng Insight!
@ContentVerve
@ContentVerve
@ContentVerve
Slide Removed
Qualita"ve Research
@ContentVerve
Interviews with:
- Customers / Poten"al customers
- Sales
- Customer service / support
@ContentVerve
Interviews with:
- Customers / Poten"al customers
- Sales
- Customer service / support
@ContentVerve
Sales
1. What are the top 3 ques"ons you get from poten"al
customers?
2. What do you answer to these ques"ons?
3. What is the biggest barrier keeping people from
buying your product?
4. Are there any specfic selling points that work
par"cularly well? If so, which ones?
5. Have we missed anything important? Got something
to add?
@ContentVerve
Customer service / Support
1. What are the top 3 ques"ons you get from
poten"al customers?
2. What do you answer to these ques"ons?
3. Are there any par"cular aspects of X that people
don’t understand?
4. What aspects of X do people like the most/least?
5. Have we missed anything important? Got
something to add?
@ContentVerve
Does A/B Tes"ng Make
Sense on Your Website?
@ContentVerve
Your Test is Only as Good as
Your Hypothesis
The be%er the research, the be%er the insight
The be%er the insight, the be%er the hypothesis
The be%er the hypothesis, the be%er the test
The be%er the test, the be%er the learning
@ContentVerve
Don’t be the Split Test Junkie
Be The Conversion Plumber
@ContentVerve
Thank You For Listening!
ConversionXL Live 2015