“Two roads diverged in a wood, and I—
I took the one less traveled by,
And that has made all the difference.”
- Robert Frost
Robert Frost laments that he may choose only one path to travel, and makes the decision based solely on his limited
knowledge about which path has been traveled less. Thankfully, as a User Experience team, we not only get to create
the paths we send our users down, but also get to observe how many users choose each one. When used intelligently,
this leads us to insights and opportunities for improvements at each step along the way.
One method we use to control and monitor our users’ traffic is A/B testing. In an A/B test, we create a new version of an
existing page within a flow in an attempt to see if we can improve the outcome. This outcome we are monitoring is
generally referred to as a “conversion.” A new design is built with a hypothesis in mind which should theoretically
produce more conversions. Examples of conversions could be things like selling more widgets, attracting more clicks, or
increasing the percentage of signups per visit.
The UX/UI team at ZirMed recently used A/B testing within our patient portal, where millions of transactions account for
millions of dollars in payments each year, meaning we have more than enough traffic for an effective A/B path analysis.
Within our portal, patients have the option to either pay individual statements, or to sign up for a full Patient Notebook
account where they can view and pay future statements online, communicate with their doctors’ offices, and update
their information in a single location. At the end of the “pay single statement” flow, we give users the opportunity to
sign up for a full account. This particular A/B test was designed to optimize the number of users who sign up for a full
account on the Receipt page after completing the payment flow.
We created a variation of the Receipt page to test whether text saying “You’re almost done” followed by a “Continue”
button would outperform the current page which featured a visually attractive banner and the message to “Sign Up for
Patient Notebook”. See the screenshots below:
Current
Variation
A very important part of A/B testing is that only one element of the flow was changed. This is so any difference in the
conversion rate can be directly attributed to the variation. Each step before the receipt, and each step of the signup flow
after the receipt, went unchanged to ensure that we were measuring the effectiveness of the changed element and
nothing else.
We designed the experiment to randomly distribute traffic to each design equally. Over the course of thousands of
visits, random distribution negates the influence of outside factors such as gender, time of day, amount owed on
statement, or any other factor not considered part of the test.
With our variation set up, and our traffic being distributed evenly, patterns began to emerge:
We collected the data into a chart which shows how many users we saw (sessions), and how many conversions each
path generated. This experiment gathered data from over 5,000 visits to gather data. The path with the original banner
led 2.9% of users to sign up for Patient Notebook. The path with the text-based variation, on the other hand, generated
almost 3 times as many, at 8.1%! Seeing such a dramatic difference, we could be confident that directing all traffic
toward the text-based variation would yield more signups.
It’s important to note that A/B tests reveal how many users perform a certain action from a specific design, but not
why. We can theorize reasons for the difference, but without talking to users we can’t really be sure. And ultimately, the
underlying reason behind the effectiveness of a design is secondary to the quantitative fact that it just IS more effective
at converting clients. To answer those deeper “why” questions, we use qualitative analysis techniques such as taskbases assessment and contextual inquiries, which the UX team leverages on site with users on an ongoing basis as well.
Wherever there is an opportunity for a more effective design, there is an opportunity for my team to create a diverging
path and monitor the results. Robert Frost’s decision to take the path less traveled hints at the possibility of missed
opportunities due to his ability to take only a single path, but by using modern technology and data-driven decisionmaking, ZirMed is able to make sure that the path most traveled is the right one every time.