1. news

Headline Testing
Orientation and User Guide
Table of Contents
01
INTRODUCTION
02
GETTING STARTED
03
CREATING A HEADLINE EXPERIMENT
04
IN-PROGRESS TESTS
05
TEST CONCLUSION AND ARCHIVE
06
TIPS AND BEST PRACTICES
07
FAQs
08
METRICS GLOSSARY
Introduction
What is “MAB” Headline Testing?
“MAB” headline testing is a multi-variate approach to running live tests to determine if one
headline is more successful at attracting visitors’ interest (clicks) and holding their attention on
the page (engaged time).
This multi-armed-bandit approach, also known as Thompson Sampling, is different from other
multivariate testing methodologies. It exploits the success of winning headlines by ‘playing’
them more frequently as they prove to outperform other headlines during the test.
To learn more about Thompson Sampling, check out the methodology report from our Data
Science team.
What does a test look like?
Simply select an article on your homepage (or another high traffic landing page or section
front) and enter the various headlines that you want to test.
Chartbeat will then dynamically serve those different headlines to your audience equally, and
as certain headlines begin to receive higher click-through-rates and engagement, they will be
served more frequently to your audience while unsuccessful headlines are served less often.
The test ends when the winning headline has been determined with 95% confidence and
Chartbeat begins to automatically serve that headline 100% of the time.
To make sure that tests come to a conclusion as efficiently as possible, we have an alternate
way to determine a winner, called ‘soft converge’. If 20 minutes have passed and we're 95%
confident that no headline is better by a margin of 25%, the leading headline will win.
How do I get started with Chartbeat Headline Testing?
Your Customer Success Manager is the best person to contact if you’re interested in
performing headline tests on your site. They’ll be your best point of contact to discuss pricing,
permissions, and other details. If you’re unsure who your Customer Success Manager is, just
reach out to the Chartcorps at [email protected] and they’ll be happy to point you in the
right direction.
01
Getting
Started
Install the Headline Testing Bookmarklet
As the interface for the Headline Testing tool is built into the Heads Up Display, you’ll need to
install a specific version of the browser overlay. Note that the Headline Testing product is
optimized for Google Chrome. Click here to get the installer.
Select Headline Testing Mode
Once the “MAB and HUD” bookmarklet has been installed, simply refresh the page, open the
Heads Up Display bar, and select the “MAB Beta” option on the right hand side.
02
Creating
a Headline Experiment
Select a Headline to Test
When entering the Headline Testing mode, the colored
pins on your homepage will disappear. As you hover
your mouse over headline links, they will become
highlighted with an edit pin.
Click on the headline or pin to initiate a test.
Note: If the headline you want to test is nested
together with an image inside a single anchor tag,
you must click directly on the headline. This will
only work if the headline is in a <div> tag nested
in the larger anchor tag.
If you click on the image instead of the actual
headline, you will see this error message.
Additionally experiments cannot be created on
headlines that link off site.
Entering Alternative Headlines
Variant A will be auto-filled with the headline that
appears by default in the page and is not editable.
As additional headlines are entered they appear
inline on the page so you can preview their
appearance before testing.
While there’s no limit to the number of headline
variants that can be tested, remember that an
experiment’s progress will be slower the more
headlines are added.
Users can delete or edit any variant before
starting the test.
Select ‘start’ to begin testing the desired headline
variants.
03
In-Progress
Tests
Following a Test’s Progress
To monitor the progress of a test, select the pin next to the headline currently being tested.
Active deadlines experiments will be indicated with a pie chart
representing the number of headlines being tested and the
play percentage of each headline. Headlines with a higher
play percentage are ones that are performing better.
Note: Editing tests, adding variants, removing variants, and editing the copy of headline
variants will automatically stop and restart the test.
A trial is the number of visitors who have loaded the
page with a headline currently being tested. While we
can’t guarantee that a visitor saw one of the headline
variants, we do know that a variant was loaded into
their browser.
Play percentage represents how often
that particular headline variant is being
shown to visitors. A higher play
percentage indicates that the a headline
is performing “better”.
The CTR chart displays the percentage
of visitors who clicked on each headline
out of the total instances that it was
served.
The Engagement chart shows the
percentage of visitors who clicked on a
headline variant and stayed engaged on
the page for at least 15 seconds.
The center of the boxes (thick line)
represents the estimated CTR & engagement of a
particular headline.
There is a 50% probability that the headline’s
performance rate is somewhere between the right and
left edge of the box. There’s a 100% probability that the
true performance of a headline is somewhere between
the right and left edge of the whiskers (the thin lines).
04
Test
Conclusion and Archive
A Completed Test
A green check mark next to a headline indicates that the experiment is complete and a winner
has been determined. Select the pin to see the the winning headline.
Experiment Archive
To see a full report of all currently running, completed and
stopped tests – select the menu icon in the bottom right of
the screen.
The Test Archive displays a list of all current and completed headline experiments, the
engagement and click-through-rate performance graphs, and the number of trials for each
headline.
Use the Test Archive as a resource to analyze patterns in your audience’s behavior and
preferences.
05
Tips and Best Practices
Test from a Hypothesis.
Insights and best practices will develop faster if you’re systematic about the experiments.
Develop a hypothesis, such as using a ‘casual tone’, and compare and analyze results.
Test Contrasting Headlines.
The more differentiation in your headline styles, the more likely you are to see distinct
audience preferences. Headlines that simply reorder words are less likely to have significantly
different click-through.
Write Daily Wrap Reports.
Create a daily summary of the insights you gained from previous experiments and share them
with the team. This way all editors and content producers can benefit and learn from all
headline tests.
Know your Audiences.
Remember that homepage audiences are primarily your site’s most loyal readers and will
generally behave differently than your social media audience. Don't assume that a headline
that does well on your homepage will also do well at attracting visitors from Twitter or
Facebook.
Test in Higher Traffic Positions.
Once comfortable with the tools, don’t be shy about trying headlines tests in your site’s most
prominent positions. The higher traffic will help a winning headline to emerge more quickly.
Don’t Stop Tests Prematurely.
It might be tempting to stop a test when the play percentage for a headline goes up quickly to
70% or 80%. However, there will likely be instances where headlines jump to an early lead but
end up being inferior.
Don’t get Discouraged by Ties.
It’s possible that there are two great headlines that are attracting similar levels of engagement.
If you’re frequently seeing similar results, it might be that the headlines being tested are too
similar.
06
FAQs
How long do headline tests last?
As a headline test is actively measuring the click-through success of each headline as
compared to the others, there is no set length of time for a headline test. Generally, the length
of an experiment depends on three factors: the volume of traffic to your homepage, the
number of headlines in the experiment, and the extent to which one headline is actually better
than the other(s).
On a site where there are hundreds of trials per minute tests will likely take 10 to 30 minutes,
perhaps longer if the two headlines aren’t that different.
Since clicks and engaged clicks are counted separately, how much does each contribute to
the success value?
Clicks to engaged clicks are weighted at a ratio of 1:2. One way of thinking about it is that a
click on an article is worth ‘one point’, while fifteen seconds of subsequent engagement is
worth ‘two points’.
What happens to a test when the headline moves down or off the page?
When a headline moves from one location to another on your page the test will stop. To see
how the test did you can go to the test archive page to see what the results were, and you can
also restart the test at the new position.
Is there a max number of tests I can run simultaneously?
Currently only 5 headline experiments may be run simultaneously, but as we continue to
rollout this product and evaluate how our backend systems can scale to accommodate the
data, this number may increase.
What browsers will Headline Testing work in?
Tests can only be created in Google Chrome
Will visitors who come to my homepage multiple times see different headlines for the same
story?
No — we will always show visitors the same headline they saw initially.
07
Metrics Glossary
Trial
A trial is every time that a test headline is loaded in a page and served to a visitor. For example
a headline that ran a total of 2,000 trials was served to 2,000 unique cookies. On the other
hand, if an experiment had 5,000 trials it means that headlines were served a total of 5,000
times.
Click Through Rate (CTR)
Percentage of visitors who click on a given trial headline (can be lower than 1%).
Play Percentage
The percentage of people being served a given headline, as well as the confidence that
headline is “better”. For example at 65%, a headline is served to 65% of people and has a 65%
certainty that it is "better". Note that at 95% play percentage a headline "wins" and is played
100% of the time.
Quality Clicks
When a visitor clicks onto a story and subsequently spends at least 15 seconds of engaged
time, it is considered a quality click. Note whenever a visitor reaches the threshold of 15
engaged seconds it will be retroactively be counted as a quality click.
08