Trend Detection and Analysis
on Twitter
Benjamin Räthlein
Henning Muszynski
Lukas Masuch
2
Agenda
Motivation
Architecture
Data Preparation
Trend Analysis
Analyzed Trends
Conclusion
3
Motivation
Predict the stock market in real time
source
“Successful results of mainly research-based projects
helped to open up new business opportunities”
Automatic crime prediction
source
Detecting influenza epidemics
source
4
Twitter
5
Architecture
Twitter Streaming API (Twython)
Early Trend Detector
Bag of Words
Bag-of-words (Hashtags, Mentions)
Bags
Count
#newyear
7
#christmas
6
@bigdata
2
@sap
3
6
Architecture
Twitter Streaming API (Twython)
Early Trend Detector
Statistical Measurement
Bag-of-words (Hashtags, Mentions)
Statistical Measurement
(growth, average usage, retweets, participating users…)
Report statistics (every 20 minutes):
• Total hashtags & user mentions
• Hashtag/mentions count
• Usage growth per hashtag/mention
• Participating users per hashtag/mention
• Retweet count per hashtag/mention
7
Architecture
Twitter Streaming API (Twython)
Early Trend Detector
Bag-of-words (Hashtags, Mentions)
Statistical Measurement
(growth, average usage, retweets, participating users…)
Time Series Analysis
Anomaly Detection
Time Series Analysis
Anomaly detection using:
• Relative & absolute fluctuation
• Total occurrences (sum)
• Minimum occurrences
• Maximum occurrences
• Average occurrences
Calculated for every hashtag / user mention
Every 2 / 4 hours based on reports
8
Architecture
Twitter Streaming API (Twython)
Trend Analyzer
Stop Word Removal
Text Preprocessing (Python NLTK)
Lowercasing & tokenizing
URL & stopword removal
This sample text shows which words will
be removed when applying stop word
removal. Mostly words like the, a or and.
9
Architecture
Twitter Streaming API (Twython)
Stemming
Trend Analyzer
Text Preprocessing (Python NLTK)
Lowercasing & tokenizing
Amazing
URL & stopword removal
Word stemming
Amazement
Amazed
amaze
10
Architecture
Twitter Streaming API (Twython)
Sentiment Analysis
Trend Analyzer
Text Preprocessing (Python NLTK)
Lowercasing & tokenizing
URL & stopword removal
Word stemming
Sentiment Analysis
I hate cookies
I love cookies
11
Architecture
Twitter Streaming API (Twython)
Trend Analyzer
Topic Modeling
Text Preprocessing (Python NLTK)
Lowercasing & tokenizing
URL & stopword removal
Word stemming
Sentiment Analysis
Topic Modeling (LDA)
Trend Classification
Topics
• …
• …
• …
14
Architecture
Twitter Streaming API (Twython)
Early Trend Detector
Trend Analyzer
GeoSpatial Visualization
CartoDB
Bag-of-words (Hashtags, Mentions)
Text Preprocessing (Python NLTK)
Statistical Measurement
Lowercasing & tokenizing
(growth, average usage, retweets, participating users…)
Time Series Analysis
URL & stopword removal
Word stemming
Sentiment Analysis
Wordcloud Visualization
Wordfreq.js
Topic Modeling (LDA)
Anomaly Detection
Trend Classification
Wordcloud2.js
15
Analyzed Trends
16
Limitations
Tweets collected: 38 million (70GB)
Only English tweets from the USA
Twitter Streaming API
17
New Year
Time Series
18
New Year
Word Cloud
19
New Year
Geospatial Analysis
Midnight Los Angeles
Midnight New York
20
New Year
Sentiment Analysis
Home sick on #nye. Horrible timing
stupid cold. Ugh. My date is my
couch & pillow watching.
#HappyNewYear everyone.
#HappyNewYear from the Youth for
Astronomy and Engineering Program
at Space Telescope Science Institute!
Happy New Year! Last year was
Positive
Neutral
Negative
amazing, and here’s to another great
year of love & happiness! #NYE2015
21
Air Asia Tragedy
22
Air Asia Tragedy
Time Series
23
Air Asia Tragedy
Word Cloud
24
Air Asia Tragedy
Topic Modeling
News
Search for the Plane
airasia, missing, flight, air,
Indonesia, singapore, asia
airasia, missing, plane, find,
plane, world, technology
Sympathy
Cause
Prayers, families, thoughts,
airasia, crash, thought, airfrance
airasia, weather, flight,
pilots, fly, bad, path
International Help
raaf, butterworth, china, australia,
Russia, trndnl, trending
25
Air Asia Tragedy
Sentiment Analysis
Prayers are USELESS! Stop repeating
meaningless crap, pretending that
you care … #PrayForAirAsia #QZ8501
#GrowABrain #ReligousNonsense
#BREAKING #AirAsia Flight #8501
likely “at the bottom of the sea”
rescue officials says.
May God’s great love shine on the
Neutral
Negative
Positive
families and loved ones of all
passengers and crew #AirAsia #8501
26
Air Asia Tragedy
Google Trends Comparison
Google Trends
Twitter Sample
27
Air Asia Tragedy
Google Trends Comparison
Google Trends
Twitter Sample
28
Sony Hack
29
Sony Hack
Time Series
30
Sony Hack
Word Cloud
31
Sony Hack
Topic Modeling
Christmas Release
Reviews
theinterview, christmas, day,
theaters, freedom, theater, showing
theinterview, jamesfrancotv, sethrogen,
movie, interview, funny, hilarious
News
Insider Joke
theinterview, sonypictures, sony,
movie, korea, north, interview
theinterview, aint, hate, cuz,
jealous, anus, peanutbutter
Suspicions
northkorea, sonyhack, korea,
north, internet, sony, amp
32
Sony Hack
Geospatial Analysis
33
Sony Hack
Sentiment Analysis
#TheInterview SUCKS!!! @sethrogen
Like I knew it would #Stupid
#NotFunny
#Sony says #TheInterview made
more than $1 million at the box office
on in 1 single day on Dec. 25.
Happy I joined my fellow Americans
in the great #TheInterview Christmas
Day Viewing. Plus it was pretty funny,
Neutral
Negative
Positive
truth be told.
34
Network Outage
35
Network Outage
Time Series
36
Network Outage
Word Cloud
37
Network Outage
Topic Modeling
Network Error
Xbox Down
xbox, psn, sign, connect,
live, error, account, issues
xbox, christmas, play, xboxlivedown,
live, xboxlive, xboxsupport, day
Hacker Group
Connection between Hacks
fuckcrucifix, lizardmafia, lizardsquad,
fuck,lizard, squad, finestsquad, stop
xbox, playstation, watch, movie,
fuckcrucifix, north, korea, interview
Caused Damage
Restored
playstation, dollar, psn, company,
lizardsquad, sony, billion, multi
psn, back, playstation, online,
askplaystation, network, psndown, working
38
Network Outage
Sentiment Analysis
@XboxSupport f*** your servers, a
big ass company like you should
handle these teenage kids, terrible
@AskPlayStation when will the
service be back online because it says
there’s maintenance?
@PlayStation thanks for the great
year. I am sure this new year will be
amazing. Don’t allow yourselves to
Neutral
Negative
Positive
be hacked ever again.
39
Conclusion
High quality insights into world’s interest
Maintaining a high data quality is important
Twitter is very good for detecting and predicting trends
40
#Questions
Benjamin Räthlein
@B3nRa
Henning Muszynski
@henningmus
Lukas Masuch
@LukasMasuch