1. style and fashion
2. accessories
3. gloves

GloVe: Global Vectors for Word Representation
Jeffrey Pennington, Richard Socher, Christopher D. Manning
Presented by Chris Kedzie
March 25, 2015
Chris Kedzie
GloVe
March 25, 2015
1 / 30
Overview
1
Introduction
2
Problem
3
GloVe Model
4
Experiments
Chris Kedzie
GloVe
March 25, 2015
2 / 30
GloVe
1
Introduction
2
Problem
3
GloVe Model
4
Experiments
Chris Kedzie
GloVe
March 25, 2015
3 / 30
Word Representations: A history
Chris Kedzie
GloVe
March 25, 2015
4 / 30
Neural Language Models – Recurrent NNLM
ht−1
..
.
..
.
wt
ht
ot+1
Chris Kedzie
GloVe
March 25, 2015
5 / 30
Neural Language Models – Recurrent NNLM
ht−1
..
.
..
.
wt
ht
ot+1
Chris Kedzie
GloVe
March 25, 2015
5 / 30
Neural Language Models – Recurrent NNLM
ht−1
..
.
..
.
wt
ht
ot+1
Chris Kedzie
GloVe
March 25, 2015
5 / 30
Neural Language Models – Continuous BOW
wt−2
wt−1
..
.
..
.
wt+1
wavg
ot
wt+2
Chris Kedzie
GloVe
March 25, 2015
6 / 30
Neural Language Models – Continuous BOW
wt−2
wt−1
..
.
..
.
wt+1
wavg
ot
wt+2
Chris Kedzie
GloVe
March 25, 2015
6 / 30
Linear Relationships
Semantic
wking − wman + wwoman ≈ wqueen
Syntactic
weasy − weasiest + wluckiest ≈ wlucky
Chris Kedzie
GloVe
March 25, 2015
7 / 30
Scalable Embedding Learning
Noise Contrastive Estimation
wt−2
wt−1
..
.
..
.
wt+1
wavg
ot
wt+2
Chris Kedzie
GloVe
March 25, 2015
8 / 30
Scalable Embedding Learning
Noise Contrastive Estimation –no more normalization required!
wt−2
wt−1
..
.
..
.
wt+1
wavg
ot
wt+2
Chris Kedzie
GloVe
March 25, 2015
8 / 30
Scalable Embedding Learning
Noise Contrastive Estimation –no more normalization required!
wt−2
wt−1
..
.
w
wavg
..
.
wt+1
wavg
ot
wt+2
Chris Kedzie
GloVe
March 25, 2015
8 / 30
GloVe
1
Introduction
2
Problem
3
GloVe Model
4
Experiments
Chris Kedzie
GloVe
March 25, 2015
9 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imagined
event before a live audience in a specific place. The performers may communicate this experience to the audience through
combinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,
presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derived
from the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).
Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,
classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis defines
theatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general.
Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie
GloVe
March 25, 2015
10 / 30
Matrix Factorization Methods
e.g. SVD, COALS, etc. directly on co-occurrence matrix.
Chris Kedzie
GloVe
March 25, 2015
11 / 30
Matrix Factorization Methods
e.g. SVD, COALS, etc. directly on co-occurrence matrix.
Main drawback: frequent words like the and a have an outsized effect on
the representation learning.
Chris Kedzie
GloVe
March 25, 2015
11 / 30
GloVe
1
Introduction
2
Problem
3
GloVe Model
4
Experiments
Chris Kedzie
GloVe
March 25, 2015
12 / 30
GloVe Model
J=
V
X
2
f (Xij ) wiT w
˜j + bi + ˜bj − log Xij
i,j=1
Chris Kedzie
GloVe
March 25, 2015
13 / 30
Notation!
X ∈ RV ×V word co-occurrence matrix
Chris Kedzie
GloVe
March 25, 2015
14 / 30
Notation!
X ∈ RV ×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
Chris Kedzie
GloVe
March 25, 2015
14 / 30
Notation!
X ∈ RV ×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
P
Xi = Vk Xik total number of occurrences of word i in corpus
Chris Kedzie
GloVe
March 25, 2015
14 / 30
Notation!
X ∈ RV ×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
P
Xi = Vk Xik total number of occurrences of word i in corpus
X
Pij = P (j|i) = Xiji
context of word i
Chris Kedzie
a.k.a. probability of word j occurring within the
GloVe
March 25, 2015
14 / 30
Notation!
X ∈ RV ×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
P
Xi = Vk Xik total number of occurrences of word i in corpus
X
Pij = P (j|i) = Xiji
context of word i
a.k.a. probability of word j occurring within the
w ∈ Rd a word embedding of dimension d
Chris Kedzie
GloVe
March 25, 2015
14 / 30
Notation!
X ∈ RV ×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
P
Xi = Vk Xik total number of occurrences of word i in corpus
X
Pij = P (j|i) = Xiji
context of word i
a.k.a. probability of word j occurring within the
w ∈ Rd a word embedding of dimension d
w
˜ ∈ Rd a context word embedding of dimension d
Chris Kedzie
GloVe
March 25, 2015
14 / 30
Motivation
Prob. and Ratio
k = solid
k = gas
k = water
k = f ashion
P (k|ice)
1.9 × 10−4
6.6 × 10−5
3.0 × 10−3
1.7 × 10−5
P (k|steam)
2.2 × 10−5
7.8 × 10−4
2.2 × 10−3
1.8 × 10−5
P (k|ice)
P (k|steam)
8.9
8.5 × 10−2
1.36
0.96
Chris Kedzie
GloVe
March 25, 2015
15 / 30
Motivation
Prob. and Ratio
k = solid
k = gas
k = water
k = f ashion
P (k|ice)
1.9 × 10−4
6.6 × 10−5
3.0 × 10−3
1.7 × 10−5
P (k|steam)
2.2 × 10−5
7.8 × 10−4
2.2 × 10−3
1.8 × 10−5
P (k|ice)
P (k|steam)
8.9
8.5 × 10−2
1.36
0.96
Chris Kedzie
GloVe
March 25, 2015
15 / 30
Motivation
Prob. and Ratio
k = solid
k = gas
k = water
k = f ashion
P (k|ice)
1.9 × 10−4
6.6 × 10−5
3.0 × 10−3
1.7 × 10−5
P (k|steam)
2.2 × 10−5
7.8 × 10−4
2.2 × 10−3
1.8 × 10−5
P (k|ice)
P (k|steam)
8.9
8.5 × 10−2
1.36
0.96
Chris Kedzie
GloVe
March 25, 2015
15 / 30
Motivation
Prob. and Ratio
k = solid
k = gas
k = water
k = f ashion
P (k|ice)
1.9 × 10−4
6.6 × 10−5
3.0 × 10−3
1.7 × 10−5
P (k|steam)
2.2 × 10−5
7.8 × 10−4
2.2 × 10−3
1.8 × 10−5
P (k|ice)
P (k|steam)
8.9
8.5 × 10−2
1.36
0.96
Chris Kedzie
GloVe
March 25, 2015
15 / 30
Motivation
Prob. and Ratio
k = solid
k = gas
k = water
k = f ashion
P (k|ice)
1.9 × 10−4
6.6 × 10−5
3.0 × 10−3
1.7 × 10−5
P (k|steam)
2.2 × 10−5
7.8 × 10−4
2.2 × 10−3
1.8 × 10−5
P (k|ice)
P (k|steam)
8.9
8.5 × 10−2
1.36
0.96
Chris Kedzie
GloVe
March 25, 2015
15 / 30
Derivation
F (wi , wj , w
˜k ) =
Chris Kedzie
GloVe
Pik
Pjk
March 25, 2015
16 / 30
Derivation
F (wi , wj , w
˜k ) =
F should encode information in the ratio
Chris Kedzie
GloVe
Pik
Pjk
Pik
Pjk .
March 25, 2015
16 / 30
Derivation
F (wi − wj , w
˜k ) =
Chris Kedzie
GloVe
Pik
Pjk
March 25, 2015
17 / 30
Derivation
Pik
F (wi − wj )T w
˜k =
Pjk
Chris Kedzie
GloVe
March 25, 2015
18 / 30
Derivation
Pik
F (wi − wj )T w
˜k =
Pjk
Some more desiderata:
F should be unchanged by exchanging w → w
˜ and X → X T
Chris Kedzie
GloVe
March 25, 2015
18 / 30
Derivation
Pik
F (wi − wj )T w
˜k =
Pjk
Some more desiderata:
F should be unchanged by exchanging w → w
˜ and X → X T
This requires that
T
F (wi − wj ) w
˜k
F wiT w
˜k
= F wjT w
˜k
⇒ F (wiT w
˜k ) = Pik
Chris Kedzie
GloVe
March 25, 2015
18 / 30
Derivation
Pik
F (wi − wj )T w
˜k =
Pjk
Some more desiderata:
F should be unchanged by exchanging w → w
˜ and X → X T
This requires that
F wiT w
˜k
T
F (wi − wj ) w
˜k = F wjT w
˜k
⇒ F (wiT w
˜k ) = Pik
F
Chris Kedzie
wiT w
˜k
−
wjT w
˜k
GloVe
F wiT w
˜k
= F wjT w
˜k
March 25, 2015
18 / 30
Derivation
Pik
F (wi − wj )T w
˜k =
Pjk
Some more desiderata:
F should be unchanged by exchanging w → w
˜ and X → X T
This requires that
F wiT w
˜k
T
F (wi − wj ) w
˜k = F wjT w
˜k
⇒ F (wiT w
˜k ) = Pik
exp
Chris Kedzie
wiT w
˜k
−
wjT w
˜k
GloVe
exp wiT w
˜k
=
exp wjT w
˜k
March 25, 2015
18 / 30
Derivation
exp
Chris Kedzie
wiT w
˜k
−
wjT w
˜k
exp wiT w
˜k
=
˜k
exp wjT w
GloVe
March 25, 2015
19 / 30
Derivation
exp
wiT w
˜k
−
wjT w
˜k
exp wiT w
˜k
=
˜k
exp wjT w
wiT w
˜k = log Pik = log Xik − log Xi
Chris Kedzie
GloVe
March 25, 2015
19 / 30
Derivation
exp
wiT w
˜k
−
wjT w
˜k
exp wiT w
˜k
=
˜k
exp wjT w
wiT w
˜k = log Pik = log Xik − log Xi
Chris Kedzie
GloVe
March 25, 2015
19 / 30
Derivation
exp
wiT w
˜k
−
wjT w
˜k
exp wiT w
˜k
=
˜k
exp wjT w
wiT w
˜k = log Pik = log Xik − log Xi
wT w
˜k = log Xik − bi − ˜bk
i
Chris Kedzie
GloVe
March 25, 2015
19 / 30
Derivation
exp
wiT w
˜k
−
wjT w
˜k
exp wiT w
˜k
=
˜k
exp wjT w
wiT w
˜k = log Pik = log Xik − log Xi
wT w
˜k = log Xik − bi − ˜bk
i
wiT w
˜k
Chris Kedzie
+ bi + ˜bk = log Xik
GloVe
March 25, 2015
19 / 30
Derivation
This suggests a least-squares objective function,
J=
V X
wiT w
˜j + bi + ˜bj − log Xij
2
i,j=1
Chris Kedzie
GloVe
March 25, 2015
20 / 30
Derivation
This suggests a least-squares objective function, but...
J=
V X
wiT w
˜j + bi + ˜bj − log Xij
2
i,j=1
Chris Kedzie
GloVe
March 25, 2015
20 / 30
Derivation
This suggests a least-squares objective function, but...
J=
V X
wiT w
˜j + bi + ˜bj − log Xij
2
i,j=1
Chris Kedzie
GloVe
March 25, 2015
20 / 30
Derivation
This suggests a least-squares objective function, but...
J=
V X
wiT w
˜j + bi + ˜bj − log Xij
2
i,j=1
Chris Kedzie
GloVe
March 25, 2015
20 / 30
Derivation
This suggests a least-squares objective function, but...
J=
V X
wiT w
˜j + bi + ˜bj − log Xij
2
i,j=1
⇒J =
V
X
2
f (Xij ) wiT w
˜j + bi + ˜bj − log Xij
i,j=1
where f has the following desiderata:
1
f (0) = 0
2
f (x) should be non-decreasing so that rare co-occurrences are not
overweighted.
3
f (x) should be relatively small for large values of x, so that frequent
co-occurrences are not overweighted.
Chris Kedzie
GloVe
March 25, 2015
20 / 30
Derivation
This suggests a least-squares objective function, but...
V X
J=
wiT w
˜j
+ bi + ˜bj − log Xij
2
i,j=1
⇒J =
V
X
2
˜j + bi + ˜bj − log Xij
f (Xij ) wiT w
i,j=1
(
xmax
where f (x) =
1
Chris Kedzie
x
α
if x < xmax
otherwise
GloVe
March 25, 2015
20 / 30
Weighting Function
Chris Kedzie
GloVe
March 25, 2015
21 / 30
Optimization
J=
V
X
2
f (Xij ) wiT w
˜j + bi + ˜bj − log Xij
i,j=1
(
where f (x) =
α
x
xmax
1
In this paper: α =
3
4
if x < xmax
otherwise
and xmax = 100.
The model is trained using AdaGrad and stochastically sampling non-zero
elements from X. An initial learning rate of .05 is used.
Chris Kedzie
GloVe
March 25, 2015
22 / 30
GloVe
1
Introduction
2
Problem
3
GloVe Model
4
Experiments
Chris Kedzie
GloVe
March 25, 2015
23 / 30
Word Analogies
a is to b as c to ?
wb − wa + wc
Chris Kedzie
GloVe
March 25, 2015
24 / 30
Word Analogies
a is to b as c to ?
Paris is to France as Tokyo is to ?
wb − wa + wc
Chris Kedzie
GloVe
March 25, 2015
24 / 30
Word Analogies
a is to b as c to ?
Paris is to France as Tokyo is to ?
arg maxw0 cosine-sim(wb − wa + wc , w0 )
Chris Kedzie
GloVe
March 25, 2015
24 / 30
Word Analogies – Results
Chris Kedzie
GloVe
March 25, 2015
25 / 30
Word Similarities
Humans scored similarity of word pairs.
word 1
king
king
king
word 2
cabbage
queen
rook
Chris Kedzie
human score (mean) (1-10)
0.23
8.58
5.92
GloVe
cosine-similarity (-1, 1)
0.11
0.78
0.25
March 25, 2015
26 / 30
Word Similarities
Humans scored similarity of word pairs.
word 1
king
king
king
word 2
cabbage
queen
rook
Chris Kedzie
human score (mean) (1-10)
0.23
8.58
5.92
GloVe
cosine-similarity (-1, 1)
0.11
0.78
0.25
March 25, 2015
26 / 30
Word Similarities
Humans scored similarity of word pairs.
word 1
king
king
king
word 2
cabbage
queen
rook
human score (mean) (1-10)
0.23
8.58
5.92
cosine-similarity (-1, 1)
0.11
0.78
0.25
Embeddings are evaluated by Spearman rank correlation of human scores
to cosine similarity.
Chris Kedzie
GloVe
March 25, 2015
26 / 30
Word Similarities – Results
Chris Kedzie
GloVe
March 25, 2015
27 / 30
Named Entity Recognition
NER is a sequence tagging task where the goal is to identify named
entities:
Jim bought 300 shares of Acme Corp
.
in 2006 .
B-PER
O
O
O O B-ORG I-ORG I-ORG O B-TIME O
Chris Kedzie
GloVe
March 25, 2015
28 / 30
Named Entity Recognition
NER is a sequence tagging task where the goal is to identify named
entities:
Jim bought 300 shares of Acme Corp
.
in 2006 .
B-PER
O
O
O O B-ORG I-ORG I-ORG O B-TIME O
Combined discrete features of existing system (Stanford NER).
Word embeddings were treated as additional features in a linear-chain CRF
model.
Chris Kedzie
GloVe
March 25, 2015
28 / 30
Named Entity Recognition – Results
Chris Kedzie
GloVe
March 25, 2015
29 / 30
The end! Thanks!
Chris Kedzie
GloVe
March 25, 2015
30 / 30