EVALUATING AUTOMATICALLY ESTIMATED CHORD SEQUENCES
Johan Pauwels and Geoffroy Peeters
STMS IRCAM-CNRS-UPMC
[email protected], [email protected]
ABSTRACT
In this paper, we perform an in-depth evaluation of a large number
of algorithms for chord estimation that have been submitted to the
MIREX competitions in 2010, 2011 and 2012. Therefore we first
present a rigorous scheme to describe evaluation methods in a sound,
unambiguous way that extends previous work specifically to take
into account the large variance in chord estimation vocabularies and
to perform evaluations on select sets of chords. Then we take a look
at the evaluation metrics used so far and propose some alternative
ones. Finally, we use these different methods to get a deeper insight
into the strengths of each of the competing algorithms and show that
the choice of evaluation measure greatly influences the ranking.
Index Terms— music information retrieval, chord estimation,
evaluation procedure, large scale evaluation
chord sequences, in which we reuse his work on the comparison of
chord pairs. It is intended as a replacement to his DBRE, especially
designed to deal with differences in estimation vocabulary between
distinct algorithms.
As far as large-scale evaluations of chord estimation algorithms
are concerned, the most known is of course the MIREX contest itself,
but as previously said, its study of the results is rather limited. More
extensive evaluations have been performed by [5, 8], but at the time
of their evaluations, the vocabularies of the systems under test were
mostly limited to major and minor triads. Our work will pay special
attention to the assessment of more complex chords.
Next, we will detail our scheme to describe evaluation methods
in Section 2. Then the measures used so far and some alternative
ones will be formulated according to this framework in Section 3.
They will afterwards be used to analyse the raw algorithmic output
in Section 4. Finally, we draw some conclusions in Section 5.
1. INTRODUCTION
Chord estimation has established itself as one of a growing number
of tasks in the field of Music Information Retrieval (MIR). Especially since its addition to the yearly returning MIREX contest in
2008, the field has seen a steady flow of published papers [1, 2]
and data sets [3, 4]. However, this increase in available data has
not led to a proportional increase in generally applicable knowledge.
Among the reasons for this, is the fact that not all automatic estimation algorithms generate the same vocabulary of chords and that
the evaluation procedure in most papers is tailored to the used chord
vocabulary, often described in an ad-hoc way that leaves some room
for interpretation. Therefore it is hard to exactly reproduce the evaluation method and to compare results between papers. Furthermore,
we feel that the richness of available data coming out of the MIREX
contest has been under-exploited. It is unique in its comparison of a
multitude of algorithms on the same data sets with the same evaluation methods, but only a minimal study of the results is being done
each year. Therefore, we take a deeper look in this paper at the data
of the 2010, 2011 and 2012 editions.
In order to make an evaluation reproducible, we need an unambiguous way to discuss metrics. The most thorough discussion
of chord sequence evaluation up to now has been done by Harte [5].
He established a general framework to describe evaluation measures.
Therefore he first studied the comparison of chord pairs and builds
upon that to arrive at the comparison of two chord sequences. For
the latter problem, he proposes the “dictionary-based recall evaluation” (DBRE) to study the performance for user-defined subsets of
chords. Alternative methods to get a deeper insight into the performance of algorithms include the reporting of results per chord
type [6] and of the amount of confusion with related chords [7, 8, 9].
The scheme we propose builds on the work by Harte. More specifically, we formulate a new framework to describe the evaluation of
This work was partly supported by the Quaero Program funded by Oseo
French agency.
2. EVALUATING: WHAT, WHERE AND HOW?
The objective of an evaluation procedure is to quantify the extent
to which an estimated chord sequence resembles a certain reference
sequence (preferably a manual annotation). This resemblance can
be quantified according to a number of different definitions, such as
resemblance in terms of chord segmentation [5, 6] or fragmentation
and estimated vocabulary [8], but in this text we limit ourselves exclusively to similarity in harmonic content, mainly because the other
ones have already been thoroughly treated by others. In order to
evaluate this similarity in harmonic content, we compare the estimated chord sequence with the reference sequence on a pair-wise
basis. In the literature, these pairs have been calculated in a framebased (where both reference and estimation get discretised on a grid
and every grid point gives a pair) or a segment-based fashion (where
segment pairs of variable length are formed by joining chord boundaries of both sequences). We adhere to previous conclusions [5, 6]
that state that the latter should be preferred because of its advantages
related to rounding errors and speed. A clear evaluation strategy
should then answer 3 questions about the comparison of these pairs
of reference and estimated chords: “what”, “where” and “how” do
we evaluate? These questions and their possible answers will be further explained in the following sections.
2.1. What: handling different chord vocabularies
Typically, the reference chord sequence is annotated much richer
than the estimated sequence, the former often having an unconstrained vocabulary only defined by the data itself while the latter
has an algorithm dependent vocabulary defined before the analysis
starts. An option to reduce the variability between the vocabularies
of the reference and one or more estimated sequences, is to define
a mapping between chords. In practice, it always is a mapping of
chord types: the root is preserved and complex chord types get
mapped to more simple chords (i.e. a reduction of the number of
constituting chromas). The “what” question thus asks what exactly
will be compared: full chord labels or their reduced versions.
Some mappings that we will use later on, are “triads” where
chords get reduced to their triads or “tetrads & triads” where every
chord gets mapped to its tetrad or in case that doesn’t exist, to its
triad. The no-chord symbol is mapped to itself for both. The most
simple mapping is of course “none”, leaving the chords as they are.
In the remainder of this text, we will also use a toy example for
illustrative purposes that consists of three mapping rules: 7→maj,
maj→maj and min→min.
2.2. Where: selecting points of interest
The “where” question deals with which segments should be included
in the evaluation. This should first and foremost only be determined
by the reference sequence, in order to avoid making the evaluation
depend on the estimated sequence. A mapping itself imposes implicit constraints on which pairs get evaluated. With each mapping
M an input domain CM I is associated for which the mapping is
defined and an output domain CM O which lists all possible chords
that can be produced by the mapping. The mapping can then be notated as a surjection M : CM I CM O . For our example mapping,
CM I = {maj, min, 7} and CM O = {maj, min}.
If the mapping is executed on a chord that is not part of CM I ,
its output is undefined. For instance, our example does not mention what to do with diminished chords, therefore the mapping of
diminished chords is undefined. Consequently, if a chord in the reference annotation does not belong to CM I , then that segment should
be skipped from evaluation. On the other hand, if a chord in the estimated sequence does not belong to CM I , the evaluation should fail
and a different mapping should be used. Summarized, if we represent a chord of the reference vocabulary as cref ∈ CREF and one of
the estimation vocabulary as cest ∈ CEST , then the segment pairs
(cref , cest ) that will be evaluated are those where cref ∈ CM I . The
actual labels that will be compared are M(cref ) and M(cest ) where
the mapping should be chosen such that CEST ⊆ CM I .
The input domain for our previously defined “triads” and
“tetrads & triads” mappings is the same, namely the no-chord and
all chords than can be mapped to a triad. We concretize this as all
chords that contain a major or minor third or one of a major second
or perfect fourth plus the no-chord. The “none” mapping obviously
has the collection of all possible chords CALL as input domain.
The most common use-case is to evaluate on all chords (or at
least all for which a mapping is defined), as this makes maximal use
of the available data. In other cases however, it might be useful to
restrict the evaluation to places where the reference chord belongs to
a subset. This can lead to a better understanding of the performance
of an algorithm on a certain category of chords, e.g. how well tetrads
are recognized compared to triads. One way to do this would be to
modify the mapping to map more input chords to “undefined” outputs, but that would mean that different mappings need to be used
for reference and estimated chord sequences. Therefore we propose
a more elegant solution where we keep the same mapping for both
sequences, and the input domain of the mappings as general as possible. Instead, we introduce the additional requirement that evaluation
only takes place when the reference chord cref belongs to a userdefined input limiting set CLI . In theory, this allows us to exactly
specify the chords for evaluation, but for some cases this might be
impractical. Suppose we want to have a more detailed look at the
performance of chords that are mapped to a major triad. For our example mapping, this would mean setting CLI = {maj, 7}, which is
still workable, but for mapping functions with a larger input domain,
this could mean that we need to list every possible combination of
a major triad and a number of tensions (e.g. maj7, maj9, maj11,
etc.). This quickly becomes impracticable. Therefore we introduce
a similar mechanism to limit the chords at the mapped side by specifying an output limiting set CLO . Applied to our example, the large
input limiting set CLI can then simply be replaced by the singleton
CLO = {maj}.
Both limiting sets are optional, their default value is CALL such
that they have no influence. Combining both limiting domains, this
gives that segments pairs (cref , cest ) will only be evaluated when
cref ∈ CLI ∩ CM I and M (cref ) ∈ M (CLI ∩ CM I ) ∩ CLO .
Taking the chord sequences in Figure 1, the initial number of
evaluation segments is 6: (Bdim, Dmin), (Dmin, Dmin), (Dmin,
Bmin), (G7, Bmin), (Cmaj, Bmin) and (Cmaj, Cmaj). When using
our toy mapping, the first segment pair will be discarded because the
mapping for Bdim is undefined. Additionally, setting CLO to {maj}
will further discard the second and third segment from the evaluation, because M(min) = min is not part of CLO . Only the last
three reference-estimation pairs then remain and due to the mapping
they will be evaluated as (Gmaj, Bmin), (Cmaj, Bmin) and (Cmaj,
Cmaj).
reference
estimation
evaluation
segments
Bdim
Dmin
Dmin
1
2
3
G7
Cmaj
Bmin
Cmaj
4
5
6
Fig. 1. Example chord sequences.
The introduction of the input limiting domain shows some
similarity to Harte’s “dictionary-based recall evaluation” (DBRE).
The main difference lies in the way how chords that don’t belong
to a user-defined dictionary are handled. Segments with out-ofdictionary chords in the annotation are discarded from evaluation,
as we propose, but segments with out-of-dictionary chords in the
estimated sequence get a score of 0 by default. For our example in
Figure 1 with dictionary {maj, 7}, this would mean that the first
3 segments get discarded, but also that the (G7, Bmin) and (Cmaj,
Bmin) segments get a score of 0 in all cases, whatever the evaluation
measure might be. We argue that these 2 pairs should be evaluated
as well, which can still lead to a score of 0, but does not have to.
The resulting value will depend on the way the pairs are compared,
where one can argue that on one hand, an estimation of Bmin for a
G7 reference fails to find the correct root, but on the other hand, that
all chromas of the estimated Bmin are part of the G7 chord.
2.3. How: assigning a score value
The last part of an evaluation scheme should answer the question
of how to assign a value to the retained reference-estimation pairs.
Therefore a scoring function S : CSR × CSE → R+ should be
defined where CSR and CSE are the reference, respectively estimation, chord vocabularies of the scoring function. At the minimum,
the scoring function should be able to handle all chords pairs coming out of the mapping subject to optional limiting. This means
that M(CM I ∩ CLI ) ∩ CLO ⊆ CSR for the reference sequence and
CM O ⊆ CSE for the estimation sequence need to hold, but it is advisable to make the scoring function input domains as large as possible.
Nonetheless, the combination of limiting the evaluation to a subset
and a mapping can make it easier to define the actual comparison by
reducing the number of reference-estimation combinations it needs
to be able to handle. For instance, in [10] the evaluation is only performed on segments annotated as triads in order to avoid the need to
define a scoring function between triads and more complex chords,
in particular between triads and tetrads that contain those triads.
This particular part of the evaluation scheme is thoroughly discussed in Harte’s thesis [5], so we’ll only briefly recap some major points. The measures can be divided into two categories: either
chords are evaluated as a whole, where the chromas are organised
with respect to the root, or as a bag of chromas, where all constituting chromas are considered equal (ordered and unordered set comparisons in Harte’s terms). A simple example of the first case assigns
a score of 1 to an exact correspondence, taking enharmonic equivalence into account and ignoring the bass note, and 0 to other cases.
This scoring function will be referred to as “exact” in the remainder
of the text. For the bag of chromas approach, some examples of a
scoring function are chroma based recall and precision. Returning
to the (G7, Bmin) pair of the previous section, we see that its score
would be 0 using the ”exact” measure, but 0.75 and 1 for chroma
based recall, respectively precision.
2.4. Summary
To summarize, an evaluation measure can be unambiguously described by the combination of a mapping function M : CM I →
CM O , a scoring function S : CSR ×CSE → R+ and optionally input
and output limiting domains CLI and CLO . The average over multiple segments is computed by taking the average of all individual
scores weighted by their segment lengths. Whether these segments
come from the same or multiple songs is not important in this regard.
special treatment of the augmented and diminished chords, nor can
two-chroma chords ever be estimated correctly. Lastly, generating
superfluous chromas in the estimated chord is not penalized at all.
In comparison, the MIREX editions before 2010 used the “exact”
scoring function.
In addition to the “Mirex2010” evaluation measure, we will use
some other metrics. All have a scoring function that ignores the pitch
spelling. The first two, named “Triads” and “Tetrads” use the aforementioned “triads” and “tetrads & triads” mapping. The mapped
chords are then compared using the “exact” scoring function. From
these, we derive two more measures that use a limiting set, in contrast to the first two. “TriadsInput” is equal to “Triads” with the
addition of an input-limiting set that consists of all triads, including inversions. Next is “OnlyTetrads”, similar to “Tetrads” but with
an output limiting set containing all tetrads, but no triads. The following two metrics don’t use a mapping, but have a simple scoring
function. “Root” assigns a score of 1 only to chord pairs that have
the same root and “Bass” does the same for bass notes. When a bass
chroma is not explicitly given, we assume it is the root. The “Root”
metric has also been used in MIREX 2008 and 2009 as an additional
evaluation. Finally, we have two measures, “ChromaPrecision” and
“ChromaRecall” that don’t use a mapping and calculate precision
and recall for the bags of chromas.
3. EVALUATION MEASURES
The results reported for MIREX editions 2010, 2011 and 2012 are
limited to a single value per song. While the measures used for editions 2008 and 2009 are thoroughly discussed in Harte’s thesis [5],
the only information about the measure of the 2010 edition (which
has been kept for the following editions) is its source code1 . Therefore we start by describing it in terms of our newly defined scheme
and comparing it with the previous evaluation versions. The advantages and drawbacks of these methods are then discussed, after
which some alternative measures are proposed.
In contrast to the 2008 and 2009 versions, the 2010 edition
doesn’t use a mapping. This is of course the most simple solution,
but has the disadvantage that the scoring function needs to be defined for every possible chord pair. The 2008 and 2009 versions
followed an orthogonal approach, all chords are mapped to either
major, minor or the no-chord. A particularity of both mappings,
although they are not exactly equal, is that every possible chord
is mapped to one of these three options, so there is no input for
which the mapping is undefined. While it has the advantage that
absolutely all available data is used for the evaluation, some of the
mappings are rather stretched and cause a bias towards major chord
types. A more detailed description of both mappings as well as their
criticisms can be found in [5].
The scoring function from MIREX 2010 is of the bag of chromas category. First, reference and estimated chords (including their
bass notes) are converted into a set of chromas. Then the cardinality
of the intersection is taken, ignoring pitch spelling. If it is 3 or more,
the segment receives a score of 1, otherwise its score is 0. For segments whose reference chord can be mapped to an augmented or a
diminished triad, the threshold is lowered to 2 or more chromas. Of
course, segments annotated with the no-chord, thus having an empty
chroma set, are only assigned a score of 1 when the estimation also
equals the no-chord symbol.
This method suffers from some important drawbacks. First of
all, like all bag of chromas scoring functions, the root chroma looses
its special position among other chromas: no distinction is made
between chords that differ in root, but contain the same chromas.
Secondly, there are no musicological underpinnings that warrant the
1 http://nemadiy.googlecode.com/svn/
4. A NEW LOOK AT EXISTING DATA
Since the introduction of the NEMA (Networked Environment for
Music Analysis) platform [23] for MIREX 2010, all algorithmic output and the used ground truth has been publicly available, offering
us output from 42 systems for two data sets. The first one has been
used in editions 2010, 2011 and 2012 and is named “Isophonics”
[3]. It consists for the main part of Beatles songs (180) with some
additional songs by Queen (19) and Zweieck (18). The second one
has only been used in 2012 and is called “Billboard” [4]. It contains
197 songs that have charted in the Billboard 100 between 1958 and
1991. Unfortunately, only the annotations according to triads (without inversions) are publicly available for the latter set, therefore it
is exclusively used in combination with the “Mirex2010”, “Triads”,
“Root” and “ChromaRecall” metrics. For other mappings, it is impossible to verify whether any chromas that have been estimated in
addition to the triad are correct and just haven’t been annotated or
whether they result from an incorrect estimation. Due to space constraints we have not printed all results here, but a selection of the
most interesting can be found in Table 2 and Table 1 for the “Billboard” and “Isophonics” sets respectively. We retained the letter
identifiers for the algorithms as used on the MIREX website2 , with
the addition of the year in which they participated. The complete
results for all the algorithms, as well as the algorithmic output used
to calculate them, can be found on-line3 .
Comparing the two data sets, we see that the best results on the
“Billboard” set are 10 % lower than on the “Isophonics” set, and
this for all evaluation measures. This can be explained by the fact
the “Isophonics” set has been publicly available before the contest,
while the “Billboard” set has been secret. It is therefore likely that
algorithms are optimised for the former.
The “Triads” scores are consistently lower than the “Mirex2010”4
ones, but the extent to which they are lesser depends on the algorithm. For example, the algorithm with the best “Mirex2010” score,
2 http://www.music-ir.org/mirex/wiki/MIREX_HOME
3 https://github.com/jpauwels/mirex-tools
4 Our reported results for the “Mirex2010” score differ somewhat from
the “Weighted average overlap ratio” score on the MIREX site because of
differences in the evaluation of no-chords and in the chord parsing
Algorithm
Mirex2010
Triads
Tetrads
TriadsInput
TetradsOnly
Root
Bass
ChromaRecall
ChromaPrecision
77.60
77.79
76.95
77.96
77.42
73.07
77.09
79.29
81.40
80.35
81.40
77.14
73.32
82.15
74.76
76.54
77.13
76.53
76.42
74.46
71.80
75.99
78.08
80.69
79.54
79.01
76.18
72.13
70.50
73.68
65.95
66.79
65.08
65.56
53.39
62.47
65.93
67.55
73.88
68.51
68.36
65.23
62.61
63.88
64.00
78.21
79.29
77.30
78.98
76.81
74.37
78.41
80.15
82.13
81.34
81.17
77.35
74.38
76.00
76.19
0.00
0.00
0.00
17.66
36.32
0.00
0.00
0.00
52.65
0.00
46.12
0.00
0.00
19.24
0.00
80.28
80.99
79.65
79.69
79.54
75.76
79.34
81.65
82.92
82.92
81.42
80.08
75.78
72.80
76.80
77.68
79.39
78.62
76.83
78.88
75.21
77.88
79.93
82.06
82.42
81.01
78.47
75.50
81.93
75.57
82.10
81.97
80.49
82.91
84.05
79.34
81.04
82.95
86.13
83.48
86.30
80.83
79.11
87.09
80.07
85.60
85.33
83.99
84.49
80.47
82.69
84.46
86.47
86.61
86.97
85.67
84.37
82.49
86.04
83.43
CWB1 (’10) [2]
EW4 (’10) [11]
KO1 (’10) [12]
MD1 (’10) [6]
MM1 (’10) [13]
OFG1 (’10) [14]
UUOS1 (’10/’11) [15]
CB2 (’11) [16]
KO1 (’11/’12) [17]
NM1 (’11) [18]
NMSD3 (’11) [18]
CCSS1 (’12) [19]
NG1 (’12) [20]
NMSD1 (’12) [18]
PMP1 (’12) [10]
Table 1. Evaluation scores for the “Isophonics” set.
Algorithm
CCSS1 (’12) [19]
DMW1 (’12) [21]
KO1 (’11/’12) [17]
NG1 (’12) [20]
NMSD4 (’12) [22]
PMP1 (’12) [10]
Mirex2010
Triads
Root
ChromaRecall
66.21
62.65
69.80
62.49
72.51
67.29
65.90
62.33
69.31
62.27
60.45
67.01
71.33
69.71
73.91
66.99
63.72
70.96
75.97
74.81
79.03
74.56
81.40
77.10
Table 2. Evaluation scores for the “Billboard” set.
“NMSD1/4 (’12)”, decreases the most in the ranking. This is because the “Mirex2010” score does not require the root to be correctly
estimated to get a good score, whereas the “Triads” score does. This
explanation is corroborated by the “Root” scores. Consequently, we
conclude that this algorithm is very capable of retrieving chromas in
a chord, but does not manage to identify the root very well.
Overall, the score lowers even more for “Tetrads”, but here too
some distinctions can be made between algorithms. For most of
them, the relative decrease is almost 15 %, but there are outliers in
both directions. The relative decrease for “MM1 (’10)” is almost
30 %, significantly worse than when no tetrads would be estimated
at all, while the decrease for “KO1 (’11/’12)” and “NMSD1 (’12)” is
less than 10 %. The picture gets clearer when we only look at chords
that are mapped to tetrads (the “TetradsOnly” score). Most algorithms that decrease 15 % in score actually can not generate triads
at all. The 15 % just corresponds to the proportion of the annotations that are mapped to a tetrad. Only 5 algorithms do generate
tetrads, but their estimation is lower than for exact triads (“TriadsInput” score) in all cases, ranging between 18 % and 53 % instead
of 77–82 %. However, the performance on the tetrads themselves
does not entirely reflect the ranking of “Tetrads” (in this case, “MM1
(’10)” is significantly better than “NMSD1 (’12)”). This brings us to
the conclusion that due to the relatively small proportion of tetrads to
triads, attention should be paid as to not overestimate triads as tetrads
before any possible gain due to the increased chord vocabulary can
be expected. It should be noted that the frequency of occurrence of
tetrads in relation to triads is strongly dependent on genre, for example tetrads are more used in jazz than in rock music, so the results
are likely to change for other data sets, and as a result, different algorithms can have different “preferences” for certain genres.
So far, we have ignored all bass notes during evaluation. There
are however algorithms that have extended their vocabulary to be
able to generate chord inversions. The measure “Bass” is specifically
designed to evaluate this. It is somewhat surprising that the two best
performing algorithms, “NM1 (’11)” and “KO1 (’11/’12)”, can not
generate inversions, so by default they always estimate root position.
The explanation is that the ratio of inversions to ground positions
is rather small, just like the tetrads/triads ratio. So similarly to the
latter, the benefit of possibly estimating the right inversion is not
enough to outweigh the option to wrongly estimate the bass on a
root position, at least not with the current implementations.
As can be seen with the two previous metrics, it is obvious that a
strongly skewed chord frequency makes it hard to evaluate improvements on less frequently occurring chords. A way to deal with this
imbalance is to weigh all chords after mapping CM O equally instead
of according to duration. The drawback is that the mapping needs
to be chosen in function of the estimation algorithm such that there
are no mapped chords that cannot be generated by the algorithm (i.e.
M(CEST ) = CM O ), as that would strongly influence the average.
Consequently, it is useful for per algorithm evaluations, but not for
a large scale one, as that would necessitate a mapping that produces
the lowest common estimation vocabulary. In our case, it would be
a mapping with CM O = {maj, min} only, thereby defeating the
purpose of a more in-depth evaluation.
Lastly, we take a look at the bag of chromas evaluation measures. “ChromaRecall” is similar to the “Mirex2010” metric, but
without threshold that binarises the score. Therefore the results are
consistently higher. The best performing algorithms are those that
can generate tetrads, the others are penalised because they simply
cannot generate enough chromas at times. In combination with the
“ChromaPrecision” measure, we can conclude that of those 5 tetrad
generating algorithms, “MM1 (’10)” clearly has a tendency to overestimate triads as tetrads, the others are better balanced.
5. CONCLUSIONS AND FURTHER WORK
We started this paper by presenting a scheme to describe chord evaluation measures, extending the work of Harte [5]. Then the measures
as used in the MIREX competitions from 2008 till 2012 have been
described according to the proposed scheme as well as some alternative ones. Finally, we used these metrics to analyse the raw data
coming from the competition. This showed us that the effects of a
change in evaluation procedure is not equal for all algorithms: some
maintain a consistent ranking, while others only excel in one particular measure. In general, we can conclude that extending the vocabulary of algorithms towards tetrads and inversions does not necessarily result in a better estimation, although the state-of-the-art is able
to estimate some tetrads correctly, albeit not as well as triads. Future
algorithmic improvements are therefore advised to verify progress in
that domain by using similar evaluation measures as presented here.
In the future, we’d like to carry out a similarly detailed evaluation of tetrads and inversions on the “Billboard” set if we can get
access to more detailed annotations. We also hope to integrate this
extended evaluation directly into the NEMA framework so that future MIREX editions can directly benefit from a more extensive evaluation.
6. REFERENCES
[1] H´el`ene Papadopoulos and Geoffroy Peeters, “Large-scale
study of chord estimation algorithms based on chroma representation and HMM,” in Proceedings of the International
Workshop on Content-Based Multimedia Indexing (CBMI),
2007, pp. 53–60.
[2] Taemin Cho, Ron J. Weiss, and Juan P. Bello, “Exploring common variations in state of the art chord recognition systems,”
in Proceedings of the Sound and Music Computing Conference,
2010.
[3] Matthias Mauch, Chris Cannam, Matthew Davies, Simon
Dixon, Chris Harte, Sefki Kolozali, Dan Tidhar, and Mark Sandler, “OMRAS2 metadata project 2009,” in Proceedings of the
10th International Conference on Music Information Retrieval
(ISMIR), 2009.
[4] John Ashley Burgoyne, Jonathan Wild, and Ichiro Fujinaga,
“An expert ground-truth set for audio chord recognition and
music analysis,” in Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR), 2011, pp.
633–638.
[5] Christopher Harte, Towards automatic extraction of harmony
information from music signals, Ph.D. thesis, Queen Mary,
University of London, August 2010.
[6] Matthias Mauch, Automatic chord transcription from audio
using computational models of musical context, Ph.D. thesis, School of Electronic Engineering and Computer Science
Queen Mary, University of London, March 2010.
[7] H´el`ene Papadopoulos, Joint estimation of musical content
information from an audio signals, Ph.D. thesis, Universit´e
Pierre et Marie Curie, July 2010.
[8] Laurent Oudre, Template-based chord recognition from audio
signals, Ph.D. thesis, Tel´ecom ParisTech, November 2010.
[13] Matthias Mauch and Simon Dixon, “Approximate note transcription for the improved identification of difficult chords,” in
Proceedings of the 11th International Conference on Music Information Retrieval (ISMIR), 2010, pp. 135–140.
[14] Laurent Oudre, C´edric F´evotte, and Yves Grenier, “Probabilistic template-based chord recognition,” IEEE Transactions on
Audio, Speech and Language Processing, vol. 19, no. 8, pp.
2249 – 2259, November 2011.
[15] Yushi Ueda, Yuki Uchiyama, Nobutaka Ono, and Shigeki
Sagayama, “Mirex 2010: Joint recognition of key and chord
from music audio signals using key-modulaton HMMs,” in
Proceedings of the Music Information Retrieval Evaluation Exchange (MIREX), 2010.
[16] Taemin Cho and Juan Pablo Bello, “A feature smoothing
method for chord recognition using recurrence plots,” in Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR), 2011, pp. 651–656.
[17] Maksim Khadkevich and Maurizio Omologo,
“Timefrequency reassigned features for automatic chord recognition,” in Proceedings of the IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP), December
2011, pp. 181 – 184.
[18] Yizhao Ni, Matt McVicar, Ra´ul Santos-Rodr´ıguez, and Tijl
De Bie, “An end-to-end machine learning system for harmonic
analysis of music,” IEEE Transactions on Audio, Speech and
Language Processing, vol. 20, no. 6, pp. 1771–1783, August
2012.
[19] Ruofeng Chen, Weibin Shen, Ajay Srinivasamurthy, and Parag
Chordia, “Chord recognition using duration-explicit hidden
Markov models,” in Proceedings of the 13th International
Conference on Music Information Retrieval (ISMIR), 2012, pp.
445 – 450.
[9] Maksim Khadkevich, Music signal processing for automatic
extraction of harmonic and rhythmic information, Ph.D. thesis,
University of Trento, 2011.
[20] Nikolay Glazyrin, “Audio chord estimation using chroma reduced spectrogram and self-similarity,” in Proceedings of the
Music Information Retrieval Evaluation Exchange (MIREX),
2012.
[10] Johan Pauwels, Jean-Pierre Martens, and Marc Leman, “Improving the key extraction accuracy of a simultaneous key and
chord estimation system,” in Proceedings of the International
Workshop on Advances in Music Information Research (AdMIRe), 2011.
[21] W. Bas de Haas, Jos´e Pedro Magalh˜aes, and Frans Wiering,
“Improving audio chord transcription by exploiting harmonic
and metric knowledge,” in Proceedings of the 13th International Conference on Music Information Retrieval (ISMIR),
2012, pp. 295 – 300.
[11] Adrian Weller, Daniel Ellis, and Tony Jebara, “Structured prediction models for chord transcription of music audio,” in
Proceedings of the IEEE International Conference on Machine
Learning and Applications (ICMLA), 2009, pp. 590 – 595.
[22] Yizhao Ni, Matt McVicar, Ra´ul Santos-Rodr´ıguez, and Tijl
De Bie, “Using hyper-genre training to explore genre information for automatic chord estimation,” in Proceedings of the
13th International Conference on Music Information Retrieval
(ISMIR), 2012, pp. 109 – 114.
[12] Maksim Khadkevich and Maurizio Omologo, “Use of hidden
Markov models and factored language models for automatic
chord recognition,” in Proceedings of the 10th International
Conference on Music Information Retrieval (ISMIR), 2009, pp.
561–566.
[23] Kris West, Amit Kumar, Andrew Shirk, Guojun Zhu,
J. Stephen Downie, Andreas Ehmann, and Mert Bay, “The
networked environment for music analysis (NEMA),” in IEEE
Fourth International Workshop on Scientific Workflows (SWF),
2010, pp. 314 – 317.