1
Predicting True Patterns of Cognitive Performance
from Noisy Data
W. Todd Maddox
W. K. Estes
University of Texas, Austin
Indiana University
In press at Psychonomic Bulletin and Review
Abstract
Starting from the premise that the purpose of cognitive modeling is to gain
information about cognitive processes of individuals, we develop a general theoretical
framework for assessment of models on the basis of tests of the models' ability to yield
information about true performance patterns of individual subjects and the processes
underlying them. To address the central problem that observed performance is a
composite of true performance and error, we present formal derivations concerning
inference from noisy data to true performance. Analyses of model fits to simulated data
illustrate the usefulness of our approach for coping with difficult issues of model
identifiability and testablity.
Introduction
In nearly all cognitive research, analyses of data are conducted by
application of statistical methods deriving from the family of linear statistical
models (analysis of variance and covariance, multivariate analysis). The methods
all depend on the assumption that any performance score constitutes a mixture
of two components: One represents true effects of experimental variables
(henceforth, the "true score") and one represents experimental error. Our
premise in this article is that the same assumption holds when scientific models
are applied to the same data. The importance of the assumption stems from our
view that the purpose of cognitive modeling is not simply to describe data but to
gain information about the cognitive processes of individuals that underlie
performance.
In the following sections, we first formalize the problem of inferring true
performance patterns from fits of models to composite data, and derive some
general relationships among observed performance scores, true scores, and
predicted scores generated by model fits. Second, using simulated data for which
the true scores and error components are known, we address the problem of
2
assessing a model's capability for recovering true parameter values from model
fits and generating useful inferences about patterns of true performance.
A Framework for Model-Based Inference of True Performance
Basic Terms and Functions
The presentation and illustration of the methods we propose refer to any
experiment to which we might apply a cognitive model in order to aid inferences about
the output of the experimental subjects' cognitive processes.
The theoretical framework for our methodology follows Estes (2002). Four
aspects of cognitive performance are distinguished: observed performance,
performance predicted by a model, true performance, and error. We denote these by P,
Pr, T, and e, respectively. Each term refers to the performance of an individual subject
on a single experimental trial.
The term T represents the output of the individual's cognitive processes before
the output enters into generation of an observed response, P. The true response, T, is
assumed to have the form of some function of a set (or vector) of parameters, θ, that is,
T = f(θ).
(1)
The function may be explicitly defined, or it may be implicit in a computer program that
generates predictions of performance in the situation. Observed performance is
represented by
P = T + e.
(2)
The error term e is assumed to be normally distributed with a mean of zero and a
standard deviation σ that is unknown a priori but can be estimated from data. Our
specific assumption about the form of the error term is not the only possibility, but it has
the advantage of conforming to long-term usage in statistics and psychometrics.
An alternative expression for observed performance is
P = f(θ). + e,
(3)
obtained by substitution from Equation 1.
For models considered in this article, values of the parameters are not known a
priori, but upon application of the model to an experiment, the parameter values are
estimated from performance data. The estimated parameter vector is denoted θe, and
Pr, performance predicted by the model, is given by
Pr = f(θe),
(4)
where f(θe) is the right hand side of Equation 1 with θ replaced by its estimate.
With this conceptual machinery in hand, we are prepared to approach the
question of how outputs of model fits can yield information about properties of true
3
performance both at a quite general level and also as it arises in analyses of specific
models and experiments.
Derivation of General Relationships between Predicted and True Performance
To illustrate our approach at the more general level, we pose the question of
whether and how constant error, or bias, may arise if the disparities between observed
and predicted performance in the output of a model fit are used to infer corresponding
disparities between predicted and true performance.
We consider first the performance of a subject in a single episode under some
condition of an experiment, using the notation defined above but adding a subscript j (j =
1, 2, ... J) to P and the other terms in order to identify the condition referred to among a
set of J possibilities. Assuming that a model has been fitted to the subject's data, we
seek a relation between Pj - Prj and Tj - Prj. By appropriate application of Equations
1-4, we obtain the identity
Pj - Prj = Tj - Prj + ej.
(5)
For any single episode, the observed difference, Pj - Prj, predicts too high or too low a
value for the true difference, Tj - Prj, depending on the sign of ej. If the episode could
be replicated a large number of times with no change in values of the model's
parameters, mean error would tend to zero and Pj - Prj would be a perfect predictor of
Tj - Prj.
In cognitive research, unlimited replication with constant parameters may not be
possible, but cumulation of Pj - Prj over the conditions of an experiment can inform us
about the corresponding cumulation for Tj - Prj. Assuming that ej is sampled from the
same normal distribution on all trials and averaging the two sides of Equation 5 over
conditions, we obtain
(1/J)Σ( Pj - Prj) = (1/J)Σ(Tj - Prj + ej),
or
M(P - Pr) = M(T - Pr) + M(e).
(6)
If the means are computed for successively larger numbers of conditions (J = 1, 2, ...),
the sign of M(e) may fluctuate, but beyond some J, Jc, it becomes highly unlikely that
further fluctuations will be observed and nearly always the observed difference for J >
Jc, will consistently imply too large or too small a value for the true difference1. The
magnitude of this disparity will progressively decrease, of course, becoming vanishingly
small for sufficently large J.
{Figure 1 about here}
For a concrete illustration, Figure 1 presents plots of Equation 6 for
simulated data (curves of forgetting) of two hypothetical subjects whose true
performance is described by the power function Tn = n-.3, with n, corresponding
to j in the preceding derivation, denoting number of unit time intervals.
4
Performance scores were obtained by adding to Tn values of e sampled from
error distributions with σ equal to .02 or .04. Although an n of 100 is not enough
for the functions to reach asymptote, stable relationships between Pj - Prj and Tj
- Prj appear to have emerged by a sequence length of about 60 for the lower
noise and 80 for the higher noise subject.
To treat mean squared disparities, a derivation analogous to that of Equation 6
yields
(1/J)Σ(Pj - Prj)2 = M(T - Pr)2 + 2M(T - Pr)M(e) + M(e2).
(7)
For any value of Pj - Prj, the cross product on the right side of Equation 7 can be either
positive or negative, depending on the sign of M(e), but as the number of conditions, J,
increases, M(e) tends toward zero and and Equation 7 is increasingly well approximated
by
M[(P - Pr)2] = M[(T - Pr)2] + M(e2).
(8)
On the average, M(e2) is constant over J, so except for small values of J, we have
approximately a linear relation between M[(P - Pr)2] and M[(T - Pr)2].
Demonstrations of how these general results can translate into informative
analyses in a specific research situation are given in the next section.
Analyses of a Specific Model in a Research Context
Owing to the ready availability of relevant data, a convenient choice of a
cognitive model for analysis is the array model of recognition memory (Estes, 2002;
Estes & Maddox, 1995; Estes & Maddox, 2002), described in the Appendix. Our
procedure constitutes the following steps: (1) simulate an experiment in a computer
program; (2) generate artificial data representing the model's predictions of performance
in the experiment by a group of hypothetical subjects; (3) recode the artificial
performance scores in accord with our assumptions about their constituents as
expressed in Equation 2; (4) by application of a standard model-fitting program,
estimate the values of the model's parameters needed to predict the recoded data ; (5)
analyze the observed and predicted performance scores generated in the simulation
exactly as we would do for the corresponding real experiment.
Constructing a simulated data set. Our simulation data are based on a fit of the
array model to performance data of 48 subjects in an experiment on study-test
recognition that we had conducted previously. The experimental conditions are factorial
combinations of shorter or longer study lists (4 vs. 8 items), weak or strong items (1 vs.
2 occurrences in the study list), and shorter or longer study time. The experiment,
henceforth termed the List-Strength experiment is representative of a class that has had
intensive use in tests of models of recognition memory (Ratcliff, Clark, & Shiffrin, 1990;
Shiffrin, Ratcliff, & Clark, 1990).
Using the parameter values estimated in the model fit2, a computer program
embodying the model generated sets of performance scores (subjects' estimates of the
5
probability that a test item is old) for 48 hypothetical subjects in two replications of the
experiment, henceforth denoted R1 and R2, with the same subjects but different
random samples of stimuli from the same item category (nonword letter strings).
However, these scores could not be used as they stand to constitute the
"observed" data for the simulations because they were error-free "true" scores as
defined in Equation 1. Thus, we added to the score for each individual on each trial an
error variable, the quantity ei in Equation 2, assumed to come from a population with a
mean of 0 and variance σ2. To obtain data at two levels of error, we drew values of ei
from this population with σ2 set equal to .01 for one simulation and to .02 for a second
simulation in each replication.
Analyzing Individual subject data. To bring out relevant properties of the
model fits, the first step is to examine for each subject and condition the difference
between the true value of the performance measure Pij and the value predicted by the
model, as illustrated for Subject #1 of the simulation in Table 1. The Conditions in Table
1 are the 8 combinations of list length, (LL4 or LL8), study duration (D1 and D2, for
short and long respectively), and study frequency (w and s for 1 and 2, respectively) for
old test items together with the associated new test items. The first column in Table 1,
headed True, gives the true Old ratings (Tj) of test items
{Table 1 about here}
for this subject in each condition; the second column gives the corresponding values
(Prj) predicted by the model at error variance level .01. The most conspicuous feature of
this tabulation is the accuracy with which the predicted scores for the standard model
track the true scores.
With similar tabulations accomplished for all 48 simulated subjects at both error
variance levels, the next step is to collate the results in a way that does not lose track of
the individuals. It is not feasible to do this for individual conditions, so we use T and Pr
means over the 12 conditions. These could be compared in frequency histograms, but
the necessary use of class intervals would obscure the results for individual subjects.
Thus, we use scatter plots of these means (for replication R1) as shown in Figure 2.
The upper panel of this figure, where the predicted scores have error
{ Figure 2 about here}
variance .01, shows no suggestion of any constant error (because all plotted points fall
on or near the main diagonal). For the lower panel, where the predicted scores have
error variance .02, the picture is similar except that there is evidence of some constant
error, the predicted values tending to fall slightly below the true values.
These results are compatible with expectation based on Equation 5 (with both
sides averaged over subjects). The sign of any constant error could not be predicted in
advance, but the magnitude should be greater in the variance .02 plot, as observed.
In the means for true-predicted performance, large positive errors of prediction
can be balanced by large negative errors, but the same is not true for a measure of
dispersion. Thus, we computed for each subject the sample standard deviation (SD) of
T - Pr over the 12 conditions, essentially the equivalent of MSE but defined on a more
familiar scale. Because the SDs are unbounded, scatterplots are not as informative as
6
for means, and we use instead plots of individual-subject SDs versus their rank order.
These are shown in Figure 3. For all but two of the 48 subjects, the plotted points fall
below about .04 SD, indicating goodness of agreement of Pr with T as good or nearly as
good as that shown for subject # 1 in Table 1.
{Figure 3 about here}
Because of its common usage, we include an auxiliary analysus of the correlation
between T and Pr. Values of Pearson r are shown as rank-order plots in Figure 4. The r
values are nearly all quite large and all are significant at the .001 level.
{Figure 4 about here}
Recovering True Values of a Model's Parameters
The next question to be addressed in this analysis is how well values of the true
parameters of a model are recovered when the model is fitted to simulated data that
contain error.
{ Table 2 about here}
For the fit of the array model to the simulation data, results in terms of mean
parameter values are summarized in Table 2. Rows of Table 2 correspond to the four
parameters of the array model, defined as in the Appendix. Columns give the true mean
values of the parameters and the estimates of them from the fits of the model to the
variance .01 and variance .02 simulation data. The estimates are not close to the true
means in absolute value, but in nearly every row the estimates move toward the true
value as error variance decreases. For both replications, the patterns of relative
magnitudes of the estimates at both variance levels mirror the patterns of true values.
To assess recovery of parameter values for individual subjects, we computed for
each parameter the average absolute differences between true values and estimates.
Illustrative results are given in Table 3 for the R1 simulation with variance .01. The large
ranges for the similarity parameters are due to a few outliers, and the median
differences signify close agreement between true values and estimates for individual
subjects. In contrast, the same is not true for the storage-probability parameters.
Generalizability
{ Table 3 about here}
For a final illustration of our approach, we consider an important, though
limited form of generalizability of a model termed "validation" by Busemeyer &
Wang, 2000: The parameters of a model are estimated from data under one set
of experimental conditions and used to predict quantitative aspects of data
obtained under a different set of conditions. Within each of the two replications of
the List-Strength Experiment, each subject yielded data under 12 conditions,
which were divided into two sets of 6 for this analysis3. The parameters of the
array model were estimated from fits to the simulation data (replications R1 and
R2, both with error variance .01) for sets A and B separately, and the estimates
7
from the fit of Set B were used to predict performance in Set A. The new feature
in our use of the validation procedure is that, whereas Busemeyer & Wang
(2000) were concerned with the correspondence of the predicted scores for Set
A with Set A performance, we focus on the goodness with which the Set A
predictions correspond to Set A true scores.
Table 4 shows mean true and predicted scores for Set A. Predictions of true
scores for Set A based on Set A parameters are extremely close, as anticipated,
providing a base line for the validation test. Of more interest, the pattern of Set A true
values is well predicted by generalization from the model fits to Set B.
{Table 4 about here}
Discussion
In this Discussion, we first consider in detail some issues that arose during
inspection of the results of applying our methods to simulated data, and, second,
address the issue of applicability of our findings to real research situations.
Accomplishments and Unfinished Business
Cognitive models have traditionally been selected and evaluated mainly
for capability to describe observed performance data. The main purpose of this
article is to point up the importance of taking explicit account also of models'
capabilities for extracting from data information about true performance and the
processes underlying it.
The first step in this effort was to present a provisional theoretical
framework for model-based inference from data to true performance. We say
"provisional," because, although the approach outlined seems promising, much
work remains to be done.
For reasons of tractability, we have assumed that the error component of
performance is localized in the translation of the output of cognitive processes
into observable performance. At present, we are not in a position to identify and
estimate other sources of error, for example, in the encoding stage of cognitive
processing, but the problem deserves attention in future work.
At a more technical level, our assumptions about the form of the error
distribution, adapted from statistical and psychometric models, may need
modification for some lines of application. In our theoretical expressions for
performance measures, starting with Equation 2, we assumed that error
components are sampled from the same normal distribution under all conditions
of an experiment, the distribution having a mean of zero and an unknown but
Because the confidence rating score used in our
constant variance σ2.
illustrative applications is bounded between 0 and 1, we may in future work need
to consider truncating the error distribution in order to keep scores for simulated
subjects within bounds. Also, for some applications it might be desirable to allow
the variance of the error distribution to vary over subjects or conditions. And if in
the research situation being simulated, subjects made discrete rather than
quantitatively graded responses, as by reporting judgments of "old" or "new" for
8
old or new test items in a recognition experiment, the assumption of normally
distributed error would be inappropriate, and it might be preferable to assume a
binomial distribution. All of the kinds of analyses we have reported could be
carried out with any of these alternative assumptions.
Analyses of the array model as applied to the list-strength experiment yielded
instructive insights that perhaps could not have been achieved in any other way. Some
of these involve recovery of true parameter values. Parameter recovery is an essential
aspect of the assessment of a model because of its bearing on issues of identifiability,
that is, the degree to which its parameters are individually and separately estimable
from data (Bamber & van Santen, 2000). In cognitive research, complete identifiability
has been achieved only for simple Markov-chain models (Wickens, 1982). In our work
with the array model, we have found that a 5-parameter version originally applied to the
list-length experiment is equivalent to the 4-parameter version described in the
Appendix of this article (Estes, 2002). Whether we can go on to produce a fully
identifiable array model by a reparameterization, we do not know. However, even partial
identifiability is basic to generalizability of a model and to inferences about underlying
processes (Bamber & van Santen, 2000; Li, Lewandowsky, & DeBrunner, 1996; Pitt,
Kim, & Myung, 2003).
For the array model, our relevant findings, summarized in Table 3, present a
mixed picture. Parameter recovery for individual subjects was reasonably good for
similarity parameters but much less so for storage-probability parameters. However,
patterns of relative values of estimated parameters mirrored the true patterns
reasonably well for a majority of subjects.
To interpret these results, we note that it is the estimate of the parameter vector,
e
θ , obtained from a model fit that mediates predictions of performance. Evidently, in the
running of the computer search program that seeks optimal parameter estimates,
values of different parameters trade off so that too high an estimate of one may be
compensated by too low an estimate of another, leaving almost unaltered the estimate
of θe, and therefore the model's predictions. Thus, properties of the configuration of
parameter values may be a stable characteristic of a subject even when individual
parameter values are not.
We suspect that alternative cognitive models we might have chosen for
illustrations, for example, GCM (Nosofsky, 1986) REM (Shiffrin & Steyvers, 1997); SAM
(Raaijmakers & Shiffrin, 1981); Todam (Murdock, 1982), contain nonlinearities of the
same kind that make the array model less than fully identifiable in its present form. A
practical import of our analyses is that caution is advisable in using extant cognitive
models as bases for testing hypotheses about specific underlying processes even
though even a less than fully identifiable model may be found capable of correctly
predicting true response patterns of individuals when evaluated by simulation results.
In this article, we have used the array model solely as a vehicle for illustrating a
proposed methodology. We assume, however, that further applications of the same
methodology can take us toward a full assessment. Success of the version of the model
described here (the "standard model") at predicting true performance should be
compared with that of alternative versions in which various assumptions are modified,
contributing evidence regarding necessity as well as sufficiency of the assumptions
(Estes, 2002). Among other items needing attention in this effort are possible
reparameterizations of the model that might yield full identifiability.
9
The Case for Applicability
The philosophy behind extrapolation of conclusions from our simulation results to
real research situations is the same as that pertaining to any kind of simulation testing,
from aeronautics to pharmacology. A method that cannot be tested directly in the real
situation is given a chance to fail in a simulation (in aeronautics, a wind tunnel; in
pharmacology, test of a medication on animal models of human physiology), and if the
method does fail, it is discarded or sent back to the bench for modification. A method
that passes the simulation test is judged tenable and passed on to more stringent
testing or to practical application.
No guarantees go with this approach, and its long-term viability in many domains
owes to continuing efforts to improve the quality of simulations and to deepen
understanding of relationships between simulation and situation simulated.
References
Bamber, D., & van Santen, J. P. H. (2000). How to assess a model's testability and
identifiability. Journal of Mathematical Psychology, 44, 20-40.
Busemeyer, J. R., & Wang, Y-M. (2000). Model comparisons and model selections
based on generalization criterion methodology. Journal of Mathematical Psychology, 44,
171-189.
Estes, W. K. Classification and Cognition. (1994). New York: Oxford Univ. Press.
Estes, W. K. (2002). Traps in the route to models of memory and decision.
Psychonomic Bulletin & Review, 9, 3-25.
Estes, W. K., & Maddox, W. T. (1995). Interactions of similarity, base rate, and feedback
in recognition. Journal of Experimental Psychology: Learning, Memory and Cognition,
21, 1075-1095.
Estes, W. K. & Maddox, W. T. (2002). On the processes underlying stimulus-familiarity
effects in recognition of words and non-words. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 28, 1003-1018.
Feller, W. (1957). An Introduction to Probability Theory and its Applications. New York:
Wiley.
Li, S.-C., Lewandowsky, S., & DeBrunner, V. E, (1996). Using parameter sensitivity and
interdependence to predict model scope and falsifiability. Journal of Experimental
Psychology, 125, 360-369.
Murdock, B. B. Jr. (1982). A theory for the storage and retrieval of item and
associative information. Psychological Review, 89, 609-626.
10
Nosofsky, R. M. (1986). Attention, Similarity, and the identification-categorization
relationship. Journal of Experimental Psychology: General, 115, 39-57.
Pitt, M. A., Kim, W., & Myung, I. J. (2003). Flexibility versus generalizability in model
selection. Psychonomic Bulletin & Review, 10,
29-44.
Raaijmakers, J. G. W., & Shiffrin, R. M. (1981). Search of associative memory.
Psychological Review, 88, 93-134.
Ratcliff, R., Clark, S. E., & Shiffrin, R. M. (1990). List-strength effect: I. Data and
discussion. Journal of Experimental Psychology: Learning, Memory, and
Cognition, 16, 163-178.
Shiffrin, R. M., Ratcliff, R., & Clark, S. E. (1990). List-strength effect: II.
Theoretical mechanisms. Journal of Experimental Psychology: Learning,
Memory, and Cognition, 16, 179-195 .
Shiffrin, R. M., & Steyvers, M. (1997). A model for recognition memory: REMretrieving effectively from memory. Psychonomic Bulletin & Review, 4, 145-166.
Wickens, T. D. (1982). Models for Behavior. San Francisco, Freeman.
Appendix
Summary of the Array Model
The array model assumes instance-based memory storage as in
categorization models from which it is derived (Estes, 1994). In application to the
present experiments, it is assumed that on each study trial, a representation of the item
presented is stored in a memory array with probability α(k-1), where α is a constant with
a value in the range 0 to 1 and k(=1,2,...) indexes the number of occurrences of the
item. The storage parameter is denoted in the text as a1 or a2, according as study
duration is short or long, a1 and a2 being equal to α and α2, respectively.
On each test trial of a recognition experiment, the similarity of the test item to
each element of the memory array is computed and the similarities are summed to
obtain a total similarity that we denote SimT(Old), where T indexes the type of test trial
(and is replaced by o if the test item is old, that is, came from the study list, and by n if
the test item is new. The probability that the learner judges an old test item to be "old"
is given by the expression
11
Po(Old) = Simo(Old)/[Simo(Old) + B]
and the corresponding probability for a new item by
Pn(Old) = Simn(Old)/[Simn(Old) + B],
where B is a constant whose value reflects the learner's criterion, or bias, for making a
"new" judgment about an item regardless of its status in memory4.
Letting k denote the number of repetitions of an item and N the number of items
in the study list, and setting α equal to 1 for simplicity,these expressions become
Po(Old) = [k+(N-k)s]/[k+(N-k)s+B],
and
Pn(Old) = Ns/(Ns+B).
For its use in this article, the model was modified by definition of two similarity
parameters: One of these, denoted sm, applies when the study frequency of a test item
matches that of the memory element it is being compared to, and a second parameter,
snm, applies when there is no match. The equations given above can readily be revised
to accommodate this modification.
Acknowledgements
The research reported was supported by NSF Grants SBR-996206 and BCS0130512 and NIH Grant MH 59196. Reprint requests or correspondence about this
article should be addressed to W. T. Maddox, Department of Psychology, Institute for
Neuroscience, University of Texas, 1 University Station A8000, Austin, TX, 78712-0187,
or to W. K. Estes, Department of Psychology, Indiana University, Bloomington, IN,
47405-7007. E-mail communications may be sent to [email protected] or to
[email protected].
Footnotes
1. This implication is based on the general theory of recurrent random variables (Feller,
1957).
2. This procedure was used in order to ensure that both the values of individual
parameters and the combinations of values would be representative of those
characterizing real subjects in the domain of application. The issue of
12
representativeness and our way of achieving it are discussed in a companion article,
Estes and Maddox, Coping with individual differences and error in applications of
cognitive models (submitted).
3. In the notation used in Table 2, Set A is 4wD1,4sD2, 4wnew, 8wD2, 8sD1, 8snew;
and set B is 4wD2, 4sD1, 4snew, 8wD1, 8sD2, 8wnew. This balancing was done in
order to minimize any variation in the error component of performance between sets A
and B.
34 It can be shown that B is a function of the other four parameters (Estes, 2002).
13
Table 1
Comparison of True and Predicted Performance Scores
for an Individual Subject
Score Type
Condition
True
Predicted
LL4 w D1
LL4 w D2
0.74
0.85
0.72
0.85
LL4 w new
LL8 w D1
LL8 w D2
LL8 w new
0.10
0.74
0.85
0.18
0.09
0.72
0.85
0.17
LL4 s D1
LL4 s D2
LL4 s new
0.89
0.96
0.12
0.88
0.96
0.11
LL8 s D1
LL8 s D2
LL8 s new
0.90
0.96
0.22
0.89
0.96
0.20
0.63
0.62
M
14
Table 2
Mean Parameter Estimates for Simulation Data
by Replications and Variance Levels
Replication Parameter
True Value
Estimated Value
Var. .01 Var. .02
R1
a1
a2
0.02
0.01
0.39
0.53
0.06
0.03
0.18
0.32
0.06
0.04
0.15
0.22
sm
snm
a1
a2
0.02
0.01
0.38
0.54
0.05
0.02
0.21
0.31
0.04
0.03
0.14
0.21
sm
snm
R2
Note: Var. = Variance
15
Table 3
Absolute Differences between True and Estimated
Parameter Values for 48 Subjects in R1, Variance .01 Simulation
Parameter
Statistics of Differences
Range
Median
sm
0 to .999
.019
snm
0 to .501
.003
.001 to 1.00
.278
a1
a2
.001 to .990
.442
16
Table 4
True Performance and Performance Predicted by Model
with Set A or Set B Parameter Estimates
Score Type
Condition
A True
A Predicted
by A Pars.
A Predicted
by B Pars.
R1
LL4 w D1
0.70
0.69
0.70
LL4 s D2
LL4 w new
0.91
0.17
0.91
0.18
0.90
0.17
LL8 w D2
LL8 s D1
LL8 s new
0.83
0.85
0.23
0.82
0.85
0.24
0.81
0.85
0.26
LL4 w D1
LL4 s D2
LL4 w new
0.72
0.88
0.19
0.71
0.88
0.19
0.69
0.87
0.20
LL8 w D2
LL8 s D1
LL8 s new
0.80
0.84
0.27
0.80
0.84
0.26
0.79
0.82
0.30
R2
Note. Pars. = Parameter estimates.
Conditions are coded as in Table 1.
17
Figure Captions
Figure 1. Cumulative mean disparities between observed and predicted performance
and between true and predicted performance over sequences of unit time intervals.
Figure 2. Means of predicted versus true performance by individuals in simulations of
the array model. Error variances equal to .01 and .02 are represented in the upper and
lower panels respectively. Each point represents the mean over 12 conditions of the
predicted and true scores for an individual subject.
Figure 3. Standard deviations (SDs) of individual subjects' observed-predicted
performance scores obtained from fit of the array model.
Figure 4. Correlations (Pearson rs) between true performance scores of individual
subjects and those predicted by the array model, plotted in rank order of magnitude.
18
Figures
Higher Noise
0.02
Cumulative Mean
tr - pr
0.01
ob - pr
0.00
-0.01
0
20
40
60
80
Sequence Length
100
19
Lower Noise
Cumulative Mean
0.02
tr - pr
0.01
ob - pr
0.00
-0.01
0
20
40
60
Sequence Length
80
100
20
Error Variance .01
1.0
M Predicted
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
M True
0.8
1.0
21
Error Variance .02
1.0
M Predicted
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
M True
0.8
1.0
22
Standard Deviation
0.12
0.08
0.04
0.00
0
8
16
24
32
Rank Order
40
48
23
1.00
Correlation
0.99
0.98
0.97
0.96
0.95
0
8
16
24
32
Rank Order
40
48