1. society
2. work
3. unions

What is the Value of Classical Test Theory,
Item Response Theory, and Rasch
Measurement Theory in the Development of
Patient-Reported Outcome Instruments?
MODERATOR:
Jennifer Petrillo, Novartis Pharmaceuticals Corporation
PANELISTS:
Stefan Cano, SCALEREPORT
Lori D. McLeod, RTI Health Solutions
Cheryl Coon, RTI Health Solutions
Why Are We Doing This Again?
ƒ Three commonly used psychometric methods currently
support the development and evaluation PRO instruments:
• Classical Test Theory (CTT)
• Item Response Theory (IRT)
• Rasch Measurement Theory (RMT)
ƒ Psychometric approach selection is an issue of much
debate, and confusion still remains about similarities and
differences
PRO = patient-reported outcomes.
2
ISPOR June 2012
Why Are We Doing This Again?
ƒ This study aimed to apply CTT, IRT, and RMT to evaluate
an existing instrument
ƒ Each method applied a set of parameters to evaluate and
diagnose
ƒ Comparisons were made using the 3 methods
ƒ Three psychometricians will present, defend, and explain
the benefits of one method each.
3
ISPOR June 2012
Visual Functioning Questionnaire (VFQ-25)
ƒ The VFQ-25 evaluates the influence of visual disability and
visual symptoms on task-oriented domains and general health
domains (e.g., emotional well-being)
ƒ Consists of 25 items, plus a single-item general health rating,
and forms 12 subscales
ƒ Response scales vary between items (4 to 6 Likert-type
response categories)
ƒ A composite total score is the weighted average of the 12
subscales and ranges from 0 to 100
• The 25-item composite score is multidimensional, but this exercise assumes a
unidimensional scale for demonstration purposes, so the results here should
not be considered a reflection on the validated 25-item composite score
Mangione CM, Lee PP, Gutierrez PR, Spritzer K, Berry S, Hays RD. Development of the 25-item National Eye Institute Visual Function
Questionnaire (VFQ-25) Archives of Ophthalmology. 2001;119:1050-8.
Mangione CM, Lee PP, Pitts J, Gutierrez P, Berry S, Hays RD. Psychometric properties of the National Eye Institute Visual Function
Questionnaire, the NEI-VFQ. Archives of Ophthalmology. 1998;116:1496-504.
4
ISPOR June 2012
Visual Functioning Questionnaire (VFQ-25)
ƒ Data available from RESTORE, a randomized, double-
masked, multicenter, laser-controlled phase 3 study of an
injectable in patients with visual impairment due to diabetic
macular edema
• Inclusion: BCVA of 78 to 39 letters (Snellen-equivalent 20/32 to
20/160)
• Mean BCVA at Baseline = 63 letters
ƒ Items were reverse-scored as necessary so that high
scores represent better functioning
ƒ Baseline data were selected for item evaluation (N = 240)
BCVA = best-corrected visual acuity.
5
ISPOR June 2012
Visual Functioning Questionnaire (VFQ-25)
6
ISPOR June 2012
Visual Functioning Questionnaire (VFQ-25)
7
ISPOR June 2012
Visual Functioning Questionnaire (VFQ-25)
8
ISPOR June 2012
Visual Functioning Questionnaire (VFQ-25)
9
ISPOR June 2012
Psychometric Analysis of the VFQ-25:
Classical Test Theory (CTT) Methods
Lori McLeod, PhD
Head, Psychometrics
RTI Health Solutions
CTT
ƒ Descriptives
• Data Quality and Scaling Evaluation
- Amount of missing per item and proposed scales
- Use of response scale through mean, SD, skewness, and category
response frequency (floor and ceiling effects)
• Scaling Assumptions
- Comparisons of means and variances
- Inter-item correlations
- Comparisons of item-to-total correlation
• Graphical evaluation of item responses versus total scores
11
ISPOR June 2012
CTT
ƒ Reliability
• Internal consistency (estimated using Cronbach’s alpha)
• Test-retest
ƒ Validity
• Construct (including correlations with other validated measures)
• Discriminant (known-groups)
• Sensitivity to change (responsiveness)
12
ISPOR June 2012
Mean
range
2.6-4.7
Negative
skewness
Blindness
(excluded)
CTT: VFQ Item Descriptives (Reverse Scored)
Item
N
N Miss
Mean
Std Dev
Skewness
Min
Max
239
1
4.02
0.72
2
6
3 – worry (1‐5)
240
0
2.58
1.15
1
5
4 – amount pain (1-5)
240
0
4.33
0.83
1
5
5 – reading normal newsprint
(1‐5)
6 – seeing well up close (1‐5)
238
2
3.13
1.25
1
5
240
0
3.49
1.14
7 – finding objects on crowded shelf (1‐5)
239
1
4.10
0.98
8 – reading street signs
(1‐5)
238
2
3.96
0.98
9 – going down stairs at night
238
2
3.84
1.06
239
1
4.19
0.99
240
0
4.39
0.88
234
6
4.68
0.71
238
2
4.54
0.86
208
32
4.17
1.09
158
82
4.09
1.56
157
83
3.18
1.47
158
82
3.44
1.49
239
1
3.53
1.27
238
2
3.89
1.19
239
1
4.44
0.93
238
2
4.30
1.18
237
3
3.72
1.32
238
2
3.83
1.40
238
2
4.29
1.18
238
2
4.24
1.23
238
2
4.38
1.09
‐0.24
0.28
‐1.03
‐0.19
‐0.31
‐0.87
‐0.62
‐0.51
‐1.07
‐1.46
‐2.23
‐2.10
‐1.29
‐1.38
‐0.36
‐0.58
‐0.41
‐0.77
‐1.78
‐1.58
‐0.64
‐0.79
‐1.55
‐1.45
‐1.63
2 – general vision
(1‐6)
(1‐5)
10 – seeing objects off to side (1‐5)
Most
11 – seeing how people react (1‐5)
missing
12 – difficulty matching clothes (1‐5)
13 – visiting others (1‐5)
14 – going out to movies/plays (1‐5)
15c – daylight driving (1‐4)
16 – nighttime driving (1‐5)
16a – difficult conditions driving (1‐5)
17 – accomplish less (1‐5)
18 – limited in endurance (1‐5)
19 – amount time pain (1‐5)
20 – stay home most of time (1‐5)
21 – frustrated (1‐5)
22 – no control (1‐5)
23 – rely too much on others’ word (1‐5)
24 – need much help from others (1‐5)
25 – embarrassment (1‐5)
13
ISPOR June 2012
1
Missing
“no 5
1 at all” 5
difficulty
1
5
1
5
1
5
1
5
2
5
1
5
1
5
1
5
1
5
1
5
1
5
1
5
1
5
1
5
1
5
1
5
1
5
1
5
1
5
CTT: VFQ Item Response Frequencies
• Response
frequencies
appear
grouped by
content
The less severe
categories were
chosen more often
for social issues
14
ISPOR June 2012
CTT: VFQ Item Responses Ordered
Potential ceiling
effects
15
ISPOR June 2012
CTT: Inter-item Polychoric Correlations
2 – general vision
3 – worry
2
1.00 .
3
4
5
6
7
8
9
10
11
12
13
14
15c
16
16a
17
18
19
20
21
22
23
24
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0.38
1.00 .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Item 4 (pain) is the least related to
.
.
.
.
.
.
.
.
.
the other items with the exception of
.
.
.
.
.
.
.
.
.
item 19 (amount of time with pain)
0.23
0.29
1.00 .
5 – reading normal
newsprint
0.63
0.37
0.24
1.00 .
6 – seeing well up
close
0.65
0.42
0.29
0.72
1.00 .
.
.
.
.
.
.
.
.
.
7 – finding objects
on crowded shelf
0.55
0.37
0.38
0.59
0.65
1.00 .
.
.
.
.
.
.
.
.
8 – reading street
signs
0.48
0.40
0.24
0.53
0.57
0.64
1.00 .
.
.
.
.
.
.
.
9 – going down
stairs at night
0.51
0.44
0.24
0.45
0.58
0.60
0.61
1.00 .
.
.
.
.
.
.
10 – seeing objects
off to side
0.53
0.51
0.36
0.53
0.60
0.69
0.63
0.67
1.00 .
.
.
.
.
.
11 – seeing how
people react
0.49
0.43
0.27
0.54
0.52
0.70
0.59
0.58
0.74
1.00 .
.
.
.
.
12 – difficulty
matching clothes
0.52
0.44
0.28
0.52
0.63
0.64
0.63
0.55
0.63
0.75
1.00 .
0.50
0.42
0.28
0.49
0.57
0.64
0.62
0.61
0.68
0.78
0.75
14 – going out to
movies/plays
0.55
0.48
0.35
0.50
0.60
0.68
0.67
0.72
0.74
0.71
0.68
0.78
1.00 .
.
.
.
.
.
.
.
15c – daylight
driving
0.53
0.30
0.04
0.51
0.53
0.47
0.46
0.46
0.49
0.48
0.58
0.58
0.63
1.00 .
.
.
.
.
.
.
16 – nighttime
driving
0.48
0.26
0.12
0.33
0.46
0.43
0.49
0.57
0.52
0.45
0.49
0.50
0.60
0.88
1.00 .
.
.
.
.
.
16a – difficult
conditions driving
0.40
0.29
0.05
0.32
0.46
0.45
0.47
0.49
0.52
0.43
0.45
0.57
0.62
0.90
0.88
1.00 .
.
.
.
.
.
.
.
17 – accomplish
less
0.48
0.29
0.20
0.52
0.56
0.56
0.47
0.43
0.51
0.49
0.56
0.48
0.55
0.45
0.35
0.41
1.00 .
.
.
.
.
.
.
18 – limited in
endurance
0.46
0.31
0.30
0.48
0.55
0.67
0.53
0.51
0.65
0.62
0.56
0.63
0.68
0.53
0.44
0.53
0.74
1.00 .
.
.
.
.
.
19 – amount time
pain
0.36
0.31
0.64
0.30
0.39
0.53
0.36
0.37
0.51
0.44
0.36
0.48
0.50
0.06
0.09
0.26
0.48
0.60
1.00 .
.
.
.
.
20 – stay home
most of time
0.61
0.42
0.31
0.51
0.62
0.62
0.64
0.60
0.66
0.62
0.64
0.67
0.71
0.71
0.68
0.71
0.61
0.68
0.49
1.00 .
.
.
.
0.56
0.49
0.20
0.45
0.51
0.50
0.51
0.49
0.50
0.46
0.52
0.53
0.58
0.50
0.43
0.45
0.51
0.53
0.34
0.71
1.00 .
.
.
0.56
0.40
0.26
0.55
0.56
0.62
0.51
0.49
0.62
0.58
0.63
0.58
0.60
0.53
0.44
0.51
0.67
0.67
0.46
0.70
0.68
1.00 .
0.43
0.40
0.22
0.46
0.51
0.56
0.50
0.52
0.60
0.49
0.53
0.55
0.63
0.43
0.33
0.38
0.63
0.61
0.45
0.66
0.58
0.76
0.45
0.37
0.24
0.49
0.53
0.58
0.48
0.48
0.59
0.49
0.52
0.51
0.62
0.54
0.42
0.48
0.56
0.59
0.48
0.67
0.59
0.71
0.88
1.00
0.56
0.49
0.22
0.39
0.51
0.50
0.46
0.47
0.61
0.57
0.52
0.49
0.50
0.43
0.41
0.48
0.52
0.54
0.48
0.64
0.57
0.68
0.73
0.78
4 – amount pain
13 – visiting others
21 – frustrated
22 – no control
23 – rely too much
on others’ word
24 – need much
help from others
25 –
embarrassment
16
ISPOR June 2012
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
A potential
.triplet
.
.
.about
.
.
.
.
.
driving
.
.
.
(items15c,
.
.
.
16,
16a)
.
.
.
.
.
.
.
.
.
1.00 .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Item
23. (rely
on
.
.
.
others)
is highly
.
.
.
related to item
.
.
24. (need
help)
.
1.00 .
CTT: VFQ Item Correlations (Inter-item and Itemtotal)
Item
Inter-item
2 – general vision
Item-total
3 – worry (1‐5)
.45
4 – amount pain (1-5)
(18 value less than .3)
.25
5 – reading normal newsprint
(1‐6)
6 – seeing well up close (1‐6)
.56
7 – finding objects on crowded shelf (1‐6)
.69
8 – reading street signs
(1‐6)
.61
9 – going down stairs at night
.62
.67
(1‐6)
10 – seeing objects off to side (1‐6)
11 – seeing how people react (1‐6)
12 – difficulty matching clothes (1‐6)
13 – visiting others (1‐6)
14 – going out to movies/plays (1‐6)
15c – daylight driving (1‐4)
16 – nighttime driving (1‐6)
16a – difficult conditions driving (1‐6)
17 – accomplish less (1‐5)
18 – limited in endurance (1‐5)
19 – amount time pain (1‐5)
20 – stay home most of time (1‐5)
21 – frustrated (1‐5)
22 – no control (1‐5)
23 – rely too much on others’ word (1‐5)
24 – need much help from others (1‐5)
25 – embarrassment (1‐5)
17
Item 23
The 2 pain
(rely on
items are
others) is
the least
highly
related to
related to
the total
item 24
score
(need help)
.58
(1‐6)
.71
.62
.65
.55
.75
DRIVING
.57
DRIVING
.61
DRIVING
.63
Cronbach’s alpha = 0.94
.65
.71
.34
.72
.68
.72
Others help
.63
Others help
.64
.61
ISPOR June 2012
CTT: Summary
Item
Missing
(>5%)
Floor/Ceiling
(40%/5%)
Skewness
(>|2|)
Inter-item
(>0.8)
Item-total
(<0.3)
Concerns
2 – general vision
3 – worry
4 – amount pain
Remove pain from the scale and measure separately
5 – reading normal newsprint
6 – seeing well up close
7 – finding objects on crowded shelf
8 – reading street signs
9 – going down stairs at night
10 – seeing objects off to side
11 – seeing how people react
13 – visiting others
Consider removing matching clothes due to missingness
and skewness
Consider scoring social and dependence items separately
14 – going out to movies/plays
Consider scoring social and dependence items separately
15c – daylight driving
16 – nighttime driving
16a – difficult conditions driving
17 – accomplish less
18 – limited in endurance
Create a triplet for driving items
Create a triplet for driving items
Create a triplet for driving items
19 – amount time pain
Remove pain from the scale and measure separately
12 – difficulty matching clothes
20 – stay home most of time
21 – frustrated
22 – no control
23 – rely too much on others’ word
Consider scoring social and dependence items separately
24 – need much help from others
Consider scoring social and dependence items separately
25 – embarrassment
18
ISPOR June 2012
Psychometric Analysis of the VFQ-25:
Rasch Measurement Theory (RMT) Methods
Stefan J. Cano PhD CPsychol AFBPsS
Director, ScaleReport Ltd
Associate Professor of Psychometrics, PCMD
RMT: Core Issues for Clinical Hypotheses and
Synthesis
ƒ The Rasch model articulates the set of requirements that must be met
for rating scale data to generate internally valid measurements that are
stable (invariant) across items and people.
• RMT… “is an experimental measurement
paradigm [in which] the criterion of the invariance
of comparisons that rating scales should meet is
specified a priori in terms of a response model.
This model is a Rasch model. A specific critical
issue in rating scales is the ordering of the
categories…in RMT, it is treated as a hypothesis
with the implication that if the ordering is not
working as intended, then the response
categories need to be studied and improved
experimentally.” (p.583)
Andrich D. Rating scales and Rasch measurement. Expert Rev Pharmacoeconomics Outcomes Res
2011;11:571-585.
20
ISPOR June 2012
RMT: Metric
General
Vision
(1)
Ocular Pain
(2)
Near
Activities
(3)
Distance
Activities
(3)
Vision-specific:
Social
Functioning
(2)
Vision-specific:
Role
Difficulties
(2)
Vision-specific:
Dependency
(3)
Vision-specific:
Mental Health
(4)
Peripheral
Vision
(1)
Driving
(3)
Color Vision
(1)
VFQ-25 frame of reference defined
(worse)
21
“Visual Functioning” TOTAL SCORE /100
(better)
ISPOR June 2012
RMT: Diagnostic Paradigm
Targeting and mapped
continuum
Fit
22
ISPOR June 2012
Response categories
Potential to measure change
RMT: Targeting
VFQ-25 frame of reference defined
(worse)
23
“Visual Functioning” TOTAL SCORE /100
(better)
ISPOR June 2012
RMT: Targeting
VFQ-25 frame of reference defined
(worse)
24
ISPOR June 2012
“Visual Functioning” TOTAL SCORE /100
(better)
RMT: Response Categories
(worse)
25
VFQ-25 frame of reference defined
“Visual Functioning” TOTAL SCORE /100
(better)
ISPOR June 2012
RMT: 13/25 Items Ordered Thresholds
VFQ-25 frame of reference defined
(worse)
26
ISPOR June 2012
“Visual Functioning” TOTAL SCORE /100
(better)
RMT: 13/25 Items Ordered Thresholds
VFQ-25 frame of reference defined
(worse)
27
“Visual Functioning” TOTAL SCORE /100
(better)
ISPOR June 2012
RMT: 12/25 Items Disordered Thresholds
VFQ-25 frame of reference defined
(worse)
28
ISPOR June 2012
“Visual Functioning” TOTAL SCORE /100
(better)
RMT: Also Important To Note…
29
ISPOR June 2012
RMT: Item Fit
30
ISPOR June 2012
RMT: 2 Items Misfit
31
ISPOR June 2012
RMT: 2 Items Misfit
32
ISPOR June 2012
RMT: 2 Items Misfit
(worse)
33
VFQ-25 frame of reference defined
“Visual Functioning” TOTAL SCORE /100
(better)
ISPOR June 2012
RMT: 2 Items Misfit
(worse)
34
ISPOR June 2012
VFQ-25 frame of reference defined
“Visual Functioning” TOTAL SCORE /100
(better)
RMT: Better Fitting Example
(worse)
35
VFQ-25 frame of reference defined
“Visual Functioning” TOTAL SCORE /100
ISPOR June 2012
RMT: Low Person Misfit
36
ISPOR June 2012
(better)
RMT: Example of Person Misfit
(worse)
37
VFQ-25 frame of reference defined
“Visual Functioning” TOTAL SCORE /100
(better)
ISPOR June 2012
RMT: Example of Better Person Fit
(worse)
38
ISPOR June 2012
VFQ-25 frame of reference defined
“Visual Functioning” TOTAL SCORE /100
(better)
RMT: Potential to Measure Change
39
ISPOR June 2012
RMT: Up to 15-Fold Difference
90-100 = 5.9 logits
45-55 = 0.4 logits
0-10 = 5.6 logits
(worse)
40
ISPOR June 2012
VFQ-25 frame of reference defined
“Visual Functioning” TOTAL SCORE /100
(better)
RMT: Overall
ƒ Mistargeting
ƒ Issues with response options
ƒ Overall fit generally okay
ƒ Raw score changes differential meaning
41
ISPOR June 2012
RMT: Other Tests?
ƒ Reliability (>0.92)
ƒ Dependency (3 pairs of items)
ƒ Differential item functioning
ƒ Tests of unidimensionality
42
ISPOR June 2012
RMT: Recommendation 1
ƒ Examine empirically some test design issues…
43
ISPOR June 2012
RMT: Items 20-25 Disordered Thresholds
44
ISPOR June 2012
RMT: Items 20-25 Disordered Thresholds
(worse)
45
VFQ-25 frame of reference defined
“Visual Functioning” TOTAL SCORE /100
(better)
ISPOR June 2012
RMT: Items 20-25 Disordered Thresholds
46
ISPOR June 2012
RMT: Items 20-25 Disordered Thresholds
47
ISPOR June 2012
RMT: Items 20-25 Rescored and Revisited
48
ISPOR June 2012
RMT: Recommendation 2
ƒ Revisit VFQ-25
ƒ Substantive Theorists
ƒ Hypothesis generation and testing
49
ISPOR June 2012
RMT: Would Not Recommend
ƒ Discourage resting solely on post-hoc analyses such as the
following:
• Rescoring
• Item splitting
• Subtesting
50
ISPOR June 2012
RMT: Summary
Item
Response categories
Item misfit
Targeting
Change potential
Recommendation
(sub-optimal)
2 – general vision
3 – worry
4 – amount pain
5 – reading normal newsprint
6 – seeing well up close
7 – finding objects on crowded shelf
8 – reading street signs
9 – going down stairs at night
10 – seeing objects off to side
11 – seeing how people react
12 – difficulty matching clothes
13 – visiting others
14 – going out to movies/plays
15c – daylight driving
16 – nighttime driving
16a – difficult conditions driving
17 – accomplish less
18 – limited in endurance
19 – amount time pain
20 – stay home most of time
21 – frustrated
22 – no control
23 – rely too much on others’ word
24 – need much help from others
25 – embarrassment
51
ISPOR June 2012
RMT
“…the use of rating scales for the assessment of
health outcomes will increase and therefore
become more important…there will be an
increasing recognition of the potential
advantages of [RMT]. With this recognition will
come a stronger relationship between
psychometricians and clinicians in the
construction and verification of rating scales,
with the consequent better understanding of
psychometrics by clinicians, and of clinical
issues by psychometricians.” (p. 583)
Andrich D. Rating scales and Rasch measurement.
Expert Rev Pharmacoeconomics Outcomes Res 2011;11:571585.
52
ISPOR June 2012
Recommendation
(in line with RMT)
RMT
53
ISPOR June 2012
Psychometric Analysis of the VFQ-25:
Item Response Theory (IRT) Methods
Cheryl Coon, PhD
Director, Psychometrics
RTI Health Solutions
IRT: Graded Response Model
ƒ Slope (ai) parameters: vary across items but are equal across response
categories within items
ƒ Threshold (bi) parameters: vary across items and response categories and
are ordered
ƒ ICCs: show the probability of an item response across the range of the scale
and reveal weak items or overlapping response categories
ƒ Item and test information functions: reflect how well the individual items and
the test as a whole estimate the construct over the entire scale range (i.e.,
reliability)
ƒ Item fit: S-X2 is a statistic that examines the observed and expected response
proportions for each test score value
ƒ Local dependence: LD χ2 is a statistic examines bivariate fit to identify
evidence of items that are excessively related given the common underlying
construct
ICC = item characteristic curves.
55
ISPOR June 2012
IRT: VFQ-25 Items 2-10 ICCs and Information
Group 1, VFQ_2rev
Group 1, VFQ_3rev
2.0
3
0.8
Probability
1.0
0.4
0
0.6
0.5
0.4
-1
0
1
2
-2
-1
Group 1, VFQ_5rev
0
0.8
Probability
0.9
1
2
0.6
1
0.5
2
3
1.0
0.4
-1
0
1
2
0.5
0.1
-1
0
1
Theta
ISPOR June 2012
2
3
Probability
3
4
2.0
0.5
2
3
1.0
0.4
0.5
0.0
-3
1.5
0.6
0.5
3
2
1.0
0.4
0.3
0
0.2
1
0.7
1.5
1
0.1
0.0
-2
2
0.8
0.6
0.3
0
1
0
0.5
0.2
0.1
0.0
-2
-1
0
Theta
1
2
3
0.0
-3
0.0
-2
-1
0
Theta
1
2
3
Inform
ation
1.0
0
0.9
Inform
ation
3
0.4
0.0
-3
-1
1.0
2.0
0.7
Inform
ation
2
0.0
-2
Group 1, VFQ_10rev
4
0.9
1.5
1
0.5
0.5
0.0
-3
3
0.8
0.6
1.0
0
Theta
1.0
2.0
0.7
3
0.4
Group 1, VFQ_9rev
4
2.0
0.1
Group 1, VFQ_8rev
0.8
4
1.5
2
0.5
Theta
0.9
3
0.2
0.0
-2
Theta
1.0
2
1
0.6
0.3
0.5
0.0
-3
3
1
Poor information
0.7
1.5
0.1
0.0
0
0
0.8
0
Probability
Probability
2.0
0.2
0.1
-1
-1
Group 1, VFQ_7rev
1.0
4
0.3
0.5
0.2
-2
0.0
-2
Inform
ation
1.0
0.3
0.0
-3
0.5
0
Theta
Inform
ation
3
Inform
ation
0.6
2
3
0.7
1.5
1
2
0.0
-3
Group 1, VFQ_6rev
0.7
Probability
2
0.9
0.8
56
1
1.0
2.0
4
0.4
1.0
1
Theta
0
0.5
0.4
0.1
0.0
0.0
-3
3
0.5
0.2
Probability
-2
1.5
0.6
0.3
0.1
0.0
1.0
0.2
1.0
0.5
Theta
0.3
4
0.2
4
0.1
0.9
3
2
1
0.3
0.5
0.2
0.0
-3
0.7
Inform
ation
0.5
2.0
3
0.8
1.5
Inform
ation
0.6
0.9
0
0.7
1.5
Inform
ation
Probability
2
1
0.7
1.0
2.0
0.9
0.8
0.3
Group 1, VFQ_4rev
1.0
0.9
Probability
1.0
IRT: VFQ-25 Items 11-18 ICCs and Information
Group 1, VFQ_11rev
Group 1, VFQ_12rev
4
1.0
0.3
Probability
0.6
0
0.5
1
1.0
0.4
2
0.3
0
0.5
0.2
0.1
-2
-1
0
1
2
0.0
-2
-1
0
Theta
Group 1, VFQ_14rev
1.0
2.0
1.0
2.0
0.6
0.5
1.0
0.4
1
2
0.0
-3
3
0.8
3
0.5
0.8
0.6
2
3
1
2
2.0
1.5
0.6
0.5
1
2
0.4
1.0
3
0.5
0.2
0.1
0.0
-3
3
4
0
0.3
0.5
0.1
0.0
0.0
-2
-1
Theta
57
1.0
1
3
0.7
1.5
0.4
2
0.9
0
0.5
1
Group 1, VFQ_18rev
2.0
0.2
1
0
0
1.0
4
0.3
2
0.2
-1
0.0
-1
Inform
ation
1.0
0.5
1
-2
Theta
Inform
ation
0.5
Inform
ation
0.6
-2
2
0.0
-3
3
0.7
1.5
Probability
0.7
0.0
-3
2
0.9
0.8
0.1
1
1.0
3
0.4
Group 1, VFQ_17rev
2.0
0.3
0.5
0.1
0
1.0
4
0.4
1.5
0.6
0.2
0.0
-1
Group 1, VFQ_16arev
0
2.0
4
Theta
0.9
3
0.7
0.5
2
1
-2
Theta
1.0
0
0.3
3
0.1
0.0
0
2
0.8
1.5
0.2
0.1
1
0.9
0.3
0.5
0.2
-1
0
Group 1, VFQ_16rev
4
0
Probability
Probability
1.0
2
-2
0.0
-1
Inform
ation
3
0.0
-3
0.5
-2
Theta
Inform
ation
0.6
1
1
0.0
-3
3
0.7
Inform
ation
Probability
2
0.8
1.5
0.3
Probability
1
0.9
0
0.7
0.4
1.0
3
Group 1, VFQ_15crev
4
0.9
0.5
2
Theta
1.0
0.8
1.5
0
0.4
0.1
0.0
-3
3
0.5
0.2
0.1
0.0
0.0
-3
0.6
0.3
0.5
0.2
Probability
Probability
3
2
0.7
1.5
Inform
ation
1
0.4
Inform
ation
0.6
0.8
0.7
1.5
2.0
0.9
0.8
0.7
4
1.0
2.0
0.9
0.8
0.5
Group 1, VFQ_13rev
3
1.0
2.0
0.9
Probability
1.0
Inform
ation
Overlapping response
categories
0
1
2
0.0
0.0
-3
3
-2
-1
0
Theta
1
2
3
Theta
ISPOR June 2012
IRT: VFQ-25 Items 19-25 ICCs and Information
Group 1, VFQ_19rev
Group 1, VFQ_20rev
4
1.0
0.9
Probability
0.4
0.5
-2
-1
0
1
2
0.0
-3
3
-2
Group 1, VFQ_22rev
-1
1
2
3
0.7
1.5
3
0.3
0.6
0.5
0.4
1.0
1
3
-2
-1
0
1
2
0.0
-3
3
Theta
4
0.7
2.0
0
1.5
0.6
0.5
1
2
3
0.5
0.2
0.1
0.0
-2
-1
1.0
0.4
0.3
0.5
0.1
0.0
0.0
-3
3
0.9
1.5
0
0.2
0.1
2
0.8
0.3
0.5
2
0.2
1
0
Theta
1
2
3
0.0
-3
2
0.0
-2
-1
0
1
2
3
Theta
Group 1, VFQ_25rev
4
1.0
2.0
0.9
0.8
1.5
0.6
1
0.5
0.4
1.0
0
0.3
3
0.2
0.5
2
0.1
0.0
-3
0.0
-2
-1
0
1
Theta
58
ISPOR June 2012
2
3
Inform
ation
Probability
0.7
Overlapping response
categories
Inform
ation
1.0
0
1.0
2.0
Inform
ation
1
Inform
ation
0.6
Probability
0.7
0.4
-1
Group 1, VFQ_24rev
4
0.9
0.8
0.5
0.0
-2
Theta
1.0
2.0
0.8
Probability
0
0.0
-3
Group 1, VFQ_23rev
4
0
0.5
2
Theta
1.0
1.0
3
0.4
0.1
0.0
Theta
0.9
1
0.5
0.2
2
0.1
0.0
1.5
0.6
0.3
0.5
0.2
0.1
0.0
-3
1.0
3
Probability
Probability
1
0.5
0.3
3
0
1
0.6
0
Inform
ation
0.2
1.0
2
0.4
0.7
1.5
Inform
ation
0.5
Inform
ation
0.6
0.8
0.7
1.5
2.0
0.9
0.8
0.7
4
1.0
2.0
0
Probability
0.8
0.3
Group 1, VFQ_21rev
4
1.0
2.0
0.9
IRT: VFQ-25 Item Parameters and Item Fit
Item
a (SE)
2 – general vision
3 – worry
4 – amount pain
5 – reading normal newsprint
6 – seeing well up close
7 – finding objects on crowded shelf
8 – reading street signs
9 – going down stairs at night
10 – seeing objects off to side
11 – seeing how people react
12 – difficulty matching clothes
13 – visiting others
14 – going out to movies/plays
15c – daylight driving
16 – nighttime driving
16a – difficult conditions driving
17 – accomplish less
18 – limited in endurance
19 – amount time pain
20 – stay home most of time
21 – frustrated
22 – no control
23 – rely too much on others’ word
24 – need much help from others
59
ISPOR June 2012
25 – embarrassment
b1 (SE)
b2 (SE)
b3 (SE)
1.63 (0.20) -3.28 (0.42) -1.15 (0.15) 1.01 (0.15)
1.07 (0.15) -1.53 (0.23) -0.05 (0.14) 1.48 (0.24)
0.61 (0.14) -6.61 (1.60) -2.63 (0.61) -0.20 (0.23)
1.54 (0.18) -1.74 (0.20) -0.80 (0.13) 0.30 (0.12)
2.02 (0.22) -2.30 (0.23) -1.14 (0.12) -0.04 (0.10)
2.32 (0.26) -3.17 (0.39) -1.80 (0.17) -0.79 (0.10)
Response categories for item 4 are
1.85 (0.21) -3.46 (0.47) -1.88 (0.19) -0.64 (0.11)
shifted
the left
1.76 (0.21)
-3.34severely
(0.43) -1.65to(0.18)
-0.51 (0.11)
2.44 (0.28) -3.14 (0.39) -1.74 (0.16) -0.90 (0.10)
2.08 (0.25) -3.28 (0.43) -2.28 (0.23) -1.31 (0.14)
2.11 (0.31) -2.69 (0.31) -1.63 (0.17) -1.05 (0.12)
2.34 (0.30) -2.75 (0.30) -2.32 (0.23) -1.38 (0.13)
2.56 (0.31) -2.27 (0.23) -1.56 (0.15) -0.87 (0.11)
1.75 (0.28) -1.06 (0.17) -1.03 (0.18) -0.96 (0.16)
1.46 (0.23) -0.94 (0.19) -0.57 (0.16) 0.13 (0.14)
Many
1.57 (0.24) -1.04 (0.19) -0.75 (0.16) -0.13 (0.13)
1.74 (0.20) -2.04 (0.22) -1.21 (0.14) -0.03 (0.10)
2.14 (0.24) -2.31 (0.23) -1.35 (0.14) -0.49 (0.10)
1.14 (0.18) -3.90 (0.63) -3.22 (0.48) -1.73 (0.26)
2.66 (0.33) -2.13 (0.20) -1.27 (0.12) -1.07 (0.11)
1.69 (0.20) -2.27 (0.25) -0.92 (0.13) -0.53 (0.11)
2.22 (0.26) -1.84 (0.18) -0.89 (0.11) -0.56 (0.09)
1.93 (0.25) -2.45 (0.27) -1.51 (0.16) -1.19 (0.13)
1.95 (0.25) -2.38 (0.26) -1.32 (0.14) -1.17 (0.13)
1.72 (0.24) -3.00 (0.37) -1.70 (0.19) -1.30 (0.15)
b4 (SE)
Avg. b
4.04 (0.71)
3.02 (0.45)
1.54 (0.20)
1.00 (0.13)
0.22 (0.10)
0.53 (0.12)
0.59 (0.12)
0.02 (0.09)
-0.29 (0.09)
-0.67 (0.10)
0.03 (0.09)
-0.47 (0.12)
1.36 (0.21)
misfitting
0.86 (0.16)
0.70 (0.13)
0.29 (0.10)
-0.76 (0.16)
-0.47 (0.09)
0.43 (0.12)
0.05 (0.09)
-0.51 (0.10)
-0.47 (0.10)
-0.71 (0.11)
S-X2 PValue
0.16 44.77
0.73 61.70
-3.15 38.24
-0.18 60.14
-0.62 47.74
-1.39 44.19
-1.36 48.34
-1.23 55.89
-1.44 39.41
-1.79 29.32
-1.79 26.79
-1.78 27.25
-1.17 34.92
-0.88 31.21
0.00 54.16
items
-0.27 39.27
-0.65 54.81
-0.97 46.90
-2.40 43.47
-1.24 24.39
-0.82 61.22
-0.81 41.41
-1.42 34.90
-1.34 31.54
-1.68 32.21
0.006
0.105
0.367
0.094
0.059
0.007
0.066
0.014
0.009
0.022
0.003
0.039
0.029
0.027
0.164
0.504
0.018
0.043
0.012
0.041
0.005
0.021
0.010
0.035
0.041
19
23
IRT: VFQ-25 Local Dependence
Item
X2
2
3
4
5
6
7
8
9
10
11
12
13
14 15c 16 16a 17
18
20
21
22
2 – general vision
0.1
3 – worry
0.0 -0.2
4 – amount pain
0.0 1.3 -0.2
5 – reading normal newsprint
0.1 2.4 -0.4 0.2
6 – seeing well up close
0.1 1.0 0.9 0.0 5.5
7 – finding objects on crowded shelf
0.3 2.8 0.0 0.5 2.4 0.4
8 – reading street signs
0.2 -0.6 0.8 -1.3 0.3 -0.4 -0.1
9 – going down stairs at night
0.1 1.3 0.2 -0.5 -0.8 -0.9 0.0 0.3
10 – seeing objects off to side
0.5 1.2 1.9 2.1 -0.1 -0.7 1.0 2.0 1.1
11 – seeing how people react
0.1 0.1 -0.2 -0.5 0.1 3.8 1.2 0.7 0.9 1.4
12 – difficulty matching clothes
0.2 0.5 -0.8 -0.9 -0.7 0.4 -0.5 -0.3 0.7 0.2 2.6
13 – visiting others
0.3 -0.3 1.5 -0.8 2.7 0.1 -1.1 1.5 0.5 1.0 2.9 0.2
14 – going out to movies/plays
1.3 -0.2 0.6 -0.7 0.6 1.3 0.3 2.0 -0.4 -0.4 0.1 0.9 3.6
15c – daylight driving
1.4 0.4 -0.2 0.9 1.2 0.9 0.1 0.1 1.1 2.7 0.9 1.6 2.0 2.7
16 – nighttime driving
4.7 2.4 0.9 1.0 2.9 3.2 0.9 1.7 2.3 1.3 1.4 0.8 2.7 3.2 8.1
16a – difficult conditions driving
5.8 3.9 1.3 1.1 3.0 1.8 1.7 2.7 1.3 1.3 2.2 1.8 2.4 4.3 9.7 7.4
17 – accomplish less
0.3 -0.9 1.6 -0.4 0.1 0.5 0.5 3.3 1.4 0.1 3.6 0.1 2.1 0.1 2.2 2.0 2.6
18 – limited in endurance
0.2 1.3 2.2 -0.5 -0.1 -0.3 -0.2 2.2 0.3 -0.4 -0.8 0.5 -0.2 -0.4 1.7 1.0 1.3 4.5
19 – amount time pain
0.0 -0.3 0.7 6.6 3.8 0.4 -0.3 -0.1 0.3 1.0 0.4 0.5 0.0 1.3 2.5 1.2 1.9 1.9 1.6
20 – stay home most of time
0.2 -0.9 0.6 -0.6 0.5 -0.5 -0.4 -1.0 -0.5 -0.7 1.8 -0.5 0.0 -1.1 2.4 2.8 4.3 1.1 -0.2 -0.2
21 – frustrated
0.1 -0.5 1.1 0.9 -0.8 0.3 -0.1 2.2 1.0 0.5 1.9 -0.8 -0.3 0.6 0.0 1.4 2.4 -0.1 -0.6 0.4 0.0
22 – no control
0.3 -1.0 -0.2 0.9 1.2 -0.5 -0.1 -0.6 1.8 1.6 0.3 1.4 1.9 0.2 1.8 0.8 0.8 2.6 0.1 -0.9 1.9 3.4
23 – rely too much on others’ word
0.6 1.0 -0.9 2.8 0.6 0.9 0.3 1.0 -0.3 1.3 -0.1 2.0 1.5 0.4 2.3 1.0 0.9 2.5 -0.6 1.4 0.3 0.3 2.4
24 – need much help from others
60
ISPOR June 2012
25 – embarrassment
0.4 -0.3 1.5 0.2 -0.6 0.1 -0.1 -0.6 0.6 1.4 0.4 0.4 1.0 0.0 0.3 0.8 0.8 1.9 1.1 0.9 0.6 1.9 0.9 7.0
24
Local dependence between items
with similar content:
•4 and 19 – pain
•5 and 6 – seeing close up
•15c, 16, and 16a – driving
•23 and 24 – relying on others
0.1 2.9 -1.0 0.7 2.6 -0.5 0.2 -0.5 -0.5 -0.4 -0.1 0.2 -0.6 0.5 0.5 1.0 1.4 2.3 -1.7 0.5 1.9 -0.5 -0.2 2.2 3.3
IRT: VFQ-25 Information and Reliability
Group 1, Total Information Curve
28
0.8
1 provide good
The items together
Reliabilit
y ≈ 1across
− a wide
information
range of the
θ ) only at the top of
scale; information Iis(poor
26
0.7
24
22
0.6
the scale corresponding to good visual
functioning
20
0.5
16
0.4
14
12
StandardError
Total Inform
ation
18
0.3
Reliability ≈ 0.90Reliability ≈ 0.90
10
8
0.2
6
4
0.1
Reliability ≈ 0.70
2
0
-3
Reliability ≈ 0.70
0.0
-2
-1
0
1
2
3
Theta
Total Information
61
Standard Error
ISPOR June 2012
IRT: VFQ-25 Recommendations
Misfit due to
categories
Item
2 – general vision
Misfit due to
relationship to theta
Local
dependence
Recommendation
No change needed
3 – worry
4 – amount pain
Delete
5 – reading normal newsprint
No change needed
6 – seeing well up close
No change needed
7 – finding objects on crowded shelf
No change needed
8 – reading street signs
9 – going down stairs at night
No change needed
10 – seeing objects off to side
No change needed
11 – seeing how people react
No change needed
12 – difficulty matching clothes
Dichotomize response categories
13 – visiting others
Dichotomize response categories
14 – going out to movies/plays
No change needed
15c – daylight driving
Combine with 16 and 16a
16 – nighttime driving
Combine with 15c and 16a
16a – difficult conditions driving
Combine with 15c and 16
17 – accomplish less
Dichotomize response categories
18 – limited in endurance
Dichotomize response categories
19 – amount time pain
Delete
20 – stay home most of time
Dichotomize response categories
21 – frustrated
Dichotomize response categories
22 – no control
Dichotomize response categories
23 – rely too much on others’ word
Delete
24 – need much help from others
Dichotomize response categories
25 – embarrassment
62
ISPOR June 2012
Dichotomize response categories
---Confirm all modifications with qualitative evidence in the targeted population---
IRT: Revised VFQ Items 2-10 ICCs and
Information
Group 1, VFQ_2rev
Group 1, VFQ_3rev
1.0
1.0
2.6
2.4
0.9
3
2.6
2.4
0.9
0
2.2
0.8
2.2
0.8
2.0
0.7
1.8
1.6
0.4
1.8
1.6
1.4
4
0.5
1.2
0.4
1.0
3
2
1
Information
1.2
Information
1.4
0.5
0.6
Probability
0.6
Probability
2.0
2
1
0.7
1.0
0
0.8
0.3
0.8
0.3
0.6
0.6
0.2
0.2
0.4
4
0.1
0.4
0.1
0.2
0.0
-2
-1
0
1
2
0.2
0.0
-3
3
0.0
-2
-1
0
Theta
Group 1, VFQ_5rev
2
3
Group 1, VFQ_6rev
1.0
2.4
0
Group 1, VFQ_7rev
1.0
2.6
4
0.9
1
Theta
4
2.4
0.8
1.6
0.2
Probability
Probability
0.4
0.4
0.1
0.2
-1
0
1
2
3
0.4
0.1
0.2
0.0
-3
0.0
-2
0.8
0.6
0.2
0.0
-3
1.0
0
0.3
0.6
0.2
0.1
1.4
1.2
0.8
0.6
1.6
0.5
0.4
1.0
0.3
1.8
3
2
-1
0
1
2
0.2
0.0
-3
0.0
-2
Theta
3
0.0
-2
-1
0
Theta
Group 1, VFQ_8rev
4
2.4
2
3
Group 1, VFQ_10rev
4
1.0
2.6
0.9
1
Theta
Group 1, VFQ_9rev
1.0
4
1.0
2.6
2.4
0.9
2.2
2.6
2.4
0.9
2.2
0.8
Information
1.2
0.8
0.3
3
0.4
1.0
1
1.4
2
0.5
0.6
Information
1.2
2
0.4
2.0
1.6
1
1
0.7
1.8
0.6
Information
1.4
3
2.2
2.0
0.7
1.8
0.5
2.4
0.8
0
2.0
0.6
2.6
0.9
2.2
0.8
0.7
4
1.0
2.6
0.9
2.2
Probability
0.0
-3
2.2
0.8
0.8
2.0
2.0
2.0
1
0.7
1
Probability
3
1.8
0.1
63
3
0.6
0.4
0.1
0.2
0.0
-3
0.0
2
0.8
0.2
0.4
0.2
1
1.0
0.3
0.6
0.4
0
1.2
0
0.8
0.2
-1
1.4
Retained items have reasonable slopes
and a good spread of categories
0
0.6
-2
1.6
2
0.4
1.0
0.3
0.2
0.1
1.2
3
0.5
Information
0.8
0
2
0.4
1.0
0.3
1.4
0.5
0.6
Information
1.2
0.4
Information
1.4
3
0.5
1.6
0.6
Probability
1.6
2
0.0
-3
0.7
1.8
1
0.6
Probability
0.7
1.8
-1
0
ISPOR June 2012
Theta
1
2
0.2
0.0
-3
0.0
-2
3
0.0
-2
-1
0
Theta
1
2
3
Theta
IRT: Revised VFQ Items 11-18 ICCs and
Information
Group 1, VFQ_12rev2
Group 1, VFQ_11rev
0.9
Group 1, VFQ_13rev2
1
1.0
2.6
1.6
3
2
1.2
1.4
0.5
1.2
0.8
2.0
1.6
0.4
1.0
0
1.2
0.6
0.4
0.0
-3
0.0
2
3
0.4
0.1
0.2
0.2
1
0.8
0.2
0.1
0
1.0
0.6
0.4
-1
1.4
0.3
0.2
-2
1.6
0.5
0.4
0.8
0.6
0.0
-3
1.8
0.6
1.0
0.3
0.2
0.1
2.4
0.7
1.8
0.6
Probability
1.4
1
0.4
0.3
0.7
-1
0
1
2
0.2
0.0
-3
0.0
-2
3
0.0
-2
-1
0
Theta
Theta
2
3
Group 1, VFQ_16rev2
4
1.0
1
1.0
2.6
2.6
0
2.4
0.9
2.4
0.9
2.2
2.2
0.8
0
2.0
0.7
2.0
0.7
1.8
1.6
0.4
1.2
1.6
1.4
1.2
2
0.4
1.0
0.8
0.3
1.8
0.5
Information
1
Information
1.4
3
0.5
0.6
Probability
0.6
1.0
0.8
0.3
0.6
0.6
0.2
0.2
0.4
0.1
0.4
0.1
0.2
0.0
0.0
-3
-2
-1
0
1
2
0.2
0.0
-3
3
0.0
-2
-1
0
Theta
1
2
3
Theta
Group 1, VFQ_17rev
1.0
Group 1, VFQ_18rev
1
0
1.0
2.6
2.4
0.9
1
0
2.6
2.4
0.9
2.2
2.2
0.8
0.8
2.0
0.7
2.0
0.7
1.8
1.6
1.2
0.4
1.0
0.8
0.3
1.8
1.6
1.4
0.5
1.2
0.4
1.0
0.8
0.3
0.6
0.2
0.6
0.2
0.4
0.1
0.0
-3
0.2
0.0
-2
-1
0
Theta
64
ISPOR June 2012
1
2
3
0.4
0.1
0.0
-3
0.2
0.0
-2
-1
0
Theta
1
2
3
Information
0.5
Information
1.4
0.6
Probability
0.6
Probability
Probability
1
Theta
Group 1, VFQ_14rev
0.8
Information
0.5
2.6
0.8
2.0
1.8
Information
Probability
0.6
1
0
2.2
0.8
2.0
0.7
0.9
2.2
Information
Response categories are
now distinct
2.4
0.9
2.2
0.8
1.0
2.6
0
2.4
Probability
4
1.0
IRT: Revised VFQ Items 19-25 ICCs and
Information
Group 1, VFQ_20rev
1.0
Group 1, VFQ_21rev
1
0
1.0
2.6
2.4
0.9
1
0
2.6
2.4
0.9
2.2
2.2
0.8
0.8
2.0
0.7
1.6
1.2
0.4
1.6
1.4
1.2
0.4
1.0
0.8
0.3
1.8
0.5
1.0
0.8
0.3
0.6
0.6
0.2
0.2
0.4
0.1
0.0
-3
0.4
0.1
0.2
-1
0
1
2
0.2
0.0
-3
0.0
-2
3
0.0
-2
-1
Theta
0
1
2
3
Theta
Group 1, VFQ_22rev
1.0
Group 1, VFQ_24rev
1
0
1.0
2.6
2.4
0.9
1
0
2.6
2.4
0.9
2.2
2.2
0.8
0.8
2.0
0.7
1.6
1.2
0.4
1.6
Probability
1.4
1.2
0.4
1.0
0.8
0.3
1.8
0.5
1.0
0.8
0.3
0.6
0.6
0.2
0.2
0.4
0.1
0.4
0.1
0.2
0.0
-3
0.0
-3
0.0
-2
-1
0
1
2
3
0.2
0.0
-2
-1
Theta
0
1
2
3
Theta
Group 1, VFQ_25rev
1
1.0
2.6
0
2.4
0.9
2.2
0.8
2.0
0.7
1.8
1.6
0.6
1.2
0.4
Information
1.4
0.5
Response categories
are now distinct
1.0
0.8
0.3
0.6
0.2
0.4
0.1
0.2
0.0
0.0
-3
-2
-1
0
1
2
3
Theta
65
ISPOR June 2012
IRT: Revised VFQ Item Parameters and Item Fit
a (SE)
b1 (SE)
b2 (SE)
b3 (SE)
b4 (SE)
Avg. b
S-X2
PValue
2 – general vision
1.64 (0.21)
-3.27 (0.42)
-1.15 (0.15)
1.01 (0.16)
4.05 (0.72)
0.16
20.97
0.051
3 – worry
1.19 (0.16)
-1.42 (0.21)
-0.04 (0.14)
1.39 (0.22)
2.82 (0.40)
0.69
35.48
0.080
5 – reading normal newsprint
1.73 (0.20)
-1.63 (0.18)
-0.76 (0.12)
0.29 (0.12)
1.47 (0.18)
-0.16
29.60
0.128
6 – seeing well up close
2.09 (0.23)
-2.26 (0.23)
-1.12 (0.13)
-0.03 (0.10)
0.99 (0.14)
-0.61
30.88
0.057
7 – finding objects on crowded
shelf
2.41 (0.28)
-3.14 (0.39)
-1.76 (0.16)
-0.78 (0.11)
0.21 (0.10)
-1.37
19.91
0.030
8 – reading street signs
1.97 (0.23)
-3.41 (0.46)
-1.83 (0.19)
-0.62 (0.11)
0.51 (0.12)
-1.34
17.54
0.354
9 – going down stairs at night
1.80 (0.21)
-3.33 (0.43)
-1.63 (0.18)
-0.49 (0.11)
0.6 0(0.13)
-1.21
15.46
0.631
10 – seeing objects off to side
2.52 (0.30)
-3.12 (0.39)
-1.71 (0.16)
-0.89 (0.11)
0.01 (0.10)
-1.43
17.52
0.014
11 – seeing how people react
2.12 (0.26)
-3.27 (0.43)
-2.24 (0.23)
-1.28 (0.14)
-0.28 (0.10)
-1.77
18.42
0.010
12 – difficulty matching clothes
2.09 (0.42)
-1.70 (0.20)
-
-
-
-1.70
5.90
0.015
13 – visiting others
3.24 (0.97)
-2.15 (0.23)
-
-
-
-2.15
14.85
0.249
14 – going out to movies/plays
2.40 (0.30)
-2.32 (0.25)
-1.58 (0.16)
-0.89 (0.12)
0.02 (0.10)
-1.19
-
-
15+16 – driving
1.32 (0.30)
-0.99 (0.23)
-
-
-
-0.99
7.54
0.276
17 – accomplish less
3.05 (0.61)
-0.57 (0.11)
-
-
-
-0.57
6.86
0.076
18 – limited in endurance
2.88 (0.57)
-0.94 (0.13)
-
-
-
-0.94
-
-
20 – stay home most of time
3.10 (0.62)
-1.16 (0.12)
-
-
-
-1.16
-
-
21 – frustrated
1.93 (0.32)
-0.70 (0.13)
-
-
-
-0.70
7.52
0.184
22 – no control
2.81 (0.50)
-0.64 (0.11)
-
-
-
-0.64
4.00
0.135
24 – need much help from others
2.69 (0.50)
-1.11 (0.13)
-
-
-
-1.11
-
-
25 – embarrassment
2.02 (0.40)
-1.44 (0.19)
-
-
-
-1.44
-
-
Item
SE = standard error.
66
ISPOR June 2012
Parameters are more reasonable
and item fit improves
Information
1.4
0.5
0.6
Information
Probability
2.0
0.7
1.8
0.6
Probability
Information
0.5
0.6
Information
1.4
Probability
0.6
Probability
2.0
0.7
1.8
IRT: Revised VFQ Local Dependence
X2
Item
2
3
5
6
7
8
9
10
11
12
13
14 15+16 17
18
20
21
22
24
2 – general vision
0.1
3 – worry
0.0 -0.1
5 – reading normal newsprint
0.1 1.8 -0.2
6 – seeing well up close
0.1 0.8 1.1 5.1
7 – finding objects on crowded shelf
0.4 2.8 0.2 2.6 0.4
8 – reading street signs
0.2 -0.4 1.0 0.4 -0.3 -0.2
9 – going down stairs at night
0.1 0.9 0.3 -0.5 -0.8 0.0 0.1
10 – seeing objects off to side
0.5 1.2 1.8 0.3 -0.6 1.0 1.7 1.0
11 – seeing how people react
0.2 0.5 0.0 0.2 4.0 1.2 0.6 0.9 1.3
12 – difficulty matching clothes
0.0 -0.5 -0.6 0.2 1.7 -0.8 -0.5 0.5 -0.3 1.9
13 – visiting others
0.1 -0.5 1.1 0.6 -0.1 -0.4 3.1 1.3 0.3 0.7 -0.6
14 – going out to movies/plays
1.1 0.0 0.5 0.9 1.2 0.3 1.8 -0.1 -0.4 0.2 -0.6 3.3
15+16 – driving
1.0 2.8 0.7 1.5 1.4 0.8 0.6 0.7 2.7 1.0 0.3 1.4 2.9
17 – accomplish less
0.5 1.3 0.2 0.8 3.3 1.4 1.5 2.1 1.9 4.3 0.0
-
1.1
5.7
18 – limited in endurance
0.9 4.1 0.0 0.7 0.7 2.8 2.0 1.5 2.5 0.5 0.2
-
1.6
3.4
8.4
20 – stay home most of time
0.1 0.2 1.0 -0.1 -0.7 0.8 0.2 -0.5 0.5 2.1 -0.5 0.0 -0.2
4.4
1.8 0.8
21 – frustrated
0.3 0.8 2.2 0.1 -0.9 1.7 3.2 0.3 1.5 0.7 -0.2
-
1.3
1.4
2.3 1.6 5.3
22 – no control
0.6 0.3 -0.1 0.4 0.8 1.4 -0.3 -0.1 0.7 0.6 1.1
-
0.3
2.1
7.9 1.6 1.7 1.7
24 – need much help from others
0.0 0.8 1.5 -0.3 -1.0 3.1 0.3 -1.1 1.6 0.5 -0.6 -0.3 0.6
1.3
2.3 1.3 0.3 0.1 2.7
25 – embarrassment
0.3 2.7 -0.1 2.5 -0.1 2.3 -0.3 -0.5 0.6 0.6 -0.5 1.5 0.2
1.7
1.7 0.7 1.7 1.1 3.6 4.1
67
Local dependence is limited and no
longer associated with item content
(beyond 5 and 6)
ISPOR June 2012
IRT: Revised VFQ Information and Reliability
Group 1, Total Information Curve
0.8
Better measurement is provided without
sacrificing reliability, but measurement could
be improved for people with better visual
functioning if items with higher thresholds
were added (to give people somewhere “to go”
as their condition improves)
25
24
23
0.7
22
21
20
19
0.6
18
17
16
0.5
14
13
0.4
12
Standard Error
Total Information
15
11
10
0.3
Reliability ≈ 0.90
9
8
7
0.2
6
5
4
0.1
3
Reliability ≈ 0.70
2
1
0
-3
0.0
-2
-1
0
1
Theta
Total Information
68
ISPOR June 2012
Standard Error
2
3
What have we learned?
Jennifer Petrillo, PhD
Associate Director, PRO Specialist
Novartis Pharmaceuticals Corporation
CTT: Summary Results
Item
Missing
(>5%)
Floor/Ceiling
(40%/5%)
Skewness
(>|2|)
Inter-item
(>0.8)
Item-total
(<0.3)
Recommendation
2 – general vision
3 – worry
4 – amount pain
Remove pain from the scale and measure separately
5 – reading normal newsprint
6 – seeing well up close
7 – finding objects on crowded shelf
8 – reading street signs
9 – going down stairs at night
10 – seeing objects off to side
11 – seeing how people react
13 – visiting others
Consider removing matching clothes due to missingness
and skewness
Consider scoring social and dependence items separately
14 – going out to movies/plays
Consider scoring social and dependence items separately
15c – daylight driving
16 – nighttime driving
16a – difficult conditions driving
17 – accomplish less
18 – limited in endurance
Create a triplet for driving items
Create a triplet for driving items
Create a triplet for driving items
19 – amount time pain
Remove pain from the scale and measure separately
12 – difficulty matching clothes
20 – stay home most of time
21 – frustrated
22 – no control
23 – rely too much on others’ word
Consider scoring social and dependence items separately
24 – need much help from others
Consider scoring social and dependence items separately
25 – embarrassment
70
ISPOR June 2012
RMT: Summary Results
Item
Response categories
Item misfit
Targeting
Change potential
Recommendation
(sub-optimal)
Recommendation
(in line with RMT)
2 – general vision
3 – worry
4 – amount pain
5 – reading normal newsprint
6 – seeing well up close
7 – finding objects on crowded shelf
8 – reading street signs
9 – going down stairs at night
10 – seeing objects off to side
11 – seeing how people react
12 – difficulty matching clothes
13 – visiting others
14 – going out to movies/plays
15c – daylight driving
16 – nighttime driving
16a – difficult conditions driving
17 – accomplish less
18 – limited in endurance
19 – amount time pain
20 – stay home most of time
21 – frustrated
22 – no control
23 – rely too much on others’ word
24 – need much help from others
25 – embarrassment
71
ISPOR June 2012
IRT: Summary Results
Item
Misfit due to
categories
Misfit due to
relationship to theta
Local
dependence
Recommendation
2 – general vision
3 – worry
4 – amount pain
Combine with 19
5 – reading normal newsprint
6 – seeing well up close
7 – finding objects on crowded shelf
8 – reading street signs
9 – going down stairs at night
10 – seeing objects off to side
11 – seeing how people react
12 – difficulty matching clothes
13 – visiting others
14 – going out to movies/plays
15c – daylight driving
Combine with 16 and 16a
16 – nighttime driving
Combine with 15c and 16a
16a – difficult conditions driving
Combine with 15c and 16
17 – accomplish less
Dichotomize response categories
18 – limited in endurance
Dichotomize response categories
19 – amount time pain
Combine with 4
20 – stay home most of time
Dichotomize response categories
21 – frustrated
Dichotomize response categories
22 – no control
Dichotomize response categories
23 – rely too much on others’ word
Delete
24 – need much help from others
Dichotomize response categories
25 – embarrassment
72
ISPOR June 2012
Dichotomize response categories
Conclusion
ƒ These were theoretical analyses; not proposing all PROs need to go
through this rigor for instrument development
ƒ In this exercise, each methodology provided complementary information
with the potential to optimize instrument composition and scoring
• Modification decisions should incorporate consideration of qualitative
results and context of use
ƒ Consideration for these analyses in the future?
• What were the original development methods,
• What are our interpretation needs,
• What are the acceptance requirements, specifically for FDA, EMA, postmarketing messages, or academic use?
Huge thanks to the RTI Health Solutions and ScaleReport teams for ensuring
a respectful, progressive discussion on measurement theory approaches!!
EMA = European Medicines Agency; FDA = Food and Drug Administration.
73
ISPOR June 2012
Audience Debate
Discussion points
ƒ Why are the recommendations different?
ƒ How would your recommendations affect the development
of an instrument?
ƒ As a PRO instrument developer and/or sponsor, why
should I use YOUR method over the others?
75
ISPOR June 2012