5/18/2015
What is the Impact of Smartphone Optimization on Long Surveys?
Introduction & Purpose
Shimon Sarraf
Jennifer Brooks
James Cole
Xiaolin Wang
Widespread adoption of mobile technologies has dramatically impacted the landscape for survey researchers (Buskirk & Andrus, 2012), and those focusing on college student populations are no exception.
National Survey of Student Engagement
Indiana University Bloomington
AAPOR 70th Annual Conference
AAPOR 70
A
lC f
Optimizing surveys for smartphones is of interest to many but ideal formats are still being developed.
May, 2015
This study investigated the impact that one smartphone
optimization approach had on a long college student survey.
National Survey of Student Engagement
 NSSE aims to understand the curricular and co‐curricular engagement of first‐year and senior college students using over 100 survey items.
NSSE Smartphone Respondents
100%
Smartphone Users (%)
Proportional Increase
63%
 Since 2000, ~ 5 million Si
2000 5 illi
students from about 1,500 US and Canadian institutions participated.
50%
38%
27%
 Formatted for “computer” though increasing numbers use smartphones to complete.
18%
13%
8%
4%
2011
2012
2013
2014
2015
Research Questions
Are there differences in respondent characteristics by smartphone optimization status. How does optimization impact: a) early abandonment, b) completion, c) item nonresponse, d) duration, e) straight‐lining, f) subjective evaluations, and
g) measurement invariance for scales?
Administration Year
1
5/18/2015
Study Details
NSSE Desktop View
• NSSE 2015 winter/spring administration
• 10 US colleges/universities
• Sample: 38,245 first‐year & senior students • Sample divided equally by smartphone optimization availability
• 7,735 respondents; 7,347 included in study
Smartphone View
Optimized – Vertical Position
Unoptimized – Vertical Position
Results
2
5/18/2015
Respondent Characteristics Early Abandonment
Optimized group less likely to abandon the survey upon viewing the very first page of survey items.
• Optimized respondents looked very similar to unoptimized and desktop groups:
26%
22%
– Gender, Age, Race/Ethnicity, Parental education, Cumulative grades, Part‐time enrollment, Academic major 12%
• Statistically significant differences found but not very large
5%
Optimized
10%
4%
Unoptimized
Desktop
Optimized First‐Year Students
Missing Data
4
3
50%
About 18% decrease in duration compared to unoptimized
group—even lower than desktop.
15.0
FY Unoptimized
FY Desktop
40%
Desktop
Seniors
Duration
Optimization appears to reduce missing data though variation exists between first‐year and senior populations.
FY Optimized
Unoptimized
12.2
2
13.0
14.6
12.9
12.2
SR Optimized
SR Unoptimized
SR Desktop
30%
20%
10%
Optimized
Unoptimized
First‐Year Students
Desktop
Optimized Unoptimized
Desktop
Seniors
106
97
103
94
100
91
88
85
82
79
76
73
70
67
64
61
58
55
52
49
46
43
40
37
34
31
28
25
22
19
16
7
13
4
10
1
0%
3
5/18/2015
Straight‐lining
Subjective Evaluations
Optimized straight‐lined less than unoptimized group
Page 1 scales (out of 6 scales)
Page 2 scales (out of 5 scales)
Page 3 scales (out of 3 scales)
1.8
1.4
1.7
1.9
Ease of use: "Very easy"
1.9
1.6
13
1.3
1.1
1.1
58%
55%
41%
1.1
Visual Design: "Excellent"
61%
59%
13
1.3
1.0
Optimization betters ease of use and visual design. 52%
46%
34%
40%
35%
37%
27%
0.5
0.3
0.3
Optimized Unoptimized
First‐Year Students
Desktop
0.3
0.4
Optimized Unoptimized
0.3
Desktop
Seniors
Optimized
Unoptimized
Desktop
Optimized First‐Year Students
Unoptimized
Desktop
Seniors
Measurement Invariance
Conclusions
Across the three groups, all first‐year and senior scales met scalar invariance criteria, except for Learning Strategies
• Optimization can improve data quality even for a long survey, while also maintaining scale properties.
• Smartphone optimized respondent data quality rivals that of desktop respondents.
• Some
Some measures indicate differences between younger measures indicate differences between younger
and older smartphone respondents in the sample. What does this mean for ongoing optimization efforts? • College student survey developers should focus on optimizaztion as smartphone usage continues to increase.
4
5/18/2015
Thank you!
Copy of this and past presentations can be found at:
nsse.iub.edu/html/publications_presentations.cfm
Additional NSSE information can be found at:
Additional
NSSE information can be found at:
nsse.indiana.edu
Feel free to contact us with any questions
regarding this study or NSSE.
ssarraf@, brooksjl@, and [email protected]
5